Skip to content

Releases: hysds/hysds-framework

v5.3.1: HySDS Framework version 5.3.1 (Freddie Freeman MVP)

31 Oct 20:01
@mcayanan mcayanan
v5.3.1
2ecbdd7
Compare
Choose a tag to compare
v5.3.1: HySDS Framework version 5.3.1 (Freddie Freeman MVP) Latest
Latest

Bug fixes and enhancements:

hysds[v1.3.5]

https://github.com/hysds/hysds/releases/tag/v1.3.5

Bug Fixes and Enhancements:

  • HC-556: Fix to make the auto-retrying of jobs work properly again (#195)

hysds-dockerfiles[v5.3.1]

https://github.com/hysds/hysds-dockerfiles/releases/tag/v5.3.1

Bug fixed and enhancements:

  • bump to v5.3.1
  • HC-567: Update containers to remove /var/lib/containers/sigstore directory

hysds_commons[v1.1.4]

https://github.com/hysds/hysds_commons/releases/tag/v1.1.4

Bug fixes and enhancements:

  • hot-fix: fix failed unit test (#61)

lightweight-jobs[v1.3.2]

https://github.com/hysds/lightweight-jobs/releases/tag/v1.3.2

Bug fixes and enhancements:

  • HC-556: Update retry and purge jobs to delete all possible instances of a job across multiple indices (#35)
Assets 20

v5.3.0: HySDS Framework version 5.3.0 (Tesla Optimus Bot)

14 Oct 17:45
@mcayanan mcayanan
v5.3.0
0227b69
Compare
Choose a tag to compare
v5.3.0: HySDS Framework version 5.3.0 (Tesla Optimus Bot)

Bug fixes and enhancements:

chimera[v2.2.4]

https://github.com/hysds/chimera/releases/tag/v2.2.4

Bug fixes and enhancements:

  • HC-522-update: Updates to support moving away from OPS user (#22)
    • Use system cp call to update SELinux context label

hysds[v1.3.4]

https://github.com/hysds/hysds/releases/tag/v1.3.4

Bug Fixes and Enhancements:

  • HC-522-update: Updates to support moving away from OPS user (#192)
    • The default PODMAN_CFG in the celeryconfig.py no longer sets the --userns=keep-id flag or the setting of the -u flag option by default. Therefore, root is the default user when running podman now, which means the host user in the podman world due to our rootless podman mode set up.
    • VERDI_HOME defaults to /root now since the Verdi container now moves away from defaulting to an OPS user. /root is also now where the HySDS Core code resides in now.
    • celeryconfig.py passes in the HOST_USER environment variable now because we need that in order to properly set the PODMAN user socket file now. This is due to the default user in podman being root now, where UID=0, but we start up the socket server with a socket file pertaining to the actual UID of the host.
    • When updating the volume source mounts, check if the beginning string matches the verdi_home argument instead of a general if verdi_home in k
    • Fixed the code to default a missing environment area in PODMAN_CFG to a list instead of a dictionary.
    • When persisting environment variables in the podman run command, use the actual environment variable value instead

hysds-dockerfiles[v5.3.0]

https://github.com/hysds/hysds-dockerfiles/releases/tag/v5.3.0

Bug fixed and enhancements:

  • bump to v5.3.0
  • HC-522-update: Updates to support moving away from the OPS user in Verdi (#12)

sdscli[v1.3.4]

https://github.com/sdskit/sdscli/releases/tag/v1.3.4

Bug fixes and Enhancements:

  • HC-522-update: Updates that support moving away from default OPS user in the Verdi container (#108)
    • Made install.sh more modular in that it uses HOST_VERDI_HOME instead of HOME to make the script support the use case of being able to deploy the PCM code bundle anywhere instead of always assuming it will be under $HOME
    • Similiarly, DATA_DIR was introduced to be able to make it more modular to reference the base working directory to be anywhere instead of assuming /data always.
    • Use cp instead of mv when deploying the AWS, .netrc, and .boto files/directories.
      supervisord.conf.tmpl was updated such that it no longer has the httpd service as a default due to JPL security in starting up webdav within Verdi. Rather, httpd service is going to be started up by the system instead.
Assets 20
Loading

v5.2.2: HySDS Framework version 5.2.2 (BeetleJuice BeetleJuice)

10 Oct 16:49
@mcayanan mcayanan
v5.2.2
c1995c6
Compare
Choose a tag to compare
v5.2.2: HySDS Framework version 5.2.2 (BeetleJuice BeetleJuice)

Bug fixes and enhancements:

hysds[v1.3.3]

https://github.com/hysds/hysds/releases/tag/v1.3.3

Bug Fixes and Enhancements:

  • hot-fix: Update return from psutil.disk_partitions function call (#190)

Due to psutil 6.0.0 release, it is found that it broke the instance_stats service with the following error:

Traceback (most recent call last):
  File "/export/home/hysdsops/mozart/ops/hysds/scripts/log_instance_stats.py", line 119, in <module>
    daemon(args.redis_url, args.redis_key, args.interval)
  File "/export/home/hysdsops/mozart/ops/hysds/scripts/log_instance_stats.py", line 79, in daemon
    for device, mnt_point, fs_type, fs_opts, max_file, max_path, *other in psutil.disk_partitions():
ValueError: not enough values to unpack (expected at least 6, got 4)

According to the psutil pull request, giampaolo/psutil#2405, max_file and max_path were removed because if there's a NFS (network filesystem) this function can potentially take a long time to complete.

  • HC-522: Added a new Podman class to support running containers using Podman (#189)

    • new Podman class added to the engine factory

  • HC-540: Cache the function objects being returned by orchestrator.py get_function method (#191)

    • Cache the function objects being returned by get_function instead of having to exec import every time. This seems to improve CPU performance 3x.

  • HC-546: Update logstash to move a failed job over to a new job_failed index (#193)

hysds-dockerfiles[v5.2.2]

https://github.com/hysds/hysds-dockerfiles/releases/tag/v5.2.2

Bug fixed and enhancements:

  • bump to v5.2.2
  • HC-522: New Script to run Verdi in Podman (#11)

hysds_commons[v1.1.3]

https://github.com/hysds/hysds_commons/releases/tag/v1.1.3

Bug fixes and enhancements:

  • HC-526: Fix search_by_id function to not throw KeyError: 'hits' error (#60)

sdscli[v1.3.3]

https://github.com/sdskit/sdscli/releases/tag/v1.3.3

Bug fixes and Enhancements:

  • HC-522: Updates to support deployment of Verdi to run with Podman (#107)

    • Add new variables HOST_VERDI_HOME and HOST_USER to job worker supervisord service

  • HC-546: Add job_failed index to job_status template (#109)

sciflo[v1.3.7]

https://github.com/hysds/sciflo/releases/tag/v1.3.7

Bug fixes and enhancements:

  • HC-522: Create sciflo working directory in a temp dir if creating it at $HOME fails (#8)

lightweight-jobs[v1.3.1]

https://github.com/hysds/lightweight-jobs/releases/tag/v1.3.1

Bug fixes and enhancements:

  • HC-550: Update job retry to allow specifying new time limits and a new job queue (#33)
  • HC-546: Update retry job to move failed job back to job_status index if retried (#34)
Assets 20
Loading

v5.2.1: HySDS Framework version 5.2.1 (Papatui)

25 Mar 17:32
@mcayanan mcayanan
v5.2.1
32f33e5
Compare
Choose a tag to compare
v5.2.1: HySDS Framework version 5.2.1 (Papatui)

Bug fixes and enhancements

hysds[v1.3.2]

https://github.com/hysds/hysds/releases/tag/v1.3.2

Bug Fixes and Enhancements:

  • HC-473: Implement publish_overwrite_ok as a toggle to existing context var (#166)
  • HC-518: Added mechansim for optionally partitioning the triage indices over time (#188)
  • HC-520: Fix job worker watchdog to properly handle cases where hard time limit was reached (#187)

hysds-dockerfiles[v5.2.1]

https://github.com/hysds/hysds-dockerfiles/releases/tag/v5.2.1

Bug fixed and enhancements:

  • bump to v5.2.1

hysds_commons[v1.1.2]

https://github.com/hysds/hysds_commons/releases/tag/v1.1.2

Bug fixes and enhancements:

  • HC-473: Passthru publish_overwrite_ok in resolve_hysds_job (#55)

mozart[v2.3.1]

https://github.com/hysds/mozart/releases/tag/v2.3.1

Bug fixes and enhancements:

  • HC-473: Expose publish_overwrite_ok via APIs (#50)

osaka[v1.2.4]

https://github.com/hysds/osaka/releases/tag/v1.2.4

Bug fixes and enhancements:

  • HC-514: Update usage of moto to conform to latest module updates (#39)

sdscli[v1.3.2]

https://github.com/sdskit/sdscli/releases/tag/v1.3.2

Bug fixes and Enhancements:

  • HC-517: Enable importing of container name-only images and user-rules via sdscli (#106)
Assets 20
Loading

v5.2.0: HySDS Framework version 5.2.0 (Red Wednesday)

16 Feb 17:23
@mcayanan mcayanan
v5.2.0
685d1f0
Compare
Choose a tag to compare
v5.2.0: HySDS Framework version 5.2.0 (Red Wednesday)

Bug fixes and enhancements

hysds-dockerfiles[v5.2.0]

https://github.com/hysds/hysds-dockerfiles/releases/tag/v5.2.0

Bug fixed and enhancements:

  • bump version

hysds_ui[v1.1.10]

https://github.com/hysds/hysds_ui/releases/tag/v1.1.10

Bug fixes and enhancements:

  • HC-509: Inconsistent query between Tosca/Figaro facet and On-Demand page (#121)
Assets 20
Loading

v5.1.0: HySDS Framework version 5.1.0 (Golden Globes Host Jo Koy)

10 Jan 18:08
@mcayanan mcayanan
v5.1.0
663ab98
Compare
Choose a tag to compare
v5.1.0: HySDS Framework version 5.1.0 (Golden Globes Host Jo Koy)

hysds[v1.3.1]

https://github.com/hysds/hysds/releases/tag/v1.3.1

  • HC-502: Enable clobbering when publishing triage datasets (#183)
  • hot-fix: Should pin lxml to less than 5.0.0 (#185)

hysds-dockerfiles[v5.1.0]

https://github.com/hysds/hysds-dockerfiles/releases/tag/v5.1.0

  • bump version

hysds_commons[v1.1.1]

https://github.com/hysds/hysds_commons/releases/tag/v1.1.1

  • HC-503: Fixed an issue where job_iterator could return duplicate results (#59)

sdscli[v1.3.1]

https://github.com/sdskit/sdscli/releases/tag/v1.3.1

  • hot-fix: Fix the index pattern in the ISM Mozart policy (#105)

hysds_ui[v1.1.9]

https://github.com/hysds/hysds_ui/releases/tag/v1.1.9

  • HC-508: Remove "Kibana" from Metrics title in Sources menu (#120)
Assets 20
Loading

v5.1.0-beta.1: HySDS Framework version 5.1.0-beta.1 (Travis and Taylor pt. 2)

31 Oct 21:39
@DustinKLo DustinKLo
v5.1.0-beta.1
8d4e6f2
Compare
Choose a tag to compare
v5.1.0-beta.1: HySDS Framework version 5.1.0-beta.1 (Travis and Taylor pt. 2)

To help you formulate the release body, links to updated repo releases are listed below:

Bug fixes and enhancements:

grq2[v2.1.0]

https://github.com/hysds/grq2/releases/tag/v2.1.0

  • HC-484: OpenSearch support (#65)

    • doubled ES exceptions
    • fixed timeout to 2m
    • setting @timestamp manually instead of using ingest pipeline
    • using hysds.es_utils for elasticsearch connection
    • fixed ES params in settings.cfg
    • removed import_ES_AWS.py and added Opensearch support in import_AWS.py
    • added opensearchpy in setup.py
    • fixed some adhoc scripts
    • Opensearch & elasticsearch use different units for timeout in the bulk API
    • fixed settings.cfg.tmpl

Co-authored-by: dustinlo [email protected]

hysds[v1.3.0]

https://github.com/hysds/hysds/releases/tag/v1.3.0

  • HC-484: OpenSearch support (#182)

    • celeryconfig.tmpl accomodate managed ES/Opensearch
    • doubling ElasticsearchExcpetion with OpenSearchException
    • halving the backoff in search_es in user_rules_datasets
    • cleaned up the celeryconfig.tmpl for ES/OS settings
    • es_utils can connect to either aws managed/standalone Elasticsearch/Opensearch
    • removed aws-requests-auth b/c opensearch-py has better signature utility
    • enclosed quotes in ES_URL
    • regex parsing aws region
    • added addtl logic to jinja templating for es_url
    • JOBS_ES_ENGINE -> MOZART_ES_ENGINE
    • fixed logstash to use opensearch plugin
    • added scripts to upload ILM/ISM and ES templates
    • JOB_AWS_ES -> MOZART_AWS_ES
    • using the opensearch plugin in logstash
    • fixed conflict errors when installing iLM/ISM
    • undid opensearch logstash conf
    • revert opensearch plugin to elasticsearch plugin
    • fixed cyclical dependency issue in task_worker
    • added ilm_enabled => false to logstash conf b/c by default logstash will make calls to _xpack which is not inclued in elasticsearch-oss
    • using the opensearch logstash plugin
    • added script to clean out elasticsearch/opensearch if using AWS
    • fixed celeryconfig to allow for flexible multi-node es cluster
    • added jinja templating to handle multiple hosts in mozart/metrics logstash
    • added opensearch-py to requirements.txt, addtl kwargs to es_util
    • added optional hosts in es_util functions
    • added metrics es connection function
    • support opensearch/elasticsearch output plugin in indexer.conf
    • fixed indexer.conf.metrics. re-added try/except
    • removed kibana if using opensearch
    • removed jinja if/else for kibana
    • fixed clean_indices_from_alias.py to not use requests
    • fix clean_indices_from_alias.py
    • Update configs/logstash/indexer.conf.metrics
    • fixed unittests to allow for sniff
    • add functions to get the ES engine
    • get_mozart_es, get_grq_es not called in utils.py
    • moved get_grq_mozart() and get_mozart_es() into functions in user_rules

Co-authored-by: dustinlo [email protected]
Co-authored-by: Mike Cayanan [email protected]

hysds-dockerfiles[v5.1.0-beta.1]

https://github.com/hysds/hysds-dockerfiles/releases/tag/v5.1.0-beta.1

  • bump version

hysds_commons[v1.1.0]

https://github.com/hysds/hysds_commons/releases/tag/v1.1.0

  • HC-484: OpenSearch support (#58)
    • split the ES and Opensearch Utility into 2 classes
    • fixed bug in _pit where size defaults to 10 when not given
    • added opensearch-py to setup.py
    • able to use multiple hosts in elasticsearch
    • added multi-host to OpenSearchUtility
    • es-oss doesnt support PIT so defaulting to scroll
    • if user calls _pit, it will then use the search_after w/o _pit

Co-authored-by: dustinlo [email protected]

lightweight-jobs[v1.3.0]

https://github.com/hysds/lightweight-jobs/releases/tag/v1.3.0

  • HC-484: OpenSearch support (#31)
    • removed _type b/c its unused and opensearch removed it

Co-authored-by: dustinlo [email protected]
Co-authored-by: Mike Cayanan [email protected]

mozart[v2.3.0]

https://github.com/hysds/mozart/releases/tag/v2.3.0

  • HC-484: OpenSearch support (#54)
    • using hysds.es_utils for ES connection
    • bumped minor version
    • removed ES_URL bc it is not needed anymore

Co-authored-by: dustinlo [email protected]

pele[v1.2.0]

https://github.com/hysds/pele/releases/tag/v1.2.0

  • HC-484: OpenSearch support (#33)
    • supporting both elasticsearch-py and opensearchpy
    • removing self.logger in Query class
    • settings file supporting both AWS managed and opensearch/ES
    • removing aws-requests-auth and using opensearchpy's builtin AWSV4 signer
    • reduced MAX_SIZE from max int to 10000
    • bump minor version
    • removed commented out lgoger
    • opensearch searches
    • pele supports cluster mode multiple hosts

Co-authored-by: dustinlo [email protected]

sdscli[v1.3.0]

https://github.com/sdskit/sdscli/releases/tag/v1.3.0

  • HC-490 - Supporting Opensearch Dashboards (#104)
    • migrating from kibana -> opensearch dashboards
    • distinguishing between ILM/ISM
    • removing hardcoded ILM in templates
    • using python scripts to install ILM/ISM instead of raw curl requests
    • commented out ingest pipeline for consolidation of databases
    • removed warm_migration action bc returning 403
    • removed cold state b/c AWS opensearch uses cold_delete and non managed uses delete only
    • copied kibana_dashboard_import into opensearch_dashboards_import
    • updated import_dashboards for opensearch
    • added mozart_redis_host in fabfile.py
    • separating the kibana/opensearch logic in the update kibana function
    • flipped logic around to make elasticseaech the default
    • fixed opensearch_dashboards.yml
    • fixed sdswatch shell script to use logstash-oss-7.16.3
    • logstash 7.9.3 -> logstash-oss 7.16.3
    • forgot to add sleep in import opensearch dashboard
    • remove field in ism
    • edited jvm.options
    • removed trailing comma
    • no kibana in opensearch
    • added opensearch support for sds status
    • added base path /metrics for dashboards
    • fixed opensearch dashboards endpoint
    • removed /metrics
    • removed cold and addtl support for multiple hosts
    • separated sdswatch for opensearch vs elasticsearch
    • fixed clean_indices_from_alias.py
    • move ism/ilm polivies inside if/else
    • use hsysds es_util to get ES status
    • call ES core to call ping function

Co-authored-by: dustinlo [email protected]
Co-authored-by: Mike Cayanan [email protected]

Contributors

timestamp
Assets 20
Loading

v5.0.1: HySDS Framework version 5.0.1 (Travis and Taylor)

13 Oct 02:29
@pymonger pymonger
v5.0.1
13307bc
Compare
Choose a tag to compare
v5.0.1: HySDS Framework version 5.0.1 (Travis and Taylor)

Bug fixes and enhancements:

chimera[v2.2.3]

https://github.com/hysds/chimera/releases/tag/v2.2.3

Bug fixes and enhancements:

  • handle localizing files in dict (#21)
  • bump version

grq2[v2.0.26]

https://github.com/hysds/grq2/releases/tag/v2.0.26

Bug fixes and enhancements:

  • hot-fix: Pin werkzeug module to resolve 'url_quote_plus' import error (#64)
  • bump version

hysds-dockerfiles[v5.0.1]

https://github.com/hysds/hysds-dockerfiles/releases/tag/v5.0.1

Bug fixed and enhancements:

  • bump version

pele[v1.1.9]

https://github.com/hysds/pele/releases/tag/v1.1.9

Bug fixes and enhancements:

  • hot-fix: pin werkzeug (#32)
  • bump version
Assets 20
Loading

v5.0.0: HySDS Framework Version 5.0.0 (ElasticSearch Pro Max Plus)

28 Sep 15:25
@mcayanan mcayanan
v5.0.0
bd2b394
Compare
Choose a tag to compare
v5.0.0: HySDS Framework Version 5.0.0 (ElasticSearch Pro Max Plus)

Bug fixes and enhancements:

hysds-dockerfiles[v5.0.0]

https://github.com/hysds/hysds-dockerfiles/releases/tag/v5.0.0

Bug fixed and enhancements:

  • Bump to v5.0.0

mozart[v2.2.5]

https://github.com/hysds/mozart/releases/tag/v2.2.5

Bug fixes and enhancements:

  • HC-496: Updates to optionally support a separate Mozart ES cluster deployment (#53)

sdscli[v1.2.3]

https://github.com/sdskit/sdscli/releases/tag/v1.2.3

Bug fixes and enhancements:

  • HC-496: Updates to optionally support a separate Mozart ES cluster deployment (#103)
Assets 20
Loading

v5.0.0-rc.2: HySDS Framework Version 5.0.0-rc.2 (ElasticSearch Pro Max)

11 Sep 19:49
@mcayanan mcayanan
v5.0.0-rc.2
660a9ec
Compare
Choose a tag to compare
v5.0.0-rc.2: HySDS Framework Version 5.0.0-rc.2 (ElasticSearch Pro Max)

Bug fixes and enhancements:

grq2[v2.0.25]

https://github.com/hysds/grq2/releases/tag/v2.0.25

Specify body of new release. Previous release's body below:

Bug fixes and enhancements:

  • HC-487: Test out and deploy locally a Elasticsearch service in clustered mode (multi-node) (#63)

hysds[v1.2.12]

https://github.com/hysds/hysds/releases/tag/v1.2.12

Bug fixes and enhancements:

  • HC-487: Test out and deploy locally a Elasticsearch service in clustered mode (multi-node) (#181)

hysds-dockerfiles[v5.0.0-rc.2]

https://github.com/hysds/hysds-dockerfiles/releases/tag/v5.0.0-rc.2

Bug fixes and enhancements:

  • bump to v5.0.0-rc.2

hysds_commons[v1.0.16]

https://github.com/hysds/hysds_commons/releases/tag/v1.0.16

Bug fixes and enhancements:

  • HC-487: Test out and deploy locally a Elasticsearch service in clustered mode (multi-node) (#57)

lightweight-jobs[v1.2.6]

https://github.com/hysds/lightweight-jobs/releases/tag/v1.2.6

Bug fixes and enhancements:

  • query only job_status-* indices when purging jobs (#30)

mozart[v2.2.4]

https://github.com/hysds/mozart/releases/tag/v2.2.4

Bug fixes and enhancements:

  • HC-487: Test out and deploy locally a Elasticsearch service in clustered mode (multi-node) (#52)
Assets 20
Loading