Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: hysds/lightweight-jobs
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v1.0.2
Choose a base ref
...
head repository: hysds/lightweight-jobs
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: develop
Choose a head ref
  • 16 commits
  • 34 files changed
  • 9 contributors

Commits on Jun 3, 2020

  1. Hc 210 (#20)

    * HC-210
    fixed imports and black formatted the notify_by_email.py
    
    * HC-210
    changing notify_by_email.sh so it takes 0 arguments.
    
    * HC-210
    updated import for MIMEMultipart
    updated job-spec to consistantly use the /home/ops/lightweight-jobs/
    directory
    
    * HC-210
    fixed encoders implementation in notify_by_email.py
    
    * HC-210
    changed directory to reference file path vs. getcdw()
    
    * HC-210
    fixed path
    
    * NSDS-210
    changed context["objectid"]
    to context["id"]. Don't know if this was supposed to have changed.
    
    * HC-210
    Changed "rule_name" to "name"
    
    * fixed get_by_id on alias bug
    
    * HC-210
    encoding string into a bytes object before b64 encoding it.
    
    * HC-210
    Forgot to include cc_recipients in send_email function.
    
    * HC-210
    specifiying ascii when encoding the query
    
    * HC-210
    decoded ascii encoded addresses.
    
    * HC-210
    fixed doc source issues.
    
    * HC-210
    fixed facetview link creation
    
    * HC-210
    Fixed facetview_url again
    
    Co-authored-by: Poreh <bporeh@benjamin-poreh.jpl.nasa.gov>
    Co-authored-by: DustinKLo <dustinlo.pcc@gmail.com>
    3 people authored Jun 3, 2020
    Copy the full SHA
    bb5877c View commit details

Commits on Sep 3, 2020

  1. HC-253: Update legacy ElasticSearch Code in Lightweight Jobs to confo…

    …rm to ES 7.1 (#21)
    
    * HC-253: Updated the get functions to ES 7.1 in ligthweight jobs
    
    The aws_get and wget functions have been updated to be compliant with ES 7.1.
    In addition, they now use the ElasticSearch utilities from HySDS commons instead of the requests library.
    
    * open the tar file as binary
    
    * fixed an issue where the first set of results don't get added to the script
    
    * fixed issue where the first set of results aren't getting captured.
    
    * read as binary
    
    * properly get the count
    
    * clear out scroll_ids after use
    
    Co-authored-by: Mike Cayanan <michael.d.cayanan@jpl.nasa.gov>
    mcayanan and Mike Cayanan authored Sep 3, 2020
    Copy the full SHA
    12f58a5 View commit details

Commits on Oct 6, 2020

  1. Copy the full SHA
    ac556d7 View commit details

Commits on Mar 13, 2021

  1. Copy the full SHA
    45926c2 View commit details

Commits on Jul 6, 2021

  1. HC-302: Add job detail information on number of datasets purged (#24)

    * HC-302: Report number of datasets purged per type
    
    * update
    
    * Support for possible list
    
    * only formulate the message if we really did find stuff that was purged.
    
    Co-authored-by: Mike Cayanan <michael.d.cayanan@jpl.nasa.gov>
    mcayanan and Mike Cayanan authored Jul 6, 2021
    Copy the full SHA
    85c2b7e View commit details

Commits on Oct 6, 2022

  1. [SSDS-2703]Parallelize purge operation (#25)

    * SSDS-2703:Initial commit
    
    * SSDS-2703:fix
    
    * SSDS-2703:Updated multiprocessing call
    
    * SSDS-2703:debug line
    
    * SSDS-2703:Fix
    
    * SSDS-2703:fix2
    
    * SSDS-2703:ServerProxy
    
    * SSDS-2703:Fix
    
    * SSDS-2703:test
    
    * SSDS-2703:Fix
    
    * SSDS-2703:Fix
    
    * SSDS-2703:Fix
    
    * SSDS-2703:Fix
    
    * SSDS-2703:test
    
    * SSDS-2703:test
    
    * SSDS-2703:test
    
    * SSDS-2703:test
    
    * SSDS-2703:test
    
    * SSDS-2703:test
    
    * SSDS-2703:test
    
    * SSDS-2703:test
    
    * SSDS-2703:test
    
    * SSDS-2703:test
    
    * SSDS-2703:test
    
    * SSDS-2703:test
    
    * SSDS-2703:test
    
    * SSDS-2703:test
    
    * SSDS-2703:test
    
    * SSDS-2703:test
    
    * SSDS-2703:test
    
    * SSDS-2703:test
    
    * SSDS-2703:test
    
    * SSDS-2703:test
    
    * SSDS-2703:test
    
    * SSDS-2703:test
    
    * SSDS-2703:test
    
    * SSDS-2703:test
    
    Co-authored-by: andrewmz <andrewmz@jpl.nasa.gov>
    azhang57 and andrewmz authored Oct 6, 2022
    Copy the full SHA
    b2de775 View commit details

Commits on Oct 26, 2022

  1. HC-439: Purge datasets v1.2.0 job time out after new pcm deployment (#26

    )
    
    * using the bulk API to purge datasets
    
    * removed multiprocessing import
    
    * edited msg details
    
    * pulling only necessary fields in purge
    
    * re-added multiprocessing for s3 deletion only
    
    * debug info, debug is too verbose
    
    * fixed bug, debug ERROR
    
    * debug INFO
    
    * added deleted datasets stats
    
    * re-did some stuff
    
    * re-did some stuff
    
    * re-did some stuff
    
    * removing objects from ES first before deleting from s3
    
    * moving log line to after we delete from es
    
    Co-authored-by: dustinlo <dustin.k.lo@jpl.nasa.gov>
    DustinKLo and dustinlo authored Oct 26, 2022
    Copy the full SHA
    0947fe0 View commit details

Commits on Dec 9, 2022

  1. HC-444: Add capability to purge job to not delete from object store (#27

    )
    
    * HC-444: Purge Job update - Add capability to not delete from object store
    
    * test using exec wrapper
    
    * Revert "test using exec wrapper"
    
    This reverts commit bb5bccd.
    
    * update message
    
    * add newline
    
    Co-authored-by: Mike Cayanan <michael.d.cayanan@jpl.nasa.gov>
    mcayanan and Mike Cayanan authored Dec 9, 2022
    Copy the full SHA
    d3ac1d1 View commit details

Commits on Apr 28, 2023

  1. need to close the multiprocessing pool (#28)

    Co-authored-by: dustinlo <dustin.k.lo@jpl.nasa.gov>
    DustinKLo and dustinlo authored Apr 28, 2023
    Copy the full SHA
    09e2e9b View commit details

Commits on Jun 15, 2023

  1. Update README.md

    DustinKLo authored Jun 15, 2023
    Copy the full SHA
    3f4083a View commit details

Commits on Jul 18, 2023

  1. Copy the full SHA
    d34be30 View commit details

Commits on Sep 1, 2023

  1. Copy the full SHA
    7c238d9 View commit details

Commits on Oct 31, 2023

  1. HC-484: OpenSearch support (#31)

    * removed _type b/c its unused and opensearch removed it
    
    * removed _type b/c its unused and opensearch removed it
    
    * skip worker by looking at the _index
    
    ---------
    
    Co-authored-by: dustinlo <dustin.k.lo@jpl.nasa.gov>
    Co-authored-by: Mike Cayanan <michael.d.cayanan@jpl.nasa.gov>
    3 people authored Oct 31, 2023
    Copy the full SHA
    6026de5 View commit details

Commits on Oct 9, 2024

  1. HC-546: Update retry job to move failed job back to job_status index …

    …if retried (#34)
    
    * HC-546: Move from job_failed to job_status index
    
    * Make sure to move to current day job_status
    
    ---------
    
    Co-authored-by: Mike Cayanan <michael.d.cayanan@jpl.nasa.gov>
    mcayanan and Mike Cayanan authored Oct 9, 2024
    Copy the full SHA
    51d703f View commit details
  2. HC-550: Update job retry to allow specifying new time limits and a ne…

    …w job queue (#33)
    
    * HC-550: Update retry job to allow updating of soft time limits and sending to another queue
    
    * add commas
    
    * cast to int
    
    * re-order
    
    ---------
    
    Co-authored-by: Mike Cayanan <michael.d.cayanan@jpl.nasa.gov>
    mcayanan and Mike Cayanan authored Oct 9, 2024
    Copy the full SHA
    a2fb7e6 View commit details

Commits on Oct 31, 2024

  1. HC-556: Update retry and purge jobs to delete all possible instances …

    …of a job across multiple indices (#35)
    
    * HC-556: Update delete_by_id function to delete by job_status-current
    
    * delete the jobs in a different way
    
    * wait for job-revoked state to be logged before continuing
    
    * use ensure_job_indexed
    
    * update how the container is being built
    
    * add user ops
    
    * revert back to print statements
    
    * revert back to print statements
    
    * query by correct thing
    
    * typo
    
    * make wait shorter
    
    * add logging
    
    * add logging format
    
    * should be retry.log
    
    * update query
    
    * update query
    
    * add comment
    
    * Revert "add comment"
    
    This reverts commit 0babf15.
    
    * update purge to remove all instances of a job
    
    * add timestamps to logging; surround with try/except
    
    * remove waiting for a revoked status
    
    ---------
    
    Co-authored-by: Mike Cayanan <michael.d.cayanan@jpl.nasa.gov>
    mcayanan and Mike Cayanan authored Oct 31, 2024
    Copy the full SHA
    8f28268 View commit details
Loading