As a user, I want to sync ESA PSA products from the Search API

[jordanpadams]

Checked for duplicates

Yes - I've already checked

🧑‍🔬 User Persona(s)

Data User

💪 Motivation

...so that I can have the ESA PSA products available through the Solr search

📖 Additional Details

I think the easiest route to do this is to have a separate Python script to download the XML to a local archive, and then execute harvest to load it into the legacy registry

Acceptance Criteria

Given a Registry Search API loaded with ESA PSA context products
When I perform pds-sync-api --node-name psa --download-path /path/to/download/XML
Then I expect a python script to download the XML files to --download-path (if they do not already exist)

⚙️ Engineering Details

• query Search API for all PSA context products, bundles, collections
• paginate through the results
  • check if the LIDVID has already been loaded into the Registry or not
  • if not, check if the XML is already in --download-path (using file name and ops:Label_File_Info.ops:md5_checksum)
  • if the file does not exist, download to --download-path
  • execute harvest on those XML

This will be a two-part ticket since we will then need a bash script to be added to this repo to actually execute harvest on the downloaded data.

🎉 I&T

No response

[jordanpadams]

@nutjob4life not sure of the best place to put this script. it can either go somewhere here (with a requirements.txt), our operations repo (which contains a bunch of ad-hoc scripts), or ?

[tloubrieu-jpl]

45.66666% done

[jordanpadams]

Converting this to an Epic because we have numerous other tasks to complete to make this a reality.

[jordanpadams]

📆 12/2024 status: Script completed. Need wrapper script and setup cron to do this on a regular basis.