MCP Pilot: Pilot Nucleus Deployment for CSS and MESSENGER MDIS Data

[jordanpadams]

💡 Description

• Work with IMG to connect to S3 Bucket with MESSENGER data #52
• Deploy and Test Nucleus DAG for MESSENGER processing #53
• Validate and Load all PDS4 MESSENGER data products with Nucleus #54 75% complete. tabled for CSS ingestion. will revisit in B15.0
• Prepare product labels as batches to be processed by Nucleus #75
• Support processing of PDS products with fz file format in Nucleus #74

[jordanpadams]

📆 May status: Delay of 1 sprint expected due to delays in onboarding.

[jordanpadams]

📆 June status: In progress. On schedule.

[jordanpadams]

📆 July status: In progress. Task is taking much longer than expected due to large data volumes being worked. Plan to process subset by end of build. POC task that does not impact build delivery.

[jordanpadams]

📆 August status: Deferring to B14.1. Issues with AWS changing permissions on us in NGAP. Completed a workaround to load the MESSENGER data without Nucleus for the time being. No impact on delivery to I&T.

[jordanpadams]

@ramesh-maddegoda Here is your parent Epic for the Nucleus deployment. Happy to change the scope if we want to work with a different data set. Or maybe an additional sub-task will be to first just test out whether or not this works in MCP, then the overall pilot is to deploy this for MESSENGER.

[jordanpadams]

@ramesh-maddegoda Here is your parent Epic for the Nucleus deployment. Happy to change the scope if we want to work with a different data set. Or maybe an additional sub-task will be to first just test out whether or not this works in MCP, then the overall pilot is to deploy this for MESSENGER.

[jordanpadams]

📆 October status: Demo for SBN and CSS completed on 10/27. Work ongoing to load more data operationally and prepare for beta production usage

[jordanpadams]

📆 November status: Delayed 2 sprints due to issues with cost associated with initial design of CSS processing. #74 . No impact on delivery expected.

[jordanpadams]

📆 December status: Delayed 1 additional sprint. Need to process data through Nucleus, but software is ready. No impact on delivery.

[tloubrieu-jpl]

@viviant100 , what would be the next step on that ?

[jordanpadams]

@ramesh-maddegoda have we loaded all MESSENGER data?

[jordanpadams]

📆 01/2024 status: delayed 2 sprints. no impact on other tasks

[ramesh-maddegoda]

@ramesh-maddegoda have we loaded all MESSENGER data?

@jordanpadams , when loading the messenger data I found the following bug.
#83

I have loaded the following messenger data directories to MCP OpenSearch. There were few missing products due to the bug above. I will load following directories to JPL AWS too.

MESSDEM_1001/
MSGRMDS_1001/
MSGRMDS_4001/
MSGRMDS_5001/
MSGRMDS_6001/
MSGRMDS_7001/
MSGRMDS_7101/
MSGRMDS_7201/
MSGRMDS_7301/
MSGRMDS_8001/

Also, the following 2 directories are very large and I get timeouts due to AWS credential expiry. So , I use aws s3 sync command to copy data time to time (when there are timeouts).due to .

MSGRMDS_2001/ - 1.4 TiB
MSGRMDS_3001/ - 3.3 TiB

[jordanpadams]

📆 02/2024 status: In work. On schedule for completion by end of sprint.