Deploy SDAP with COG support to Unity-Venue-Dev account

[jjacob7734]

Deploy the development branch of SDAP that supports Cloud Optimized GeoTIFF (COG) as the data store to the Unity-Venue-Dev project account.

Acceptance Criteria:

• SDAP with COG support deployed to Unity-Venue-Dev
• Ingest test data into SDAP
• SDAP time series endpoint tested with test data

[jjacob7734]

The SDAP branch with CoG support is actually in this fork by Riley Kuttruff: https://github.com/RKuttruff/incubator-sdap-nexus/tree/SDAP-484-CoG. That branch is not thoroughly tested and has not been merged into the main SDAP branch. Riley's development branch includes support for:

1. SDAP's normal datastore based on Solr and Cassandra
2. CoG
3. Zarr

Riley has Docker images tagged with SDAP-484-CoG that can be used to deploy his variant of SDAP:

1. rileykkjpl/sdap-nexus-webapp
2. rileykkjpl/sdap-collection-manager
3. rileykkjpl/sdap-granule-ingester

[GodwinShen]

@jjacob7734 can we please get an update on this ticket?

[GodwinShen]

@jjacob7734 ping again, please provide an update on this ticket. It was originally planned for 24.1.2 and the end of 24.1.3 is quickly approaching.

[GodwinShen]

From slack chat with @jjacob7734:

The SDAP deployment isn't done yet. I did get the development fork of SDAP that has the CoG support we need. I need to try deploying it. That is a prerequisite to the SBG L2 ingest (#16)

[GodwinShen]

@jjacob7734 can we get a quick update on this ticket?

[jjacob7734]

@jjacob7734 can we get a quick update on this ticket?

I am adapting our Terraform for process mapper deployment to provision my new cluster automatically. I can do a manual provisioning using U-CS tools instead so I can proceed.

[jjacob7734]

• Working with Galen (U-CS) on process for provisioning new Unity EKS cluster.

[jjacob7734]

Status Update

• Working on Terraform automation of the SDAP deployment to MCP/EKS (see https://github.com/orgs/unity-sds/projects/3/views/17?pane=issue&itemId=30130124 for details)

[jjacob7734]

• Can provision and destroy the EKS cluster for SDAP on demand in Unity-Venue-Dev.
• Created terraform.tfvars to set up Kubernetes nodegroups in the same minimal configuration that I have tested on Unity-Dev. That configuration is 1 nodegroup with 1 m5.xlarge instance as orchestrator/webapp and 1 nodegroup with 1 memory-optimized r5.2xlarge instance (scalable up to 3).
• Added required Unity tags to the terraform deployment.

Working on:

• Provision a small EC2 bastion host with kubectl, helm, and eksctl installed to in order to script (fully automate) the remaining steps
• Automate deployment of the SparkOperator (SDAP's backend parallelization platform is Apache Spark).
• Fork the SDAP code to create a deployable version with my helm chart modifications required in order to get SDAP to deploy in my minimal configuration. That is a more orderly and controlled deployment than the default, and I use Kubernetes node taints and selectors to accomplish that.

[jjacob7734]

Moving this issue to the backlog until this dependency is resolved: https://github.com/orgs/unity-sds/projects/3/views/17?pane=issue&itemId=30130124