updates to support 1000-node SPS scaling test using the IDPS SRL pipeline

[pymonger]

Purpose

The purpose of this PR is to provide updates to SPS deployment, configuration and DAGs that will allow it to support a large scale test (1000 worker nodes).

Proposed Changes

• [ADD] update tasks in EDRgen, RDRgen and vic2png DAGs with weight_rule and priority_weight so that large scale processing is not bottlenecked by the default priority scheme
• [CHANGE] published EDRgen, RDRgen and vic2png multi-arch docker images to ECR and updated KPOs to pull from it (during large scale testing, pulling images from dockerhub will quickly run into rate limiting error)
• [CHANGE] set retries to 3 for all KPO calls to improve dagRun resiliency
• [CHANGE] change KPO instance types to minimum instance type that can handle c6i.large PGE requirements
• [CHANGE] instantiate KPO instances as spot for cost savings
• [CHANGE] tuned various parameters in airflow/helm/values.tmpl.yaml that were required in order to perform the 1000-node SPS scaling test; the updated values were determined as a result of iterating over the large scale test many times and making these tweaks to resolve various bottlenecks
• [CHANGE] increased DB instance size to handle the traffic generated by the 1000-node scaling test
• [ADD] DAGs for evaluating readiness of the EDRgen, RDRgen, and vic2png jobs
• [ADD] added 2 new DAGs to aid in housecleaning of airflow dagRuns

Issues

#285

Testing

Video of the large scaling test can be viewed here: https://drive.google.com/file/d/1CZ1U0AOUy8bzm9HDWlq6YUVH75InzyEG/view?usp=drive_link.