App package deploy and docker localization

[mike-gangl]

App package deploy and docker localization

This feature is to provide the capability to "deploy" an OGC application package to Airflow. Currently, we use a singular DAG (either cwl_dag.py or cwl_modular.py) to run all packages.

By calling a deploy OGC endpoint with a valid request (CWL?), the following steps should be taken:

1. The CWL file is loaded within the OGC API
2. The docker container specified within the App Package is pulled and registered locally within the ADES (e.g. in an ECR or S3 backed store)
3. The CWL file is updated with the newly (locally) registered container.
4. The CWL file is stored in a persistent store (s3, database) and made available to the ADES / Pods
5. (value add steps follow)
6. A new DAG is created to call the k8s operator with the specific, updated CWL (step 4).
7. The input fields are created for the DAG by parsing the inputs to the CWL
8. A user can execute the CWL by using the Airflow UI and providing specific inputs in individualized fields.

[rtapella]

Thanks Mike. In my view steps 6–8 are MVP. The reason is that it is very difficult to separate the tracking of jobs by algorithm if we don't split up the associated Airflow DAGs.

In an operational scenario I imagine the single-DAG would be a huge barrier to acceptance because it impacts anomaly resolution. Unless I'm missing something.

[rtapella]

Also 9. A user can execute the app by using the OGC Processes job-execution API