[New Feature] Create DAG to execute the Application Package generator

[LucaCinquini]

PM Ticket Reference: unity-sds/unity-app-build-trigger#2

Description:

Convert App Builds system into a docker container runnable in Airflow - Allows user to provide tokens, credentials. See Status of builds. Etc. Initial version should be what we're doing today. enhancements (future) should include pointing to an eCR repository, non-docker repo, etc. Needs Documentation. Should this be its own DAG that can run 'faster' than the cwl operators (e.g. SPOT instances).

Acceptance Criteria:

Docker Container Conversion: The App Builds system must be successfully converted into a Docker container.
Airflow Integration: Ensure the Docker container runs smoothly within the Airflow environment.
Validation and Testing: The Docker container and its integration with Airflow must be thoroughly validated and tested to ensure it operates without errors.
Documentation: Provide detailed documentation on the app-pack process, including steps taken, and configurations applied

Dependencies:
From U-ADS: Docker container to execute the Application Package Generation, together with complete documentation.

[LucaCinquini]

Goal (restated): follow James’s manual instructions to create a Docker container and DAG that will replace the automatic build system on GitLab (invoked via the Unity-py API)

As discussed in meeting with @mcduffie , the following steps will be required:

• Preliminary: publish the repos: “unity-app-generator” and “unity-pack-generator” to PyPI
• Build a Docker container that executes the following steps, run manually on laptop
  • build_ogc_app init
  • build_ogc_app build_docker --no_owner
  • build_ogc_app push_docker to GitHub
• Build a DAG to execute that Docker container via UI
• Update the current Lambda to invoke the DAG. The Lambda is invoked via the existing Unity API.
• Future optional: add a Github Action to push automatically push any changes to the “unity-app-generator” and “unity-pack-generator” repositories to PyPI

References:

• How to build the app-package generator manually: https://unity-sds.gitbook.io/docs/mdps-overview/tutorials/algorithms-and-algorithm-development/packaging-an-algorithm
• How to invoke the current automatic build system: https://github.com/unity-sds/mdps-tutorial/blob/main/jupyter-notebooks/tutorials/5_building_application_packages.ipynb
• Example slide: https://docs.google.com/presentation/d/1CrUsnoKpCkQTECEjHZz4ysduBfrkDH4APcx4CmtYJXM/edit#slide=id.g3092c7e18ae_0_42