Docker container to execute the Unity App Generator Script

[mcduffie]

Create a Docker-in-Docker container capable of running the unity-app-generator software.

Currently application packages are built in two manners:

• Manually through direct use of the unity-app-generator
• Automatically using the Application Package Generation API endpoint

This ticket aims to replace the latter. The current approach involves a API Gateway end point that calls a Lambda. That Lambda then triggers a process in the MCP self hosted Gitlab instance. The Gitlab actions there run the unity-app-generator to build a package and place it within Dockerhub and Dockstore.

But that approach is opaque to users of the API endpoint because there is no feedback on success or failure. Due to the licensing nature of Gitlab, only 2 people have direct access to the build logs.

Instead we would like to replace the Gitlab portion of this process with one that uses Airflow for the building of packages using unity-app-generator. This will require in this order:

1. A Docker image that uses Docker-in-Docker to call unity-app-generator:

• build_ogc_app init
• build_ogc_app build_docker
• build_ogc_app push_docker
• build_ogc_app build_cwl
• build_ogc_app push_app_registry

2. An Airflow DAG that calls item 1 - [New Feature]: Create DAG to execute the Application Package generator unity-sps#275
3. Modifications to the existing Application Package Generation Lambda to call item 2 instead of Gitlab

Item 1 will require the ability to pass credentials for push_docker and push_app_registry to the Docker image.

Item 2 will require the ability to either store the credentials in a secret store within Airflow or pass them along from item 3.

Item 3 may or may not pull credentials from the AWS Parameter store.

Since the current way Airflow is exposed allows multiple users to look at jobs, credential obfuscation in logs is an essential element of the task. Why? Because in the future it is conceivable that the Application Package Generation API endpoint could allow users to pass their credentials to direct the storage of Docker images and CWL files.

The two credentials involved are one to a Docker registry and one to an application catalog. In the current configuration the Docker registry is Dockerhub and the application catalog is Dockstore.

[mcduffie]

Follow up task will be: unity-sds/unity-sps#275

[grallewellyn]

Finished item 1 on the above ticket
This Dockerfile runs this shell script as an entrypoint
I got the base code for this from Mike and Luca said this response is acceptable because we normally run our DAGs in privileged mode (which is necessary for this approach because we are running docker in docker)
I ran through this process and confirmed it works
These are the commands I am running to build and run the docker image

docker build --no-cache -t unity-app-gen -f docker/Dockerfile .
docker run --privileged --env GITHUB_REPO=https://github.com/unity-sds/unity-example-application --env DOCKER_USER=<DockerHub username> --env DOCKER_PAT=<DockerHub password> --env DOCKSTORE_TOKEN=<Dockstore token> unity-app-gen

To get the process.cwl generated by the above commands to work with our Modular DAG example and https://raw.githubusercontent.com/unity-sds/unity-example-application/refs/heads/main/test/stage_in/catalog.json, I had to build this on a Linux Machine

The next steps are to find a more secure way to pass the DockerHub password and Dockstore token and also creating the DAG

cc @LucaCinquini

[mcduffie]

In which Git repo do you plan to commit the Dockerfile? unity-app-generator?

[grallewellyn]

I was working in a branch off of unity-app-generator. I thought that repo made the most sense since the entrypoint.sh is putting together the commands outlined in the unity app gen README
https://github.com/unity-sds/unity-app-generator/blob/docker-container-appgen/docker/Dockerfile

[LucaCinquini]

Hi @grallewellyn : great job with completing step 1) and finishing the Dockerfile and the shell script.
In order to use secure information for DOCKER_USER, DOCKER_PAT and DOCKSTORE_TOKEN, one option is the following:

1. Store those values as SSM parameters in the specific venue. You can start with manually entering using the AWX console, but eventually they would need to be added by Terraform when a venue is deployed (@jl-0 : this is probably a task for the Management Console?)

2a) Retrieve those parameters from within the shell script, with a command like:
aws ssm get-parameter --name <parameter_name> --with-decryption --region us-west-2

the use those values as needed. The AWS boto library must be installed inside the Docker container that runs the shell script.

2b) Another option is to retrieve the SSM parameters from within the DAG, and pass them as env variables to the KubernetesPodOperator - see example here: https://github.com/unity-sds/unity-sps/blob/develop/airflow/dags/sbg_preprocess_no_cwl.py
But I am not sure if the values would be echoed in the logs, so I would try the other option first.

[grallewellyn]

I will try the first option!
If we want to store DOCKER_USER, DOCKER_PAT and DOCKSTORE_TOKEN as SSM parameters, how should we get the users to enter them? Through a separate form for their account (i.e. do it once and it persists for your account)? Or via the DAG input parameters on each run?

[LucaCinquini]

Actually @grallewellyn I might have made a mistake... I thought those parameters would be the same for all runs, in which case it makes sense to store them as system-wide SSM parameters. But instead different users will want to push Docker images to different repositories? If that is the case, could you investigate what are the recommended ways to securely pass credentials from the UI (such that they are not echoed in the logs).

[grallewellyn]

Would it be the case that different users want to push Docker images to different repositories? I am not very familiar with our user working group but I think that would make sense as a case
If you agree with this, then I can investigate ways to securely pass credentials from the UI

[LucaCinquini]

Yes, I think we need to support different users using the same SPS installation to push Docker images to different repos. @mcduffie would you agree? Or would a project always push to the same Docker repo?