Workflow Manager Utility

[jordanpadams]

💡 Description

Currently if we want to develop a new workflow for Nucleus, we have to complete the following steps.

1. Create and push the docker image to ECR
2. Create an EFS volume to share data/configuration files between docker containers (optional, only if the use case must share data/configuration between tasks)
3. Create an ECS task (This can be done through AWS Console or Terraform)
4. Implement a DAG (workflow) primarily using the Airflow ECS Operator
5. Upload DAGs to S3 bucket

We have to execute above multiple steps because different PDS docker images have different setups. For example, some of the PDS docker images need various environment variables and others need various volumes to be mounted. As a result, we cannot use a common template.

This problem is there regardless of the workflow tool we use (not specific to Airflow). With any workflow tool, we will have to maintain different setups for different docker images (for example, when we used docker compose, we had different settings for each components in docker compose file).

Implementing and uploading the DAG is easy if the PDS components are already available as ECS tasks. If we plan to automatically generate Terraform scripts, then we have to discuss,

1. What are the INPUTS that we get from users to generate these scripts?
2. How should those INPUTS looks like?
3. Should we maintain a catalog of docker components that they have to select from?
4. Should we have a UI to select docker components required for a workflow and after making the the selection a way to generate and execute the scripts?

A Workflow Manager Utility can be solution to address above issues.