How we can configure Multiple workflow in same Git Repo

[Riki-1mg]

I am exploring Bodywork for Last few days.
We have multiple models running in Production. I want to configure multiple model workflow in same git repo without changing branch. Can you please help me ?

Use Case:
Suppose we have 2 micro service A and B, Service A contain 4 ML models which and all 2 models follow DAG: train_model -> serve_model. and same for B service (2 Models and follow same work flow). I want to deploy these in a single repo in a single branch, so that I can automate from any CD pipeline (Jenkins)

Expected Directory Structure

ml_model_pipeline_service/
|-- service_a/
|-- model_1/
|-- stage-1-train-model/
|-- train_model.py
|-- requirements.txt
|-- config.ini
|-- stage-2-serve-model/
|-- serve_model.py
|-- requirements.txt
|-- config.ini
|-- model_2/
|-- stage-1-train-model/
|-- train_model.py
|-- requirements.txt
|-- config.ini
|-- stage-2-serve-model/
|-- serve_model.py
|-- requirements.txt
|-- config.ini
|-- service_b/
|-- model_1/
|-- stage-1-train-model/
|-- train_model.py
|-- requirements.txt
|-- config.ini
|-- stage-2-serve-model/
|-- serve_model.py
|-- requirements.txt
|-- config.ini
|-- model_2/
|-- stage-1-train-model/
|-- train_model.py
|-- requirements.txt
|-- config.ini
|-- stage-2-serve-model/
|-- serve_model.py
|-- requirements.txt
|-- config.ini
|-- bodywork.ini

how I can achieve this ?

[AlexIoannides]

Starting with your proposed directory structure, I would re-arrange it as follows:

ml_model_pipeline_service/
    |-- service-a-model-1-stage-1-train-model/
        |-- train_model.py
        |-- requirements.txt
        |-- config.ini
    |-- service-a-model-1-stage-2-serve-model/
        |-- serve_model.py
        |-- requirements.txt
        |-- config.ini
    |-- service-a-model-2-stage-1-train-model/
        |-- train_model.py
        |-- requirements.txt
        |-- config.ini
    |-- service-a-model-2-stage-2-serve-model/
        |-- serve_model.py
        |-- requirements.txt
        |-- config.ini
    |-- service-b-model-1-stage-1-train-model/
        |-- train_model.py
        |-- requirements.txt
        |-- config.ini
    |-- service-b-model-1-stage-2-serve-model/
        |-- serve_model.py
        |-- requirements.txt
        |-- config.ini
    |-- service-b-model-2-stage-1-train-model/
        |-- train_model.py
        |-- requirements.txt
        |-- config.ini
    |-- service-b-model-2-stage-2-serve-model/
        |-- serve_model.py
        |-- requirements.txt
        |-- config.ini
    |-- bodywork.ini

Then configure the DAG in bodywork.ini to one of the following:

DAG=service-a-model-1-stage-1-train-model, service-a-model-2-stage-1-train-model >> service-a-model-1-stage-2-serve-model, service-a-model-2-stage-2-serve-model >> service-b-model-1-stage-1-train-model, service-b-model-2-stage-1-train-model >> service-b-model-1-stage-2-serve-model, service-b-model-2-stage-2-serve-model

If you want to execute 'train-and-serve' for Service A and then execute 'train-and-serve' for Service B. Alternatively, you could use,

DAG=service-a-model-1-stage-1-train-model, service-a-model-2-stage-1-train-model, service-b-model-1-stage-1-train-model, service-b-model-2-stage-1-train-model >> service-a-model-1-stage-2-serve-model, service-a-model-2-stage-2-serve-model, service-b-model-1-stage-2-serve-model, service-b-model-2-stage-2-serve-model

Which will train all models, for both services, concurrently, and then deploy all services concurrently. More info on DAG configuration here and here.

Note, that you appear to have two -serve-model stages for each service - one for model-1 and one for model-2. This will result in the deployment of two microservices - one for model-1 and model-2. So 4 microservices overall.

An alternative, would be the following project structure,

ml_model_pipeline_service/
    |-- service-a-stage-1-train-models/
        |-- train_models.py
        |-- requirements.txt
        |-- config.ini
    |-- service-a-stage-2-serve-models/
        |-- serve_models.py
        |-- requirements.txt
        |-- config.ini
    |-- service-b-stage-1-train-models/
        |-- train_models.py
        |-- requirements.txt
        |-- config.ini
    |-- service-b-stage-2-serve-models/
        |-- serve_models.py
        |-- requirements.txt
        |-- config.ini
    |-- bodywork.ini

Where:

• service-a-stage-1-train-models/train_models.py will train both model-1 and model-2 for service-a
• service-a-stage-2-serve-models/serve_models.py will serve both models from the same microservice (i.e. with a different endpoint for each one), for service-a
• service-b-stage-1-train-models/train_models.py will train both model-1 and model-2 for service-b
• service-b-stage-2-serve-models/serve_models.py will serve both models from the same microservice (i.e. with a different endpoint for each one), for service-b

Then, the DAG could be,

DAG=service-a-stage-1-train-models >> service-a-stage-2-serve-models >> service-b-stage-1-train-models >> service-b-stage-2-serve-models

Or,

DAG=service-a-stage-1-train-models, service-b-stage-1-train-models >> service-a-stage-2-serve-models, service-b-stage-2-serve-models

Following the same reasoning as before.