Enable plugins folder

[LucaCinquini]

Purpose

• This PR creates a "plugins/" folder that will be used to store code and utilities common to all DAGs

Proposed Changes

• [ADD] "plugins/" folder to the Airflow custom containers under /opt/airflow/plugins

Issues

• Enable Airflow "plugins" folder #96

Testing

• Tested the execution of the "karpenter_test" DAG when using the "unity_sps_utils" library loaded from the plugins folder.

[drewm-swe]

Thanks Luca. I'm seeing a failure in the unit tests during dag validation:

ModuleNotFoundError: No module named 'unity_sps_utils'

I'm wondering if we should try to follow this technique for incorporating additional sps python modules in our dags:
https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/modules_management.html#modules-management

[LucaCinquini]

That module should be part of the "plugins/" folder. Which version of the Docker image is used by the unit tests? It will probably need to be rebuilt.

[drewm-swe]

The unit tests do not utilize a docker image, they are simply static tests.

If we were to use a dag directory structure like this, it seems like we could still utilize our static unit tests:

[LucaCinquini]

We could... Let me see if I can make the changes.

[LucaCinquini]

But it does becomes messy with the "dags/" and "dags-catalog/" folder because we would need to copy selected portions of that "dags/" folder to "/opt/airflow/dags", and not others.

[drewm-swe]

Another option could be to have the custom dag utils/plugins be a python package which is pip installed in the Dockerfile. For all of the other dependencies in our DAGs, they are made available to pytest through pip install as part of our pyproject.toml:
https://github.com/unity-sds/unity-sps/blob/develop/pyproject.toml#L34-L45

Airflow docs about this:
https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/modules_management.html#creating-a-package-in-python