Feature/plugin dbt parser

[kiranvasudev]

Ticket: DATA-2081

Enabling dagger to import and use plugins. Given a folder that contains plugins(python modules), Dagger can now import the classes in these modules and enable their usage using jinja in the task configs.

How this is done:

1. Add a new plugin attribute in the dagger_config.yaml:

plugin:
  paths:
    - <path relative to AIRFLOW_HOME>
    - 

2. When above is added, all python classes in all modules are loaded into the Jinja environment using their class name.
3. If you want to use this class using Jinja in the dagger task template, you just need to add {{ <class_name>.<function_name>() }} and it would replace this Jinja function call with the output of the function.

To test out the functionality, there are two ways:

First way

1. Debug the tests in this file -> tests/utilities/test_plugins.py

Second way

1. Modify this test to point to a main module config from airflow-dags and comment out the @skip. It should look like below:

    # @skip("Run only locally")
    def test_generate_task_configs(self):
        module = Module(
            path_to_config="/Users/kiran.vasudev/work/dataeng-airflow-dags/dags/analytics_engineering/pipelines/buyer_product_daily.yaml",
            target_dir="./tests/fixtures/modules/",
        )

        module.generate_task_configs()

3. Set the AIRFLOW_HOME and create a plugin folder with a sample plugin(you can copy this from the tests/fixtures/plugins directory as an example)
4. Run the test(easier from Pycharm i guess) and you should see the task config in the target_dir

Other changes:
Apart from introducing this plugin approach, i also disabled the use of the dbt_config_parser, so for dbt configs that are generated using code from this branch, the inputs and outputs will be null and the dbt_manifest is not available to the jinja environment.

[siklosid]

I don't understand the testing part in the PR description.

[swarmia]

✅  Linked to Task DATA-2081 · Move dbt parser logic away from dagger
➡️  Part of Epic DATA-136 · 📉 DataTechDebt

[claudiazi]

so cooool!!!!