Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extend dag generator to inject env for jobs #32

Open
pandemicsyn opened this issue Sep 20, 2022 · 2 comments · May be fixed by #36
Open

Extend dag generator to inject env for jobs #32

pandemicsyn opened this issue Sep 20, 2022 · 2 comments · May be fixed by #36

Comments

@pandemicsyn
Copy link

pandemicsyn commented Sep 20, 2022

When invoked a legacy ELT schedule, the schedule is executed via meltano schedule run which takes care of populating env if one is specified on the schedule, but job's are executed directly via meltano run and have no idea that they're being executed as part of a schedule.

We could (should?) update the dag generator to inject the env when calling meltano run. Should be as simple as just extending the env and adding the schedule env dict to the BashOperators env:

task = BashOperator(
    task_id=task_id,
    bash_command=f"cd {PROJECT_ROOT}; {MELTANO_BIN} run {run_args}",
    dag=dag,
    env=schedule.get("env", {}),   # snag an env from the schedule
    append_env=True, # append , don't replace the inherited env
) 

slack ref: https://meltano.slack.com/archives/C01TCRBBJD7/p1663679958598429

@pandemicsyn
Copy link
Author

@tayloramurphy weights a 1

@tayloramurphy
Copy link

@pandemicsyn thanks!

@aaronsteers added to the engineering assignments board. This is a quick fix that closes a big gap on expectations around schedules between run vs elt.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants