[daggy-u][dbt] Update dbt course to use DbtProject #23098

maximearmstrong · 2024-07-18T21:01:52Z

Summary & Motivation

This PR updates the dbt course in Dagster University to use DbtProject.

To do outside this PR:

update the knowledge check/quiz in Lesson 4. The DAGSTER_DBT_PARSE_PROJECT_ON_LOAD env var is no longer used.

How I Tested These Changes

maximearmstrong · 2024-07-18T21:02:09Z

[daggy-u][dbt] Update dbt course to use DbtProject #23098 👈
master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @maximearmstrong and the rest of your teammates on Graphite

github-actions · 2024-07-18T21:04:20Z

Deploy preview for dagster-university ready!

✅ Preview
https://dagster-university-4o24647sg-elementl.vercel.app
https://maxime-ds-336-update-dbt-course-to-use-dbtproject.dagster-university.dagster-docs.io

Built with commit 66dd24f.
This pull request is being automatically deployed with vercel-action

github-actions · 2024-07-19T18:22:29Z

Deploy preview for dagster-docs ready!

Preview available at https://dagster-docs-mzucs7np6-elementl.vercel.app
https://maxime-ds-336-update-dbt-course-to-use-dbtproject.dagster.dagster-docs.io

Direct link to changed pages:

cmpadden · 2024-07-29T18:51:15Z

This is looking great.

@maximearmstrong we will also need to update the "completed" dagster-and-dbt project here:

https://github.com/dagster-io/project-dagster-university/tree/module/dagster-and-dbt

cmpadden · 2024-07-30T19:10:07Z

docs/dagster-university/pages/dagster-dbt/lesson-3/3-representing-the-dbt-project-in-dagster.md

+
+```python
+dbt_project = DbtProject(
+  project_dir=Path(__file__).joinpath("..", "..", "..", "analytics").resolve(),


I know we did this in the previous lesson as well, but I'm not a fan of repeating .. relative paths.

Would it be more intuitive if we used dagster home, or DagsterInstance.get().root_directory()? This can merge before this change, but wanted to bring it up.

I understand your point, but I think we should avoid this for two main reasons:

it is common to see a dbt project outside the Dagster project directory - we would need to resolve the path using DagsterInstance.get().root_directory() and more .. to indicate the dbt project location.

the current scaffold template uses .. relative paths. If we want to change that in Dagster University, we should start by the scaffold.

erinkcochran87

Thanks for doing this! A few things, but overall it's looking good.

Let me know when this is ready to go and I'll update the quiz and lesson title in the LMS.

docs/dagster-university/pages/dagster-dbt/lesson-3/3-representing-the-dbt-project-in-dagster.md

erinkcochran87 · 2024-08-02T21:17:37Z

docs/dagster-university/pages/dagster-dbt/lesson-3/3-representing-the-dbt-project-in-dagster.md

+
+As you’ll frequently point your Dagster code to the `target/manifest.json` file and your dbt project in this course, it’ll be helpful to keep a reusable representation of the dbt project. This can be easily done using the `DbtProject` class.
+
+Create a new file in the `assets` directory called `dbt.py`. Open that file and add the following import at the top:


Suggested change

Create a new file in the `assets` directory called `dbt.py`. Open that file and add the following import at the top:

In the `assets` directory, create a new `dbt.py` file and add the following imports:

This is what I see now:

In the dagster_university directory, create a new project.py file and add the following imports:

But later on in the lesson, dbt.py is still used (ex: line 31).

Oh good catch! Fixed in 66dd24f

docs/dagster-university/pages/dagster-dbt/lesson-3/3-representing-the-dbt-project-in-dagster.md

...dagster-university/pages/dagster-dbt/lesson-3/5-loading-dbt-models-into-dagster-as-assets.md

docs/dagster-university/pages/dagster-dbt/lesson-4/2-speeding-up-the-development-cycle.md

docs/dagster-university/pages/dagster-dbt/lesson-7/4-creating-the-manifest-during-deployment.md

docs/dagster-university/pages/dagster-dbt/lesson-7/5-preparing-for-a-successful-run.md

maximearmstrong · 2024-08-05T17:26:18Z

Thanks for doing this! A few things, but overall it's looking good.

@erinkcochran87 All comments were addressed in 54fd52f

Let me know when this is ready to go and I'll update the quiz and lesson title in the LMS.

Perfect. We need to update the final example then we are ready to go. Will let you know!

maximearmstrong · 2024-08-05T20:44:45Z

After fully testing the updated final example here, I've updated this PR in commit 305f9ff.

tl;dr: the DbtProject object should be on a file on its own to avoid import issues during development when calling dagster-dbt project prepare-and-package.

cc @cmpadden @erinkcochran87 if you want to review the changes in this commit before we merge.

erinkcochran87

Back to you!

## Summary & Motivation This PR updates the dbt course in Dagster University to use DbtProject. To do outside this PR: - update the knowledge check/quiz in Lesson 4. The `DAGSTER_DBT_PARSE_PROJECT_ON_LOAD` env var is no longer used. ## How I Tested These Changes

## Summary & Motivation This PR updates the dbt course in Dagster University to use DbtProject. To do outside this PR: - update the knowledge check/quiz in Lesson 4. The `DAGSTER_DBT_PARSE_PROJECT_ON_LOAD` env var is no longer used. ## How I Tested These Changes (cherry picked from commit b6156a4)

maximearmstrong self-assigned this Jul 24, 2024

maximearmstrong force-pushed the maxime/ds-336/update-dbt-course-to-use-dbtproject branch from 68dbc27 to 07b7ca3 Compare July 24, 2024 23:00

maximearmstrong marked this pull request as ready for review July 24, 2024 23:03

graphite-app bot added the area: dagster-university Related to Dagster University label Jul 24, 2024

graphite-app bot requested a review from erinkcochran87 July 24, 2024 23:03

maximearmstrong requested review from cmpadden, yuhan, PedramNavid and erinkcochran87 and removed request for erinkcochran87 July 24, 2024 23:03

cmpadden reviewed Jul 30, 2024

View reviewed changes

erinkcochran87 reviewed Aug 2, 2024

View reviewed changes

maximearmstrong force-pushed the maxime/ds-336/update-dbt-course-to-use-dbtproject branch from bf706a6 to 54fd52f Compare August 5, 2024 17:20

maximearmstrong mentioned this pull request Aug 5, 2024

Update dbt course final example to use DbtProject dagster-io/project-dagster-university#32

Merged

maximearmstrong added 8 commits August 5, 2024 16:31

Update code in lessons 5, 6 and 7

7d7a33c

Update lesson 3 to use DbtProject

bbdb9f3

Update lesson 4 part 2

352ef69

Update code in lesson 5 and 6

6148c76

Update lesson 7

69c3d5a

Update lesson 7

541226a

Update dbt source post review

5db629a

Update course after moving DbtProject to project.py

305f9ff

maximearmstrong force-pushed the maxime/ds-336/update-dbt-course-to-use-dbtproject branch from 54fd52f to 305f9ff Compare August 5, 2024 20:31

maximearmstrong requested a review from erinkcochran87 August 5, 2024 20:44

maximearmstrong requested a review from cmpadden August 5, 2024 20:44

erinkcochran87 reviewed Aug 5, 2024

View reviewed changes

Fix file name in lesson 3 part 3

66dd24f

maximearmstrong requested a review from erinkcochran87 August 5, 2024 21:02

erinkcochran87 approved these changes Aug 6, 2024

View reviewed changes

maximearmstrong merged commit b6156a4 into master Aug 7, 2024
2 checks passed

maximearmstrong deleted the maxime/ds-336/update-dbt-course-to-use-dbtproject branch August 7, 2024 01:56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[daggy-u][dbt] Update dbt course to use DbtProject #23098

[daggy-u][dbt] Update dbt course to use DbtProject #23098

maximearmstrong commented Jul 18, 2024 •

edited

Loading

maximearmstrong commented Jul 18, 2024

github-actions bot commented Jul 18, 2024 •

edited

Loading

github-actions bot commented Jul 19, 2024 •

edited

Loading

cmpadden commented Jul 29, 2024

cmpadden Jul 30, 2024

maximearmstrong Aug 1, 2024

erinkcochran87 left a comment

erinkcochran87 Aug 2, 2024

erinkcochran87 Aug 5, 2024

maximearmstrong Aug 5, 2024

maximearmstrong commented Aug 5, 2024

maximearmstrong commented Aug 5, 2024

erinkcochran87 left a comment


		As you’ll frequently point your Dagster code to the `target/manifest.json` file and your dbt project in this course, it’ll be helpful to keep a reusable representation of the dbt project. This can be easily done using the `DbtProject` class.

		Create a new file in the `assets` directory called `dbt.py`. Open that file and add the following import at the top:

	Create a new file in the `assets` directory called `dbt.py`. Open that file and add the following import at the top:
	In the `assets` directory, create a new `dbt.py` file and add the following imports:

[daggy-u][dbt] Update dbt course to use DbtProject #23098

[daggy-u][dbt] Update dbt course to use DbtProject #23098

Conversation

maximearmstrong commented Jul 18, 2024 • edited Loading

Summary & Motivation

How I Tested These Changes

maximearmstrong commented Jul 18, 2024

github-actions bot commented Jul 18, 2024 • edited Loading

github-actions bot commented Jul 19, 2024 • edited Loading

cmpadden commented Jul 29, 2024

cmpadden Jul 30, 2024

Choose a reason for hiding this comment

maximearmstrong Aug 1, 2024

Choose a reason for hiding this comment

erinkcochran87 left a comment

Choose a reason for hiding this comment

erinkcochran87 Aug 2, 2024

Choose a reason for hiding this comment

erinkcochran87 Aug 5, 2024

Choose a reason for hiding this comment

maximearmstrong Aug 5, 2024

Choose a reason for hiding this comment

maximearmstrong commented Aug 5, 2024

maximearmstrong commented Aug 5, 2024

erinkcochran87 left a comment

Choose a reason for hiding this comment

maximearmstrong commented Jul 18, 2024 •

edited

Loading

github-actions bot commented Jul 18, 2024 •

edited

Loading

github-actions bot commented Jul 19, 2024 •

edited

Loading