[docs] Update recommended structure to remove Dagster code from __init__.py
#23316
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This stack of pull requests is managed by Graphite. Learn more about stacking. Join @maximearmstrong and the rest of your teammates on |
maximearmstrong
changed the title
__init__.py__init__.py
maximearmstrong
force-pushed
the
maxime/ds-375/remove-dagster-code-from-init-py-in-docs
branch
from
a6e9d94 to
44b1d45
Compare
This was referenced Jul 31, 2024
maximearmstrong
force-pushed
the
maxime/ds-375/remove-dagster-code-from-init-py-in-docs
branch
from
44b1d45 to
c9cd6e3
Compare
maximearmstrong
force-pushed
the
maxime/ds-375/remove-dagster-code-from-init-py-in-docs
branch
from
5485c9d to
009d0bb
Compare
maximearmstrong
commented
Jul 31, 2024
| @@ -131,7 +132,7 @@ In this example, we grouped resources (e.g., database connections, Spark session | |||
|
|
|||
| In complex projects, we find it helpful to make resources reusable and configured with pre-defined values via <PyObject object="configured" />. This approach allows your teammates to use a pre-defined resource set or make changes to shared resources, thus enabling more efficient project development. | |||
|
|
|||
| This pattern also helps you easily execute jobs in different environments without code changes. In this example, we dynamically defined a code location based on the deployment in [`__init__.py`](https://github.com/dagster-io/dagster/blob/master/examples/project_fully_featured/project_fully_featured/\__init\_\_.py) and can keep all code the same across testing, local development, staging, and production. Read more about our recommendations in the [Transitioning data pipelines from Development to Production](/guides/dagster/transitioning-data-pipelines-from-development-to-production) guide. | |||
| This pattern also helps you easily execute jobs in different environments without code changes. In this example, we dynamically defined a code location based on the deployment in [`definitions.py`](https://github.com/dagster-io/dagster/blob/master/examples/project_fully_featured/project_fully_featured/\__init\_\_.py) and can keep all code the same across testing, local development, staging, and production. Read more about our recommendations in the [Transitioning data pipelines from Development to Production](/guides/dagster/transitioning-data-pipelines-from-development-to-production) guide. | |||
There was a problem hiding this comment.
@maximearmstrong linked PR has been approved if you want to merge / update URL.
maximearmstrong
force-pushed
the
maxime/ds-375/remove-dagster-code-from-init-py-in-docs
branch
2 times, most recently
from
fbf7d39 to
df8e566
Compare
| @@ -20,20 +20,20 @@ dagster dev -f my_file.py -f my_second_file.py | |||
| </TabItem> | |||
| <TabItem name="From a module"> | |||
|
|
|||
| Dagster can also load Python modules as [code locations](/concepts/code-locations). When this approach is used, Dagster loads the definitions defined at the top-level of the module, in a variable containing the <PyObject object="Definitions" /> object of its root `__init__.py` file. As this style of development eliminates an entire class of Python import errors, we strongly recommend it for Dagster projects deployed to production. | |||
| Dagster can also load Python modules as [code locations](/concepts/code-locations). When this approach is used, Dagster loads the definitions defined in the module passed to the command line. It is recommended to define a variable containing the <PyObject object="Definitions" /> object in a submodule named `definitions` inside the Python module. In practice, the submodule can be created by adding a file named `definitions.py` at the root level of the Python module. As this style of development eliminates an entire class of Python import errors, we strongly recommend it for Dagster projects deployed to production. | |||
There was a problem hiding this comment.
Suggested change
| Dagster can also load Python modules as [code locations](/concepts/code-locations). When this approach is used, Dagster loads the definitions defined in the module passed to the command line. It is recommended to define a variable containing the <PyObject object="Definitions" /> object in a submodule named `definitions` inside the Python module. In practice, the submodule can be created by adding a file named `definitions.py` at the root level of the Python module. As this style of development eliminates an entire class of Python import errors, we strongly recommend it for Dagster projects deployed to production. | |
| Dagster can also load Python modules as [code locations](/concepts/code-locations). When this approach is used, Dagster loads the definitions defined in the module passed to the command line. | |
| We recommend defining a variable containing the <PyObject object="Definitions" /> object in a submodule named `definitions` inside the Python module. In practice, the submodule can be created by adding a file named `definitions.py` at the root level of the Python module. | |
| As this style of development eliminates an entire class of Python import errors, we strongly recommend it for Dagster projects deployed to production. |
Let's break this up a bit - the paragraph is pretty visually dense and there's a lot to take in.
There was a problem hiding this comment.
Updated in 32844cf
maximearmstrong
force-pushed
the
maxime/ds-375/remove-dagster-code-from-init-py-in-docs
branch
from
df8e566 to
32844cf
Compare
cmpadden
approved these changes
Aug 13, 2024
maximearmstrong
deleted the
maxime/ds-375/remove-dagster-code-from-init-py-in-docs
branch
maximearmstrong
added a commit
that referenced
this pull request
Aug 13, 2024
## Summary & Motivation This PR updates the URL to the file containing Definitions in project-fully-featured. That was moved from #23316 to this PR to avoid failure when running the doc tests - the file must exist before being referenced, and it is updated in #23345. ## How I Tested These Changes BK
PedramNavid
pushed a commit
that referenced
this pull request
Aug 14, 2024
…it__.py` (#23316) ## Summary & Motivation This PR updates the recommended folder structure to include the Definitions object in `definitions.py` instead of `__init__.py`. It also updates related guides and references. Tutorials in the docs are not yet updated. Will open more PRs: - Update the project in examples/ - Update the tutorials in examples/ - Update tutorials in docs - Update Dag Uni content - Update Dag Uni starter projects once we have a PR for Dag Uni content ## How I Tested These Changes make mdx-full-format
PedramNavid
pushed a commit
that referenced
this pull request
Aug 14, 2024
## Summary & Motivation This PR updates the URL to the file containing Definitions in project-fully-featured. That was moved from #23316 to this PR to avoid failure when running the doc tests - the file must exist before being referenced, and it is updated in #23345. ## How I Tested These Changes BK
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary & Motivation
This PR updates the recommended folder structure to include the Definitions object in
definitions.pyinstead of__init__.py. It also updates related guides and references.Tutorials in the docs are not yet updated. Will open more PRs:
How I Tested These Changes
make mdx-full-format