refactor: consolidate config to pyproject.toml only

docs/content/getting-started/create-new-project.mdx

@@ -33,7 +33,7 @@ pip install dagster
 dagster project scaffold --name my-dagster-project
 ```
 
-The `dagster project scaffold` command generates a folder structure with a single Dagster code location and other files, such as `pyproject.toml` and `setup.py`. This takes care of setting things up with an empty project, enabling you to quickly get started.
+The `dagster project scaffold` command generates a folder structure with a single Dagster code location and the canonical Python configuration file `pyproject.toml`. This empty project enables you to quickly get started.
 
 </TabItem>
 <TabItem name="Official example">

docs/content/guides/dagster/recommended-project-structure.mdx

@@ -162,7 +162,7 @@ We recommend defining ops and graphs a job file along with the job definition wi
 
 ## For multiple code locations
 
-So far, we've discussed our recommendations for structuring a large project which contains only one code location. Dagster also allows you to structure a project with multiple definitions. We don't recommend over-abstracting too early; in most cases, one code location should be sufficient. A helpful pattern uses multiple code locations to separate conflicting dependencies, where each definition has its own package requirements (e.g., `setup.py`) and deployment specs (e.g., Dockerfile).
+So far, we've discussed our recommendations for structuring a large project which contains only one code location. Dagster also allows you to structure a project with multiple definitions. We don't recommend over-abstracting too early; in most cases, one code location should be sufficient. A helpful pattern uses multiple code locations to separate conflicting dependencies, where each definition has its own package requirements (e.g., `pyproject.toml`) and deployment specs (e.g., Dockerfile).
 
 To include multiple code locations in a single project, you'll need to add a [configuration file](/guides/understanding-dagster-project-files#configuration-files) to your project:
 

docs/content/guides/understanding-dagster-project-files.mdx

@@ -27,8 +27,6 @@ The following demonstrates a Dagster project using the default project skeleton,
 │   └──  definitions.py
 ├── my_dagster_project_tests
 ├── pyproject.toml
-├── setup.cfg
-├── setup.py
 └── tox.ini
 ```
 
@@ -133,8 +131,8 @@ Let's take a look at what each of these files and directories does:
     <tr>
       <td>pyproject.toml</td>
       <td>
-        A file that specifies package core metadata in a static, tool-agnostic
-        way.
+        A file that specifies package configuration in a static, tool-agnostic
+        format.
         <br />
         <br />
         This file includes a <code>tool.dagster</code> section which references the
@@ -150,16 +148,10 @@ Let's take a look at what each of these files and directories does:
         <a href="https://peps.python.org/pep-0518/">
           PEP-518
         </a>{" "}
-        and meant to replace <code>setup.py</code>, but we may still include a{" "}
-        <code>setup.py</code> for compatibility with tools that do not use this
+        and meant to replace <code>setup.py</code>, but we may still include {" "}
+        <code>setup.cfg</code> and {" "}
+        <code>setup.py</code> files for compatibility with tools that do not use this
         spec.
-      </td>
-    </tr>
-    <tr>
-      <td>setup.py</td>
-      <td>
-        A build script with Python package dependencies for your new project as
-        a package. Use this file to specify dependencies.
         <br /><br />
         <strong>Note</strong>: If using Dagster+, add <code>dagster-cloud</code> as a dependency.
       </td>
@@ -171,6 +163,13 @@ Let's take a look at what each of these files and directories does:
         commands.
       </td>
     </tr>
+    <tr>
+      <td>setup.py</td>
+      <td>
+        A optional build script for your new project that specifies Python package dependencies
+        and project setup. Some developers and tooling may work best with this file.
+      </td>
+    </tr>
   </tbody>
 </table>
 
@@ -303,8 +302,6 @@ For local development, a project with a single code location might look like thi
 ├── my_dagster_project_tests
 ├── dagster.yaml      ## optional, used for instance settings
 ├── pyproject.toml    ## optional, used to define the project as a module
-├── setup.cfg
-├── setup.py
 └── tox.ini
 ```
 
@@ -325,8 +322,6 @@ For local development, a project with multiple code locations might look like th
 ├── my_dagster_project_tests
 ├── dagster.yaml      ## optional, used for instance settings
 ├── pyproject.toml
-├── setup.cfg
-├── setup.py
 ├── tox.ini
 └── workspace.yaml    ## defines multiple code locations
 ```
@@ -352,8 +347,6 @@ A Dagster project deployed to your infrastructure might look like this:
 ├── my_dagster_project_tests
 ├── dagster.yaml      ## optional, used for instance settings
 ├── pyproject.toml
-├── setup.cfg
-├── setup.py
 ├── tox.ini
 └── workspace.yaml    ## defines multiple code locations
 ```
@@ -380,8 +373,6 @@ For a Dagster+ Serverless deployment, a project might look like this:
 ├── dagster_cloud.yaml         ## defines code locations
 ├── deployment_settings.yaml   ## optional, defines settings for full deployments
 ├── pyproject.toml
-├── setup.cfg
-├── setup.py
 └── tox.ini
 ```
 
@@ -404,8 +395,6 @@ For a Dagster+ Hybrid deployment, a project might look like this:
 ├── dagster_cloud.yaml          ## defines code locations
 ├── deployment_settings.yaml    ## optional, defines settings for full deployments
 ├── pyproject.toml
-├── setup.cfg
-├── setup.py
 └── tox.ini
 ```
 

docs/content/integrations/dbt/quickstart.mdx

@@ -143,8 +143,6 @@ my_dagster_project
 ├── assets.py
 ├── definitions.py
 ├── pyproject.toml
-├── setup.cfg
-└── setup.py
 ```
 
 1. Change directories to the Dagster project directory:

docs/docs-beta/docs/guides/project-structure.md

@@ -29,8 +29,7 @@ $ dagster project scaffold --name example-dagster-project
 │   ├── __init__.py
 │   └── test_assets.py
 ├── pyproject.toml
-├── setup.cfg
-└── setup.py
+
 ```
 
 This is a great structure as you are first getting started, however, as you begin to introduce more assets, jobs, resources, sensors, and utility code, you may find that your Python files are growing too large to manage.

examples/assets_dbt_python/pyproject.toml

@@ -1,7 +1,37 @@
+[project]
+name = "assets_dbt_python"
+version = "0.1.0"
+description = "Example Dagster project demonstrating dbt."
+readme = "README.md"
+requires-python = ">=3.8,<3.13"
+dependencies = [
+    "boto3",
+    "dagster",
+    "dagster-cloud",
+    "dagster-dbt",
+    "dagster-duckdb",
+    "dagster-duckdb-pandas",
+    "dbt-core",
+    "dbt-duckdb",
+    "numpy",
+    "packaging<22.0", # packaging v22 has build compatibility issues with dbt as of 2022-12-07
+    "pandas",
+    "scipy",
+]
+
+[project.optional-dependencies]
+dev = ["dagster-webserver", "pytest"]
+
 [build-system]
 requires = ["setuptools"]
 build-backend = "setuptools.build_meta"
 
 [tool.dagster]
 module_name = "assets_dbt_python.definitions"
 code_location_name = "assets_dbt_python"
+
+[tool.setuptools.packages.find]
+exclude = ["assets_dbt_python_tests"]
+
+[tool.setuptools.package-data]
+assets_dbt_python = ["dbt-project/**/*"]

examples/assets_dynamic_partitions/pyproject.toml

@@ -1,7 +1,26 @@
+[project]
+name = "assets_dynamic_partitions"
+version = "0.1.0"
+description = "Example Dagster project demonstrating dynamic partitions."
+readme = "README.md"
+requires-python = ">=3.8,<3.13"
+dependencies = [
+    "dagster",
+    "dagster-cloud",
+    "dagster-duckdb-pandas",
+    "requests",
+]
+
+[project.optional-dependencies]
+dev = ["dagster-webserver", "pytest"]
+
 [build-system]
 requires = ["setuptools"]
 build-backend = "setuptools.build_meta"
 
 [tool.dagster]
 module_name = "assets_dynamic_partitions.definitions"
 code_location_name = "assets_dynamic_partitions"
+
+[tool.setuptools.packages.find]
+exclude = ["assets_dynamic_partitions_tests"]

examples/assets_modern_data_stack/pyproject.toml

@@ -1,7 +1,43 @@
+[project]
+name = "assets_modern_data_stack"
+version = "0.1.0"
+description = "Example Dagster project demonstrating modern data stack."
+readme = "README.md"
+requires-python = ">=3.8,<3.13"
+dependencies = [
+    "boto3",
+    "dagster",
+    "dagster-airbyte",
+    "dagster-cloud",
+    "dagster-dbt",
+    "dagster-postgres",
+    "dbt-core",
+    "dbt-postgres",
+    "numpy",
+    "pandas",
+    "scipy",
+    "packaging<22.0",  # match dbt-core's requirement to workaround a resolution issue
+]
+
+[project.optional-dependencies]
+dev = ["dagster-webserver", "pytest"]
+test = [
+    # cant build psycopg2 in buildkite
+    # something about the 1.8.0 dependency setup to avoid psycopg2-binary on linux
+    # seems to prevent that dependency from being used even if explicitly added
+    "dbt-postgres<1.8.0"
+]
+
 [build-system]
 requires = ["setuptools"]
 build-backend = "setuptools.build_meta"
 
 [tool.dagster]
 module_name = "assets_modern_data_stack.definitions"
 code_location_name = "assets_modern_data_stack"
+
+[tool.setuptools.packages.find]
+exclude = ["assets_modern_data_stack_tests"]
+
+[tool.setuptools.package-data]
+assets_modern_data_stack = ["../dbt_project/*", "../dbt_project/*/*"]

examples/assets_pandas_pyspark/pyproject.toml

@@ -1,7 +1,27 @@
+[project]
+name = "assets_pandas_pyspark"
+version = "0.1.0"
+description = "Example Dagster project demonstrating pandas and pyspark."
+readme = "README.md"
+requires-python = ">=3.8,<3.13"
+dependencies = [
+    "dagster",
+    "pandas",
+    "pyspark",
+    # "pyarrow",
+]
+
+[project.optional-dependencies]
+dev = ["dagster-webserver", "pytest"]
+test = ["pandas", "pyarrow; python_version < '3.9'", "pyspark"]
+
 [build-system]
 requires = ["setuptools"]
 build-backend = "setuptools.build_meta"
 
 [tool.dagster]
 module_name = "assets_pandas_pyspark.definitions"
 code_location_name = "assets_pandas_pyspark"
+
+[tool.setuptools.packages.find]
+exclude = ["assets_pandas_pyspark_tests"]