Doc 302 new etl tutorial - part 1

docs/docs-beta/docs/tutorial/01-etl-tutorial-introduction.md

@@ -0,0 +1,144 @@
+---
+title: Build an ETL Pipeline
+description: Learn how to build an ETL pipeline with Dagster
+last_update:
+  author: Alex Noonan
+---
+
+# Build your first ETL pipeline
+
+In this tutorial, you'll build an ETL pipeline with Dagster that:
+
+1. Imports sales data to DuckDB
+2. Transforms data into reports
+3. Runs scheduled reports automatically
+4. Generates one-time reports on demand
+
+## What you'll learn
+
+- Setting up a Dagster project with the recommended project structure
+- Creating Assets with metadata
+- Using Resources to connect to external systems
+- Building dependencies between assets
+- Running a pipeline by materializing assets
+- Adding schedules, sensors, and partitions to your assets
+- Refactor project into recommended structure
+
+[Add image for what the completed global asset graph looks like]
+
+## Prerequisites
+
+<details>
+  <summary>Prerequisites</summary>
+
+To follow the steps in this guide, you'll need:
+
+- Basic Python knowledge
+- Python 3.9+ installed on your system. Refer to the [Installation guide](/getting-started/installation) for information.
+- Familiarity with SQL or Python data manipulation libraries (Pandas or Polars).
+- Understanding of data pipelines and the extract, transform, and load process.
+</details>
+
+
+## Step 1: Set up your Dagster environment
+
+First, set up a new Dagster project.
+
+1. Open your terminal and create a new directory for your project:
+
+   ```bash
+   mkdir dagster-etl-tutorial
+   cd dagster-etl-tutorial
+   ```
+
+2. Create and activate a virtual environment:
+
+   <Tabs>
+   <TabItem value="macos" label="MacOS">
+   ```bash
+   python -m venv dagster_tutorial
+   source dagster_tutorial/bin/activate
+   ```
+   </TabItem>
+   <TabItem value="windows" label="Windows">
+   ```bash
+   python -m venv dagster_tutorial
+   dagster_tutorial\Scripts\activate
+   ```
+   </TabItem>
+   </Tabs>
+
+3. Install Dagster and the required dependencies:
+
+   ```bash
+   pip install dagster dagster-webserver pandas dagster-duckdb
+   ```
+
+## Step 2: Create the Dagster project structure
+
+Next, you'll create the project directories and files for this tutorial with the `dagster project from-example` command:
+
+   ```bash 
+      dagster project from-example --example getting_started_etl_tutorial
+   ```
+
+Your project should have this structure:
+{/* vale off */}
+```
+dagster-etl-tutorial/
+├── data/
+│   └── products.csv
+│   └── sales_data.csv
+│   └── sales_reps.csv
+│   └── sample_request/
+│       └── request.json
+├── etl_tutorial/
+│   └── definitions.py
+├── pyproject.toml
+├── setup.cfg
+├── setup.py
+```
+{/* vale on */}
+
+:::info
+Dagster has several example projects you can install depending on your use case. To see the full list, run `dagster project list-examples`. For more information on the `dagster project` command, see the [API documentation](https://docs-preview.dagster.io/api/cli#dagster-project).
+::: 
+
+## Dagster Project Structure
+
+In the root directory there are three configuration files that are common in Python package management. These manage dependencies and identifies the Dagster modules in the project. The `etl_tutorial` folder is where our Dagster definition for this code location exists. The data directory is where the raw data for the project is stored and we will reference these files in our software-defined assets.
+### File/Directory Descriptions
+
+#### dagster-etl-tutorial directory 
+
+In the `dagster-etl-tutorial` root directory, there are three configuration files that are common in Python package management. These files manage dependencies and identify the Dagster modules in the project.
+| File | Purpose |
+|------|---------|
+| pyproject.toml | This file is used to specify build system requirements and package metadata for Python projects. It is part of the Python packaging ecosystem. |
+| setup.cfg | This file is used for configuration of your Python package. It can include metadata about the package, dependencies, and other configuration options. |
+| setup.py | This script is used to build and distribute your Python package. It is a standard file in Python projects for specifying package details. |
+
+#### etl_tutorial directory
+
+main directory where you will define your assets, jobs, schedules, sensors, and resources.
+| File | Purpose |
+|------|---------|
+| definitions.py | This file is typically used to define jobs, schedules, and sensors. It organizes the various components of your Dagster project. This allows Dagster to load the definitions in a module. |
+
+#### data directory
+
+The data directory contains the raw data files for the project. We will reference these files in our software-defined assets in the next step of the tutorial.
+
+## Launch Dagster
+
+Start the Dagster webserver from your project's root directory. If you are not in the project root directory navigate there now.
+
+  ```bash
+    cd getting_started_etl_tutorial
+  ```
+
+Run the `dagster dev` command. Dagster should open up in your browser.
+
+## Next steps
+
+- Continue this tutorial with [create and materialize assets](/tutorial/02-create-and-materialize-assets)

docs/docs-beta/docs/tutorial/02-create-and-materialize-assets.md

@@ -0,0 +1,95 @@
+---
+title: Create and materialize assets
+description: Load project data and create and materialize assets
+last_update:
+  author: Alex Noonan
+---
+
+# Create and materialize assets
+
+In the first step of the tutorial, you created your Dagster project with the raw data files. In this step, you will:
+- Create your initial definitions object
+- Add a DuckDB resource
+- Build software-defined assets
+- Materialize your assets
+
+## 1. Create a Definitions object
+
+In Dagster, the [Definitions](/api/definitions) object is where you define and organize various components within your project, such as assets and resources.
+
+## 2. Define the DuckDB resource
+
+In Dagster,[Resources](/api/resources) are external services, tool, and storage necessary to do your job. In this project, we'll use [DuckDB](https://duckdb.org/) - a fast, in-process SQL database that runs inside your application - for storage. We'll define it once in the definitions object, making it available to all assets and objects that need it.
+
+Open the `definitions.py` file in the `etl_tutorial` directory and copy the following code into it:
+
+  ```python
+  import json
+  import os
+
+  from dagster_duckdb import DuckDBResource
+
+  import dagster as dg
+
+  defs = dg.Definitions(
+    assets=[],
+    resources={},
+  )
+  ```
+
+## 3. Create Assets
+
+Software defined [assets](/api/assets) are the main building block in Dagster. An asset is composed of the asset key which is how its identified, a op which is a function that is invoked to produce the asset and upstream dependencies that the asset depends on. You can read more about our philosophy behind the [asset centric approach](https://dagster.io/blog/software-defined-assets).
+
+### Products asset
+
+First, we will create an asset that creates a DuckDB table to hold data from the products CSV. This asset takes the `duckdb` resource defined earlier and returns a `MaterializeResult` object.
+Additionally, this asset contains metadata in the `@dg.asset` decorator parameters to help categorize the asset, and in the `return` block to give us a preview of the asset in the Dagster UI.
+To create this asset, open the `definitions.py `file and copy the following code into it:
+
+<CodeExample filePath="guides/tutorials/etl_tutorial/etl_tutorial/definitions.py" language="python" lineStart="8" lineEnd="33"/>
+
+### Sales Reps Asset
+
+The code for the sales reps asset is similar to the product asset code. In the `definitions.py` file, add the following code below the product asset code:
+
+<CodeExample filePath="guides/tutorials/etl_tutorial/etl_tutorial/definitions.py" language="python" lineStart="35" lineEnd="61"/>
+
+### Sales Data Asset
+
+To add the sales data asset, copy the following code into your `definitions.py` file below the product repos asset:
+
+<CodeExample filePath="guides/tutorials/etl_tutorial/etl_tutorial/definitions.py" language="python" lineStart="62" lineEnd="87"/>
+
+## 4. Add assets to the Definitions object
+
+Now to pull these assets into our Definitions object, add them to the empty list in the assets parameter.
+
+  ```python
+  defs = dg.Definitions(
+      assets=[products,
+          sales_reps,
+          sales_data,
+      ],
+      resources={"duckdb": DuckDBResource(database="data/mydb.duckdb")},
+  )
+  ```
+
+## 5. Materialize Assets
+
+To materialize your assets:
+1. In a browser, navigate to the URL of the Dagster server that we started earlier. 
+2. Navigate to **Deployment**
+3. Reload Definitions 
+4. Click **Assets**, then click "View global asset lineage" to see all of your assets.
+
+   ![2048 resolution](/images/tutorial/etl-tutorial/etl-tutorial-first-asset-lineage.png)
+
+Click materialize all. Navigate to the runs tab and select the most recent run. Here you can see the logs from the run.
+
+   ![2048 resolution](/images/tutorial/etl-tutorial/first-asset-run.png)
+
+
+## Next steps
+
+- Continue this tutorial with your [Asset Dependencies and Checks](/tutorial/03-creating-a-downstream-asset)

docs/docs-beta/docs/tutorial/03-creating-a-downstream-asset.md

@@ -0,0 +1,46 @@
+---
+title: Creating a Downstream Asset
+description: Reference Assets as dependencies to other assets
+last_update:
+  author: Alex Noonan
+---
+
+# Asset Dependencies
+
+Now that we have the raw data loaded into DuckDB, we need to create a [downstream asset](guides/asset-dependencies.md) that combines the staging assets together. In this step, you will:
+
+- Create a downstream asset
+- Materialize that asset
+
+## Creating a Downstream asset
+
+Now that we have all of our raw data loaded and staged into DuckDB our next step is to merge it together in a .
+
+The data structure that of a fact table (sales data) with 2 dimensions off of it (sales reps and products). To accomplish that in SQL we will bring in our `sales_data` table and then left join on `sales_reps` and `products` on their respective id columns. Additionally, we will keep this view concise and only have relevant columns for analysis.
+
+<CodeExample filePath="guides/tutorials/etl_tutorial/etl_tutorial/definitions.py" language="python" lineStart="89" lineEnd="132"/>
+
+As you can see here this asset looks a lot like our previous ones with a few small changes. We put this asset into a different group. To make this asset dependent on the raw tables we add the asset keys the `deps` parameter in the asset definition.
+
+## Materialize the Asset
+
+We need to add the Asset we just made to the Definitions object.
+
+Your Definitions object should now look like this:
+
+  ```python
+  defs = dg.Definitions(
+    assets=[products,
+        sales_reps,
+        sales_data,
+        joined_data,
+    ],
+    resources={"duckdb": DuckDBResource(database="data/mydb.duckdb")},
+  )
+  ```
+
+Go back into the UI, reload definitions, and materialize the `joined_data` asset.
+
+## Next steps
+
+- Continue this tutorial with [Asset Checks](/tutorial/04-ensuring-data-quality-with-asset-checks)

docs/docs-beta/docs/tutorial/05-partitions.md

@@ -0,0 +1,10 @@
+---
+title: Partitions
+description: Partitioning Assets by datetime and categories
+last_update:
+  date: 2024-10-16
+  author: Alex Noonan
+---
+
+
+

docs/docs-beta/sidebars.ts

@@ -11,7 +11,11 @@ const sidebars: SidebarsConfig = {
       type: 'category',
       label: 'Tutorial',
       collapsed: false,
-      items: ['tutorial/tutorial-etl'],
+      items: [
+        'tutorial/etl-tutorial-introduction',
+        'tutorial/create-and-materialize-assets',
+        'tutorial/creating-a-downstream-asset',
+      ],
     },
     {
       type: 'category',

docs/docs-beta/src/theme/MDXComponents.tsx

@@ -1,6 +1,6 @@
 // Import the original mapper
 import MDXComponents from '@theme-original/MDXComponents';
-import { PyObject } from '../components/PyObject';
+import {PyObject} from '../components/PyObject';
 import CodeExample from '../components/CodeExample';
 import Tabs from '@theme/Tabs';
 import TabItem from '@theme/TabItem';