Doc 302 new etl tutorial - part 1

docs/docs-beta/docs/tutorial/01-etl-tutorial-introduction.md

@@ -0,0 +1,103 @@
+---
+title: Build an ETL Pipeline
+description: Learn how to build an ETL pipeline with Dagster
+last_update:
+  date: 2024-08-10
+  author: Pedram Navid
+---
+
+# Build your first ETL pipeline
+
+Welcome to this hands-on tutorial where you'll learn how to build an ETL pipeline with Dagster while exploring key parts of Dagster.
+
+## What you'll learn
+
+- Setting up a Dagster project with the recommended project structure
+- Creating Assets and using Resources to connect to external systems
+- Adding metadata to your assets
+- Building dependencies between assets
+- Running a pipeline by materializing assets
+- Adding schedules, sensors, and partitions to your assets
+
+[Add image for what the completed global asset graph looks like]
+
+## Step 1: Set up your Dagster environment
+
+First, set up a new Dagster project.
+
+1. Open your terminal and create a new directory for your project:
+
+   ```bash title="Create a new directory"
+   mkdir dagster-etl-tutorial
+   cd dagster-etl-tutorial
+   ```
+
+2. Create a virtual environment and activate it:
+
+   ```bash title="Create a virtual environment"
+   python -m venv dagster_tutorial 
+   source dagster_tutorial/bin/activate
+   # On Windows, use `dagster_tutorial\Scripts\activate`
+   ```
+
+3. Install Dagster and the required dependencies:
+
+   ```bash title="Install Dagster and dependencies"
+   pip install dagster dagster-webserver pandas dagster-duckdb
+   ```
+
+## Step 2: Copying Project Scaffold
+
+Next we will get the raw data for the project. As well as the project scaffold, Dagster has several pre-built scaffolds you can install depending on your use case. You can see the full up to date list by running. `dagster project list-examples`
+
+Use the project scaffold command for this project. 
+   ```bash title="ETL Project Scaffold"
+      dagster project from-example --example getting_started_etl_tutorial
+   ```
+
+The project should have this structure. 
+
+```
+dagster-etl-tutorial/
+├── data/
+│   └── products.csv
+│   └── sales_data.csv
+│   └── sales_reps.csv
+│   └── sample_request/
+│       └── request.json
+├── etl_tutorial/
+│   └── definitions.py
+├── pyproject.toml
+├── setup.cfg
+├── setup.py
+```
+
+## Dagster Project Structure
+
+In the root directory there are three configuration files that are common in Python package management. These manage dependencies and identifies the Dagster modules in the project. The etl_tutorial folder is where our Dagster definition for this code location exists. The data directory is where the raw data for the project is stored and we will reference these files in our software-defined assets. 
+
+### File/Directory Descriptions
+
+#### Dagster files
+
+- **etl_tutorial/**: This is a Python module that contains your Dagster code. It is the main directory where you will define your assets, jobs, schedules, sensors, and resources.
+
+  - **definitions.py**: This file is typically used to define jobs, schedules, and sensors. It organizes the various components of your Dagster project. This allows Dagster to load the definitions in a module.
+
+#### Python files
+
+- **pyproject.toml**: This file is used to specify build system requirements and package metadata for Python projects. It is part of the Python packaging ecosystem.
+
+- **setup.cfg**: This file is used for configuration of your Python package. It can include metadata about the package, dependencies, and other configuration options.
+
+- **setup.py**: This script is used to build and distribute your Python package. It is a standard file in Python projects for specifying package details.
+
+## What you've learned
+
+- Set up a Python virtual environment and installed Dagster
+- Setup project scaffold
+- How a Dagster project is structured and what these files do 
+
+## Next steps
+
+- Continue this tutorial with [your first asset](/tutorial/02-your-first-asset)

docs/docs-beta/docs/tutorial/02-your-first-asset.md

@@ -0,0 +1,106 @@
+---
+title: Your First Asset 
+description: Get the project data and create your first Asset
+last_update:
+  date: 2024-10-16
+  author: Alex Noonan
+---
+
+# Your First Software Defined Asset
+
+Now that we have the raw data files and the Dagster project setup lets create some loading those csv's into DuckDB. 
+
+Asset definitions enable a declarative approach to data management, in which code is the source of truth on what data assets should exist and how those assets are computed.
+
+<iframe width="560" height="315" src="https://www.youtube.com/embed/In4CUoFKOfY?si=Xnk_CADS1pf7D5BA" title="YouTube video player" frameborder="0" allow="accelerometer; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" referrerpolicy="strict-origin-when-cross-origin" allowfullscreen></iframe>
+
+## What you'll learn
+
+- Creating our initial definitions object
+- Adding a DuckDB resource
+- Building some basic software defined assets 
+
+## Building definitions object
+
+The [definitions](/api/definitions) object in Dagster serves as the central configuration point for defining and organizing various components within a Dagster Project. It acts as a container that holds all the necessary configurations for a code location, ensuring that everything is organized and easily accessible. 
+
+1. Creating Definitions object and DuckDB resource
+
+Open the definitions.py file and add the following import statements and definitions object. 
+
+  ```python
+  import json
+  import os
+
+  from dagster_duckdb import DuckDBResource
+
+  import dagster as dg
+
+  defs = dg.Definitions(
+    assets=[],
+    resources={"duckdb": DuckDBResource(database="data/mydb.duckdb")},
+  )
+  ```
+
+## Loading raw data
+
+1. Products Asset
+
+We need to create an asset that creates a duckdb table for the products csv. Additionally we should add meta data to help categorize this asset and give us a preview of what it looks like in the Dagster UI. 
+
+<CodeExample filePath="guides/tutorials/etl_tutorial/etl_tutorial/definitions.py" language="python" lineStart="8" lineEnd="33"/>
+
+You'll notice here that we have meta data for the compute kind for this asset as well as making it part of the ingestion group. Additionally, at the end we add the row count and a preview of what the table looks like. 
+
+2. Sales Reps Asset
+
+This code will be very similar to the product asset but this time its focused on Sales Reps.
+
+<CodeExample filePath="guides/tutorials/etl_tutorial/etl_tutorial/definitions.py" language="python" lineStart="35" lineEnd="61"/>
+
+3. Sales Data Asset
+
+Same thing for Sales Data
+
+<CodeExample filePath="guides/tutorials/etl_tutorial/etl_tutorial/definitions.py" language="python" lineStart="62" lineEnd="87"/>
+
+4. Bringing our assets into the Definitions object
+
+Now to pull these assets into our definitions object, add them to the empty list in the assets parameter. 
+
+  ```python
+  defs = dg.Definitions(
+      assets=[products,
+          sales_reps,
+          sales_data,
+      ],
+      resources={"duckdb": DuckDBResource(database="data/mydb.duckdb")},
+  )
+  ```
+
+## Materialize Assets
+
+Let's fire up Dagster and materialize these assets. If you are not in the project root directory navigate there now. 
+
+  ```bash title="Navigate to Project Directory"
+    cd getting_started_etl_tutorial
+  ```
+
+Run the `dagster dev` command. Dagster should open up in your browser. Navigate to the Global asset lineage page. You should see this
+
+   ![2048 resolution](/images/tutorial/etl-tutorial/etl-tutorial-first-asset-lineage.png)
+
+Click on products and then materialize. Navigate to the runs tab and select the most recent run. 
+
+   ![2048 resolution](/images/tutorial/etl-tutorial/first-asset-run.png)
+
+Do the same for sales_reps and sales_data. Now we have all our ingestion assets materialized
+
+## What you've learned
+
+- Created a Dagster Definition
+- Built our ingestion assets
+
+## Next steps
+
+- Continue this tutorial with your [Asset Dependencies and Checks](/tutorial/03-asset-dependencies-and-checks)

docs/docs-beta/docs/tutorial/03-asset-dependencies-and-checks.md

@@ -0,0 +1,69 @@
+---
+title: Asset Dependencies and Checks
+description: Reference Assets as dependencies to other assets and asset checks. 
+last_update:
+  date: 2024-10-16
+  author: Alex Noonan
+---
+
+# Asset Dependencies and Asset Checks
+
+The DAG or Directed Acyclic Graph is a key part of Dagster. This is an improvement over the typical cron workflow for orchestration. With a Dag approach you can easily understand complex data pipelines. The key benefits of Dags are
+
+1. Clarity: The DAG provides a clear visual representation of the entire workflow.
+2. Efficiency: Parallel tasks can be identified and executed simultaneously.
+3. Reliability: Dependencies ensure that tasks are executed in the correct order.
+4. Scalability: Complex workflows can be managed effectively.
+5. Maintenance: It's easier to update or troubleshoot specific parts of the workflow.
+
+## What you'll learn
+
+- Creating [Asset Dependencies](guides/asset-dependencies.md) 
+- How to make an [Asset Check](guides/asset-checks.md)
+
+## Creating a Downstream asset
+
+Now that we have all of our raw data loaded and staged into the DuckDB database our next step is to merge it together. The data structure that of a fact table (sales data) with 2 dimensions off of it (sales reps and products). To accomplish that in SQL we will bring in our sales_data table and then left join on sales reps and products on their respective id columns. Additionally, we will keep this view concise and only have relevant columns for analysis.
+
+<CodeExample filePath="guides/tutorials/etl_tutorial/etl_tutorial/definitions.py" language="python" lineStart="89" lineEnd="132"/>
+
+As you can see here this asset looks a lot like our previous ones with a few small changes. We put this asset into a different group. To make this asset dependant on the raw tables we add the asset keys the `deps` parameter in the asset definition.  
+
+## Asset checks
+
+Data Quality is critical in analytics. Just like in a factory producing cars, manufacturers inspect parts after they complete steps to identify defects and processes that may be creating more than acceptable. In this case we want to create a test to identify if there are any rows that have a product or sales rep that are not in the table. 
+
+<CodeExample filePath="guides/tutorials/etl_tutorial/etl_tutorial/definitions.py" language="python" lineStart="134" lineEnd="149"/>
+
+
+## Materialize the Assets
+
+We need to add the asset and asset check we just made to the definitions object. 
+
+Your Definitions object should now look like this:
+
+  ```python 
+  defs = dg.Definitions(
+    assets=[products,
+        sales_reps,
+        sales_data,
+        joined_data,
+    ],
+    asset_checks=[missing_dimension_check],
+    resources={"duckdb": DuckDBResource(database="data/mydb.duckdb")},
+  )
+  ```
+
+Go back into the UI, reload definitions, and materialize the joined_data asset. If you navigate to the asset details page, there is tab for asset checks where you can see the run history and metadata. 
+
+  ![2048 resolution](/images/tutorial/etl-tutorial/asset-check.png)
+
+## What you've learned
+
+- Creating downstream assets
+- Software defined asset checks. 
+
+
+## Next steps
+
+- Continue this tutorial with your [Partitions](/tutorial/04-partitions)

docs/docs-beta/docs/tutorial/04-partitions.md

@@ -0,0 +1,10 @@
+---
+title: Partitions
+description: Partitioning Assets by datetime and categories
+last_update:
+  date: 2024-10-16
+  author: Alex Noonan
+---
+
+
+

docs/docs-beta/sidebars.ts

@@ -11,7 +11,11 @@ const sidebars: SidebarsConfig = {
       type: 'category',
       label: 'Tutorial',
       collapsed: false,
-      items: ['tutorial/tutorial-etl'],
+      items: [
+        'tutorial/etl-tutorial-introduction',
+        'tutorial/your-first-asset',
+        'tutorial/asset-dependencies-and-checks',
+      ],
     },
     {
       type: 'category',

docs/docs-beta/src/theme/MDXComponents.tsx

@@ -1,6 +1,6 @@
 // Import the original mapper
 import MDXComponents from '@theme-original/MDXComponents';
-import { PyObject } from '../components/PyObject';
+import {PyObject} from '../components/PyObject';
 import CodeExample from '../components/CodeExample';
 import Tabs from '@theme/Tabs';
 import TabItem from '@theme/TabItem';