dagster-io · neverett · Dec 5, 2024 · Dec 5, 2024 · Dec 5, 2024 · Dec 5, 2024
@@ -22,11 +22,11 @@ The default I/O manager cannot be used if you are a Serverless user who:
 - Are otherwise working with data subject to GDPR or other such regulations
   :::
 
-In Serverless, code that uses the default [I/O manager](/guides/build/configure/io-managers) is automatically adjusted to save data in Dagster+ managed storage. This automatic change is useful because the Serverless filesystem is ephemeral, which means the default I/O manager wouldn't work as expected.
+In Serverless, code that uses the default [I/O manager](/guides/operate/io-managers) is automatically adjusted to save data in Dagster+ managed storage. This automatic change is useful because the Serverless filesystem is ephemeral, which means the default I/O manager wouldn't work as expected.
 
 However, this automatic change also means potentially sensitive data could be **stored** and not just processed or orchestrated by Dagster+.
 
-To prevent this, you can use [another I/O manager](/guides/build/configure/io-managers#built-in) that stores data in your infrastructure or [adapt your code to avoid using an I/O manager](/guides/build/configure/io-managers#before-you-begin).
+To prevent this, you can use [another I/O manager](/guides/operate/io-managers#built-in) that stores data in your infrastructure or [adapt your code to avoid using an I/O manager](/guides/operate/io-managers#before-you-begin).
 
 :::note
 You must have [boto3](https://pypi.org/project/boto3/) or `dagster-cloud[serverless]` installed as a project dependency otherwise the Dagster+ managed storage can fail and silently fall back to using the default I/O manager.

@@ -8,4 +8,4 @@ unlisted: true
 
 ## Disabling compute logs
 
-{/* NOTE this is a placeholder section so the Hybrid deployment index page has somewhere to link to */}
+{/* NOTE this is a placeholder section so the Hybrid deployment index page has somewhere to link to */}
@@ -115,7 +115,7 @@ TODO: add picture previously at "/images/dagster-cloud/user-token-management/cod
 | Start and stop [schedules](/guides/automate/schedules)                          | ❌     | ❌       | ✅     | ✅    | ✅                       |
 | Start and stop [schedules](/guides/automate/sensors)                            | ❌     | ❌       | ✅     | ✅    | ✅                       |
 | Wipe assets                                                              | ❌     | ❌       | ✅     | ✅    | ✅                       |
-| Launch and cancel [schedules](/guides/build/backfill) | ❌     | ✅       | ✅     | ✅    | ✅                       |
+| Launch and cancel [schedules](/guides/automate/schedules) | ❌     | ✅       | ✅     | ✅    | ✅                       |
 | Add dynamic partitions                                                   | ❌     | ❌       | ✅     | ✅    | ✅                       |
 
 ### Deployments

@@ -17,7 +17,7 @@ In this guide, you'll learn how to create, access, and share catalog views with
 <summary>Prerequisites</summary>
 
 - **Organization Admin**, **Admin**, or **Editor** permissions on Dagster+
-- Familiarity with [Assets](/guides/build/assets-concepts/index.mdx and [Asset metadata](/guides/build/create-a-pipeline/metadata)
+- Familiarity with [Assets](/guides/build/create-asset-pipelines/assets-concepts/index.mdx and [Asset metadata](/guides/build/create-asset-pipelines/metadata)
 
 </details>
 

@@ -21,7 +21,7 @@ You'll need one or more assets that emit the same metadata key at run time. Insi
 are most valuable when you have multiple assets that emit the same kind of metadata, such as
 such as the number of rows processed or the size of a file uploaded to object storage.
 
-Follow [the metadata guide](/guides/build/create-a-pipeline/metadata#runtime-metadata) to add numeric metadata
+Follow [the metadata guide](/guides/build/create-asset-pipelines/metadata#runtime-metadata) to add numeric metadata
 to your asset materializations.
 
 ## Step 2: Enable viewing your metadata in Dagster+ Insights

@@ -7,7 +7,7 @@ Dagster+ is a managed orchestration platform built on top of Dagster's open sour
 
 Dagster+ is built to be the most performant, reliable, and cost effective way for data engineering teams to run Dagster in production. Dagster+ is also great for students, researchers, or individuals who want to explore Dagster with minimal overhead.
 
-Dagster+ comes in two flavors: a fully [Serverless](/dagster-plus/deployment/deployment-types/serverless) offering and a [Hybrid](/dagster-plus/deployment/deployment-types/hybrid) offering. In both cases, Dagster+ does the hard work of managing your data orchestration control plane. Compared to a [Dagster open source deployment](/guides/), Dagster+ manages:
+Dagster+ comes in two flavors: a fully [Serverless](/dagster-plus/deployment/deployment-types/serverless) offering and a [Hybrid](/dagster-plus/deployment/deployment-types/hybrid) offering. In both cases, Dagster+ does the hard work of managing your data orchestration control plane. Compared to a [Dagster open source deployment](guides/deploy/index.md), Dagster+ manages:
 
 - Dagster's web UI at https://dagster.plus
 - Metadata stores for data cataloging and cost insights

@@ -1,7 +1,6 @@
 ---
 title: Glossary
 sidebar_position: 30
-sidebar_label: Glossary
 unlisted: true
 ---
 

@@ -5,8 +5,6 @@ sidebar_position: 20
 sidebar_label: Installation
 ---
 
-# Installing Dagster
-
 To follow the steps in this guide, you'll need:
 
 - To install Python 3.9 or higher. **Python 3.12 is recommended**.
@@ -72,4 +70,4 @@ If you encounter any issues during the installation process:
 ## Next steps
 
 - Get up and running with your first Dagster project in the [Quickstart](/getting-started/quickstart)
-- Learn to [create data assets in Dagster](/guides/build/create-a-pipeline/data-assets)
+- Learn to [create data assets in Dagster](/guides/build/create-asset-pipelines/data-assets)
@@ -1,12 +1,10 @@
 ---
-title: "Dagster quickstart"
+title: Build your first Dagster project
 description: Learn how to quickly get up and running with Dagster
 sidebar_position: 30
 sidebar_label: "Quickstart"
 ---
 
-# Build your first Dagster project
-
 Welcome to Dagster! In this guide, you'll use Dagster to create a basic pipeline that:
 
 - Extracts data from a CSV file
@@ -154,4 +152,4 @@ id,name,age,city,age_group
 Congratulations! You've just built and run your first pipeline with Dagster. Next, you can:
 
 - Continue with the [ETL pipeline tutorial](/tutorial/tutorial-etl) to learn how to build a more complex ETL pipeline
-- Learn how to [Think in assets](/guides/build/assets-concepts/index.md)
+- Learn how to [Think in assets](/guides/build/create-asset-pipelines/assets-concepts/index.md)
@@ -3,6 +3,8 @@ title: About Automation
 unlisted: true
 ---
 
+{/* TODO combine with index page and delete this page */}
+
 There are several ways to automate the execution of your data pipelines with Dagster.
 
 The first system, and the most basic, is the [Schedule](/guides/automate/schedules), which responds to time.
@@ -24,8 +26,6 @@ as the schedule is processed.
 Schedules were one of the first types of automation in Dagster, created before the introduction of Software-Defined Assets.
 As such, you may find that many of the examples can seem foreign if you are used to only working within the asset framework.
 
-For more on how assets and ops inter-relate, read about [Assets and Ops](/guides/build/assets-concepts#assets-and-ops)
-
 The `dagster-daemon` process is responsible for submitting runs by checking each schedule at a regular interval to determine
 if it's time to execute the underlying job.
 

@@ -1,22 +1,17 @@
 ---
-title: Triggering cross-job dependencies with Asset Sensors
-sidebar_position: 300
-sidebar_label: Cross-job dependencies
+title: Trigger cross-job dependencies with asset sensors
+sidebar_position: 40
 ---
 
 Asset sensors in Dagster provide a powerful mechanism for monitoring asset materializations and triggering downstream computations or notifications based on those events.
 
 This guide covers the most common use cases for asset sensors, such as defining cross-job and cross-code location dependencies.
 
-<details>
-<summary>Prerequisites</summary>
+:::note
 
-To follow this guide, you'll need:
+This documentation assumes familiarity with [assets](/guides/build/create-asset-pipelines/assets-concepts/index.md) and [ops and jobs](/guides/build/ops-jobs)
 
-- Familiarity with [Assets](/guides/build/assets-concepts/index.mdx
-- Familiarity with [Ops and Jobs](/guides/build/ops-jobs)
-
-</details>
+:::
 
 ## Getting started
 
@@ -54,7 +49,7 @@ This is an example of an asset sensor that triggers a job when an asset is mater
 
 <CodeExample filePath="guides/automation/simple-asset-sensor-example.py" language="python" />
 
-## Customize evaluation logic
+## Customizing the evaluation function of an asset sensor
 
 You can customize the evaluation function of an asset sensor to include specific logic for deciding when to trigger a run. This allows for fine-grained control over the conditions under which downstream jobs are executed.
 
@@ -83,15 +78,15 @@ In the following example, the `@asset_sensor` decorator defines a custom evaluat
 
 <CodeExample filePath="guides/automation/asset-sensor-custom-eval.py" language="python"/>
 
-## Trigger a job with configuration
+## Triggering a job with custom configuration
 
 By providing a configuration to the `RunRequest` object, you can trigger a job with a specific configuration. This is useful when you want to trigger a job with custom parameters based on custom logic you define.
 
 For example, you might use a sensor to trigger a job when an asset is materialized, but also pass metadata about that materialization to the job:
 
 <CodeExample filePath="guides/automation/asset-sensor-with-config.py" language="python" />
 
-## Monitor multiple assets
+## Monitoring multiple assets
 
 When building a pipeline, you may want to monitor multiple assets with a single sensor. This can be accomplished with a multi-asset sensor.
 

@@ -0,0 +1,4 @@
+---
+title: Arbitrary Python automation conditions
+sidebar_position: 500
+---
@@ -0,0 +1,10 @@
+---
+title: Automation conditions operands and operators
+sidebar_position: 600
+---
+
+## Operands
+
+## Operators
+
+## Composite conditions
@@ -0,0 +1,4 @@
+---
+title: Describing conditions with labels
+sidebar_position: 100
+---
@@ -0,0 +1,4 @@
+---
+title: Ignoring missing upstream data
+sidebar_position: 200
+---
@@ -0,0 +1,10 @@
+---
+title: Customizing automation conditions
+sidebar_position: 10
+---
+
+Each `AutomationCondition` consists of a set of operands and various operators that you can combine to suit your needs. For a full list of operands, operators, and composition conditions, see [Automation condition operands and operators](automation-condition-operands-and-operators).
+
+import DocCardList from '@theme/DocCardList';
+
+<DocCardList />
@@ -0,0 +1,4 @@
+---
+title: Targeting upstream dependencies
+sidebar_position: 400
+---
@@ -0,0 +1,4 @@
+---
+title: Updating older time partitions
+sidebar_position: 300
+---
@@ -0,0 +1,12 @@
+---
+title: Declarative automation
+sidebar_position: 20
+---
+
+{/* TODO: What is DA, when would you use it */}
+
+## Automation conditions
+
+## Sensors
+
+{/* the fact that you need to enable the automation condition sensor in the dagster UI needs to be mentioned much earlier — as is, the current page is very confusing */}
@@ -0,0 +1,4 @@
+---
+title: Migrating to Declarative Automation
+sidebar_position: 100
+---
@@ -0,0 +1,4 @@
+---
+title: GraphQL triggers
+sidebar_position: 50
+---
@@ -1,7 +1,6 @@
 ---
 title: "Automating pipelines"
 description: Learn how to automate your data pipelines.
-sidebar_position: 40
 sidebar_class_name: hidden
 ---
 
@@ -12,7 +11,7 @@ Automation is key to building reliable, efficient data pipelines. This guide pro
 
 Before continuing, you should be familiar with:
 
-- [Asset definitions](/guides/build/assets-concepts/index.mdx
+- [Asset definitions](/guides/build/create-asset-pipelines/assets-concepts/index.mdx
 - [Jobs](/guides/build/ops-jobs)
 
 </details>

@@ -1,7 +1,6 @@
 ---
-title: "Schedule cron-based pipelines"
-sidebar_label: "Schedules"
-sidebar_position: 100
+title: "Automate jobs with schedules"
+sidebar_position: 10
 ---
 
 Schedules enable automated execution of jobs at specified intervals. These intervals can range from common frequencies like hourly, daily, or weekly, to more intricate patterns defined using cron expressions.
@@ -11,7 +10,7 @@ Schedules enable automated execution of jobs at specified intervals. These inter
 
 To follow the steps in this guide, you'll need:
 
-- Familiarity with [Assets](/guides/build/assets-concepts/index.mdx
+- Familiarity with [Assets](/guides/build/create-asset-pipelines/assets-concepts/index.mdx
 - Familiarity with [Ops and Jobs](/guides/build/ops-jobs/index.md)
 </details>
 
@@ -41,7 +40,7 @@ If using partitions and jobs, you can create a schedule using the partition with
 <Tabs>
 <TabItem value="assets" label="Assets">
 
-If you have a [partitioned asset](/guides/build/create-a-pipeline/partitioning) and job:
+If you have a [partitioned asset](/guides/build/create-asset-pipelines/partitioning) and job:
 
 <CodeExample filePath="guides/automation/schedule-with-partition.py" language="python" />
 

@@ -1,7 +1,6 @@
 ---
-title: Creating event-based pipelines with sensors
-sidebar_label: Event triggers
-sidebar_position: 200
+title: Create event-based pipelines with sensors
+sidebar_position: 30
 ---
 
 Sensors enable you to trigger Dagster runs in response to events from external systems. They run at regular intervals, either triggering a run or explaining why a run was skipped. For example, you can trigger a run when a new file is added to an Amazon S3 bucket or when a database row is updated.
@@ -15,7 +14,7 @@ An alternative to polling with sensors is to push events to Dagster using the [D
 
 To follow the steps in this guide, you'll need:
 
-- Familiarity with [Assets](/guides/build/assets-concepts/index.mdx
+- Familiarity with [Assets](/guides/build/create-asset-pipelines/assets-concepts/index.mdx
 - Familiarity with [Ops and Jobs](/guides/build/ops-jobs)
 </details>
 

@@ -1,7 +1,6 @@
 ---
-title: Connecting to APIs
+title: Connect to APIs
 sidebar_position: 600
-sidebar_label: API connections
 ---
 
 When building a data pipeline, you'll likely need to connect to several external APIs, each with its own specific configuration and behavior. This guide demonstrates how to standardize your API connections and customize their configuration using Dagster resources.
@@ -11,8 +10,8 @@ When building a data pipeline, you'll likely need to connect to several external
 
 To follow the steps in this guide, you'll need:
 
-- Familiarity with [Assets](/guides/build/assets-concepts/index.mdx
-- Familiarity with [Resources](/guides/build/configure/resources)
+- Familiarity with [Assets](/guides/build/create-asset-pipelines/assets-concepts/index.mdx
+- Familiarity with [Resources](/guides/build/connect-to-external-systems/resources)
 - To install the `requests` library:
     ```bash
     pip install requests

@@ -0,0 +1,4 @@
+---
+title: Connect to cloud services
+sidebar_position: 600
+---
@@ -1,8 +1,6 @@
 ---
-title: Connecting to databases
-description: How to configure resources to connect to databases
+title: Connect to a database
 sidebar_position: 500
-sidebar_label: Database connections
 ---
 
 When building a data pipeline, you may need to extract data from or load data into a database. In Dagster, resources can be used to connect to a database by acting as a wrapper around a database client.
@@ -14,7 +12,7 @@ This guide demonstrates how to standardize database connections and customize th
 
 To follow the steps in this guide, you'll need:
 
-- Familiarity with [Assets](/guides/build/create-a-pipeline/data-assets)
+- Familiarity with [Assets](/guides/build/create-asset-pipelines/data-assets)
 
 To run the examples in this guide, you'll need:
 
@@ -59,5 +57,5 @@ You can also fetch environment variables using the `os` library. Dagster treats
 
 ## Next steps
 
-- Explore how to use resources for [Connecting to APIs](/guides/build/configure/apis)
-- Go deeper into [Understanding Resources](/guides/build/configure/resources)
+- Explore how to use resources for [Connecting to APIs](/guides/build/connect-to-external-systems/apis)
+- Go deeper into [Understanding Resources](/guides/build/connect-to-external-systems/resources)
Original file line number	Diff line number	Diff line change
Expand Up		@@ -8,4 +8,4 @@ unlisted: true

		## Disabling compute logs

		{/* NOTE this is a placeholder section so the Hybrid deployment index page has somewhere to link to */}
		{/* NOTE this is a placeholder section so the Hybrid deployment index page has somewhere to link to */}