[docs-beta] Rework sidebar (#24305)

## Summary & Motivation

After the Docathon, I realized our Sidebar had gotten out of hand. This
is one attempt at taming it. I think we went too far down the path of
'jobs to be done' which made the sidebar harder to read and parse.

I went with three over-arching categories: build, scale, deploy to help
organize the high-level flows of a user as they build with Dagster.

Within each category, I split things into smaller groups: e.g. within
Build we have: Create a pipeline, Configure behavior, abstracting logic,
and common use cases

Within Scale, I focused on things like creating resources, adding data
quality, and automation.

In Deployment, we have self-hosting, and Dagster plus, along with things
like secrets management, and locations.

I also moved all the guides into a single folder. Having many subfolders
with specific names makes it hard to refactor the sidebar quickly. We
can but them back into folders once we have settled on a sidebar method
if needed.

I am least confident in the 'Scale' category, open to ideas here.

## How I Tested These Changes

Collapsed


![image](https://github.com/user-attachments/assets/69617be2-a814-4586-a8b9-dfe5c2fe2979)

Some open


![20240907_12h55m03s_grim](https://github.com/user-attachments/assets/568c4ced-6f0a-4fc2-a6d2-c576b200e56f)

## Changelog

NOCHANGELOG

docs/docs-beta/docs/concepts/automation.md

@@ -4,11 +4,11 @@ title: About Automation
 
 There are several ways to automate the execution of your data pipelines with Dagster.
 
-The first system, and the most basic, is the [Schedule](/guides/automation/schedules), which responds to time.
+The first system, and the most basic, is the [Schedule](/guides/schedules), which responds to time.
 
-[Sensors](/guides/automation/sensors) are like schedules, but they respond to an external event defined by the user.
+[Sensors](/guides/sensors) are like schedules, but they respond to an external event defined by the user.
 
-[Asset Sensors](/guides/automation/asset-sensors) are a special case of sensor that responds to changes in asset materialization
+[Asset Sensors](/guides/asset-sensors) are a special case of sensor that responds to changes in asset materialization
 as reported by the Event Log.
 
 Finally, the Declarative Automation system is a

docs/docs-beta/docs/dagster-plus/access/rbac/user-roles-permissions.md

@@ -117,7 +117,7 @@ TODO: add picture previously at "/images/dagster-cloud/user-token-management/cod
 | Start and stop [schedules](/concepts/schedules)                          | ❌     | ❌       | ✅     | ✅    | ✅                       |
 | Start and stop [schedules](/concepts/sensors)                            | ❌     | ❌       | ✅     | ✅    | ✅                       |
 | Wipe assets                                                              | ❌     | ❌       | ✅     | ✅    | ✅                       |
-| Launch and cancel [schedules](/guides/ingestion-transformation/backfill) | ❌     | ✅       | ✅     | ✅    | ✅                       |
+| Launch and cancel [schedules](/guides/backfill) | ❌     | ✅       | ✅     | ✅    | ✅                       |
 | Add dynamic partitions                                                   | ❌     | ❌       | ✅     | ✅    | ✅                       |
 
 ### Deployments

docs/docs-beta/docs/dagster-plus/insights/asset-metadata.md

@@ -25,7 +25,7 @@ You'll need one or more assets that emit the same metadata key at run time. Insi
 are most valuable when you have multiple assets that emit the same kind of metadata, such as
 such as the number of rows processed or the size of a file uploaded to object storage.
 
-Follow [the metadata guide](/guides/data-modeling/metadata#attaching-metadata-to-an-asset-at-runtime) to add numeric metadata
+Follow [the metadata guide](/guides/metadata#attaching-metadata-to-an-asset-at-runtime) to add numeric metadata
 to your asset materializations.
 
 ## Step 2: Enable viewing your metadata in Dagster+ Insights

docs/docs-beta/docs/dagster-plus/saved-views.md

@@ -19,7 +19,7 @@ In this guide, you'll learn how to create, access, and share catalog views with
 <summary>Prerequisites</summary>
 
 - **Organization Admin**, **Admin**, or **Editor** permissions on Dagster+
-- Familiarity with [Assets](/concepts/assets) and [Asset metadata](/guides/data-modeling/metadata)
+- Familiarity with [Assets](/concepts/assets) and [Asset metadata](/guides/metadata)
 
 </details>
 

docs/docs-beta/docs/dagster-plus/whats-dagster-plus.md

@@ -10,7 +10,7 @@ Dagster+ is a managed orchestration platform built on top of Dagster's open sour
 
 Dagster+ is built to be the most performant, reliable, and cost effective way for data engineering teams to run Dagster in production. Dagster+ is also great for students, researchers, or individuals who want to explore Dagster with minimal overhead.
 
-Dagster+ comes in two flavors: a fully [Serverless](/dagster-plus/deployment/serverless) offering and a [Hybrid](/dagster-plus/deployment/hybrid) offering. In both cases, Dagster+ does the hard work of managing your data orchestration control plane. Compared to a [Dagster open source deployment](/guides/deployment/), Dagster+ manages:
+Dagster+ comes in two flavors: a fully [Serverless](/dagster-plus/deployment/serverless) offering and a [Hybrid](/dagster-plus/deployment/hybrid) offering. In both cases, Dagster+ does the hard work of managing your data orchestration control plane. Compared to a [Dagster open source deployment](/guides/), Dagster+ manages:
 
 - Dagster's web UI at https://dagster.plus
 - Metadata stores for data cataloging and cost insights
@@ -33,5 +33,5 @@ Ready to [get started](/dagster-plus/getting-started)?
 - Learn more about Dagster+ [pricing and plan types](https://dagster.io/pricing) or [contact the Dagster team](https://dagster.io/contact)
 - Dagster+ includes support, [click here](https://dagster.io/support) to learn more.
 - Dagster+ is HIPAA compliant, SOC 2 Type II certified, and meets GDPR requirements. Learn more about Dagster+[ security](https://dagster.io/security).
-- Migrate [from a Dagster open source deployment to Dagster+](/guides/deployment/self-hosted-to-dagster-plus)
+- Migrate [from a Dagster open source deployment to Dagster+](/guides/self-hosted-to-dagster-plus)
 - Dagster+ [status page](https://dagstercloud.statuspage.io/)

docs/docs-beta/docs/guides/external-systems/apis.md → docs/docs-beta/docs/guides/apis.md

@@ -74,7 +74,7 @@ You can also fetch environment variables using the `os` library. Dagster treats
 
 ## Next steps
 
-- [Authenticate to a resource](/guides/external-systems/authentication.md)
+- [Authenticate to a resource](/guides/authentication.md)
 - [Use different resources in different execution environments](/todo)
 - [Set environment variables in Dagster+](/todo)
 - Learn what [Dagster-provided resources](/todo) are available to use

docs/docs-beta/docs/guides/data-modeling/asset-factories-with-deps.md → docs/docs-beta/docs/guides/asset-factories-with-deps.md

@@ -1,7 +1,7 @@
 ---
 title: 'Programmatically defining dependencies using asset factories'
 sidebar_position: 60
-sidebar_label: 'Programmatically defining dependencies'
+sidebar_label: 'Asset Factories (2)'
 ---
 
 In data engineering, it's often helpful to reuse code to define similar assets. For example, you may want to represent every file in a directory as an asset.
@@ -20,7 +20,7 @@ Using an asset factory reduces complexity and creates a pluggable entry point to
 <details>
   <summary>Prerequisites</summary>
 
-This guide builds upon the concepts in the [asset factories](/guides/data-modeling/asset-factories) tutorial.
+This guide builds upon the concepts in the [asset factories](/guides/asset-factories) tutorial.
 </details>
 
 ---

docs/docs-beta/docs/guides/data-modeling/asset-factories.md → docs/docs-beta/docs/guides/asset-factories.md

@@ -1,7 +1,7 @@
 ---
 title: 'Creating domain-specific languages with asset factories'
 sidebar_position: 60
-sidebar_label: 'Creating domain-specific languages'
+sidebar_label: 'Asset factories'
 ---
 
 Often in data engineering, you'll find yourself needing to create a large number of similar assets. For example:

docs/docs-beta/docs/guides/automation/asset-sensors.md → docs/docs-beta/docs/guides/asset-sensors.md

@@ -1,7 +1,7 @@
 ---
 title: Trigger cross-job dependencies with Asset Sensors
 sidebar_position: 30
-sidebar_label: Trigger cross-job dependencies
+sidebar_label: Cross-job dependencies
 ---
 
 Asset sensors in Dagster provide a powerful mechanism for monitoring asset materializations and triggering downstream computations or notifications based on those events.

docs/docs-beta/docs/guides/automation.md

@@ -45,7 +45,7 @@ To make creating cron expressions easier, you can use an online tool like [Cront
 - You need to run jobs at regular intervals
 - You want basic time-based automation
 
-For examples of how to create schedules, see [How-To Use Schedules](/guides/automation/schedules).
+For examples of how to create schedules, see [How-To Use Schedules](/guides/schedules).
 
 For more information about how Schedules work, see [About Schedules](/concepts/schedules).
 
@@ -62,7 +62,7 @@ Like schedules, sensors operate on a selection of assets, known as [Jobs](/conce
 - You need event-driven automation
 - You want to react to changes in external systems
 
-For more examples of how to create sensors, see the [How-To Use Sensors](/guides/automation/sensors) guide.
+For more examples of how to create sensors, see the [How-To Use Sensors](/guides/sensors) guide.
 
 For more information about how sensors work, see the [About Sensors](/concepts/sensors) concept page.
 
@@ -75,7 +75,7 @@ Asset Sensors trigger jobs when specified assets are materialized, allowing you
 - You need to trigger jobs based on asset materializations
 - You want to create dependencies between different jobs or code locations
 
-For more examples of how to create asset sensors, see the [How-To Use Asset Sensors](/guides/automation/asset-sensors) guide.
+For more examples of how to create asset sensors, see the [How-To Use Asset Sensors](/guides/asset-sensors) guide.
 
 ## Declarative Automation
 

docs/docs-beta/docs/guides/data-modeling/configuring-assets.md → docs/docs-beta/docs/guides/configuring-assets.md

@@ -1,6 +1,6 @@
 ---
 title: Configuring assets using the Dagster UI
-sidebar_label: Configuring assets using the Dagster UI
+sidebar_label: Configure asset runs
 sidebar_position: 50
 ---
 

docs/docs-beta/docs/guides/data-modeling/data-assets.md → docs/docs-beta/docs/guides/data-assets.md

@@ -1,5 +1,6 @@
 ---
 title: "Defining data assets with decorators"
+sidebar_label: "Create Assets"
 sidebar_position: 10
 ---
 
@@ -79,6 +80,6 @@ flowchart LR
 
 ## Next steps
 
-- Learn to create [dependencies between assets](/guides/data-modeling/asset-dependencies)
-- Enrich Dagster's built-in data catalog with [asset metadata](/guides/data-modeling/metadata)
-- Learn to use a [factory pattern](/guides/data-modeling/asset-factories) to create multiple, similar assets
+- Learn to create [dependencies between assets](/guides/asset-dependencies)
+- Enrich Dagster's built-in data catalog with [asset metadata](/guides/metadata)
+- Learn to use a [factory pattern](/guides/asset-factories) to create multiple, similar assets

docs/docs-beta/docs/guides/external-systems/databases.md → docs/docs-beta/docs/guides/databases.md

@@ -56,6 +56,6 @@ See [Dagster Integrations](https://dagster.io/integrations) for resource types t
 
 ## Next steps
 
-- Explore how to use resources for [Connecting to APIs](/guides/external-systems/apis)
+- Explore how to use resources for [Connecting to APIs](/guides/apis)
 - Go deeper into [Understanding Resources](/concepts/resources)
 

docs/docs-beta/docs/guides/declarative-automation.md

@@ -0,0 +1,4 @@
+---
+title: "Declarative automation"
+sidebar_label: "Declarative automation"
+---

docs/docs-beta/docs/guides/data-modeling/external-assets.md → docs/docs-beta/docs/guides/external-assets.md

@@ -1,7 +1,7 @@
 ---
 title: Representing external data sources with external assets
 sidebar_position: 80
-sidebar_label: 'Representing external data sources'
+sidebar_label: 'External data assets'
 ---
 
 One of Dagster's goals is to present a single unified lineage of all of the data assets in an organization. This can include assets orchestrated by Dagster and assets orchestrated by other systems.
@@ -26,8 +26,8 @@ Examples of external assets could be files in a data lake that are populated by
 
 To follow the steps in this guide, you'll need:
 
-- A basic understanding of Dagster and assets. See the [Quick Start](/getting-started/quickstart) tutorial for an overview.
-- Familiarity with [Sensors](/guides/automation/sensors)
+- A basic understanding of Dagster and assets. See the [Quick Start](/tutorial/quick-start) tutorial for an overview.
+- Familiarity with [Sensors](/guides/sensors)
 </details>
 
 ---
@@ -48,13 +48,13 @@ There are two main ways to do this: "pulling" external assets events with sensor
 
 ### "Pulling" with sensors
 
-You can use a Dagster [sensor](/guides/automation/sensors) to regularly poll the external system and "pull" information about the external asset into Dagster.
+You can use a Dagster [sensor](/guides/sensors) to regularly poll the external system and "pull" information about the external asset into Dagster.
 
 For example, here's how you would poll an external system (like an SFTP server) to update an external asset whenever the file is changed.
 
 <CodeExample filePath="guides/data-modeling/external-assets/pulling-with-sensors.py" language="python" title="Pulling external asset events with sensors" />
 
-See the [sensors guide](/guides/automation/sensors) for more information about sensors.
+See the [sensors guide](/guides/sensors) for more information about sensors.
 
 ### "Pushing" with the REST API
 

docs/docs-beta/docs/guides/ingestion-transformation/ingesting-data.md → docs/docs-beta/docs/guides/ingesting-data.md

@@ -2,7 +2,7 @@
 title: Ingesting data with Dagster
 description: Learn how to orchestrate data ingestion with Dagster
 sidebar_position: 10
-sidebar_label: Ingesting data
+sidebar_label: Ingest data with embedded-elt
 ---
 
 Dagster is often used to orchestrate the ingestion of data into a data warehouse or data lake, where it can be queried and transformed. Dagster integrates with several tools that are purpose-built for data ingestion, and it also enables writing custom code for ingesting data.
@@ -58,4 +58,4 @@ For example, imagine there's a CSV file of counties on the internet and you want
 
 {/* TODO add next steps */}
 
-- Learn how to [transform data using Dagster's dbt integration](/guides/ingestion-transformation/transform-dbt)
+- Learn how to [transform data using Dagster's dbt integration](/guides/transform-dbt)

docs/docs-beta/docs/guides/external-systems/io-managers.md → docs/docs-beta/docs/guides/io-managers.md

@@ -1,7 +1,7 @@
 ---
 title: "Managing stored data with I/O managers"
 sidebar_position: 50
-sidebar_label: "Storing data with I/O managers"
+sidebar_label: "I/O managers"
 ---
 
 I/O managers in Dagster provide a way to separate the code that's responsible for logical data transformation from the code that's responsible for reading and writing the results.  This can help reduce boiler plate code and make it easy to swap out where your data is stored.

docs/docs-beta/docs/guides/data-modeling/metadata.md → docs/docs-beta/docs/guides/metadata.md

@@ -2,7 +2,7 @@
 title: 'Adding tags and metadata to assets'
 description: 'Learn how to add tags and metadata to assets to improve observability in Dagster'
 sidebar_position: 40
-sidebar_label: 'Enriching assets with metadata'
+sidebar_label: 'Add metadata'
 ---
 
 Assets feature prominently in the Dagster UI. It's often helpful to attach information to assets to understand where they're stored, what they contain, and how they should be organized.
@@ -156,7 +156,7 @@ Dagster can automatically attach code references to your assets during local dev
 
 ### Customizing code references
 
-If you want to customize how code references are attached - such as when you are building [domain-specific languages with asset factories](/guides/data-modeling/asset-factories) - you can manually add the `dagster/code_references` metadata to your asset definitions.
+If you want to customize how code references are attached - such as when you are building [domain-specific languages with asset factories](/guides/asset-factories) - you can manually add the `dagster/code_references` metadata to your asset definitions.
 
 <CodeExample filePath="guides/data-modeling/metadata/custom-local-references.py" language="python" title="Custom local source code references" />
 

docs/docs-beta/docs/guides/external-systems/non-python.md → docs/docs-beta/docs/guides/non-python.md

@@ -1,5 +1,6 @@
 ---
-title: "Running non-Python languages"
+title: "External Compute"
+sidebar_label: "Dagster Pipes"
 sidebar_position: 60
 ---
 

docs/docs-beta/docs/guides/data-modeling/partitioning.md → docs/docs-beta/docs/guides/partitioning.md

@@ -1,7 +1,7 @@
 ---
 title: Partitioning assets
 description: Learn how to partition your data in Dagster.
-sidebar_label: Partitioning assets
+sidebar_label: Partition data
 sidebar_position: 30
 ---
 

docs/docs-beta/docs/guides/resources.md

@@ -0,0 +1,6 @@
+---
+title: Using Resources to manage external systems
+sidebar_label: Resources
+---
+
+Dagster resources are objects that provide access to external systems, databases, or services. Resources are used to manage connections to external systems, and are used by Dagster ops and assets.

docs/docs-beta/docs/guides/automation/schedules.md → docs/docs-beta/docs/guides/schedules.md

@@ -1,6 +1,6 @@
 ---
 title: "Schedule cron-based pipelines"
-sidebar_label: "Run pipelines on a schedule"
+sidebar_label: "Schedules"
 sidebar_position: 10
 ---
 
@@ -59,7 +59,7 @@ partitioned_op_schedule = build_schedule_from_partitioned_job(
 ## Next steps
 
 - Learn more about schedules in [Understanding Automation](/concepts/automation)
-- React to events with [sensors](/guides/automation/sensors)
+- React to events with [sensors](/guides/sensors)
 - Explore [Declarative Automation](/concepts/automation/declarative-automation) as an alternative to schedules
 
 By understanding and effectively using these automation methods, you can build more efficient data pipelines that respond to your specific needs and constraints.

docs/docs-beta/docs/guides/automation/sensors.md → docs/docs-beta/docs/guides/sensors.md

@@ -1,6 +1,6 @@
 ---
 title: Create event-based pipelines with sensors
-sidebar_label: Create event-based pipelines
+sidebar_label: Sensors
 sidebar_position: 20
 ---
 

docs/docs-beta/docs/guides/ingestion-transformation/transform-dbt.md → docs/docs-beta/docs/guides/transform-dbt.md

@@ -1,6 +1,7 @@
 ---
 title: Transforming data with dbt
 sidebar_position: 20
+sidebar_label: Transform data with dbt
 last_update:
   date: 2024-08-26
   author: Nick Roach