dagster-io · neverett · Dec 12, 2024 · Dec 6, 2024 · Dec 6, 2024 · Dec 10, 2024
@@ -446,7 +446,7 @@ Another option is to launch a pod for each asset by telling Dagster to use the K
 
 <TabItem value="pipes" label="Resources if using kubernetes pipes">
 
-Dagster can launch and manage existing Docker images as Kubernetes jobs using the [Dagster kubernetes pipes integration](/integrations/kubernetes). To request resources for these jobs by supplying the appropriate Kubernetes pod spec.
+Dagster can launch and manage existing Docker images as Kubernetes jobs using the [Dagster kubernetes pipes integration](/integrations/libraries/dagster-supported/kubernetes). To request resources for these jobs by supplying the appropriate Kubernetes pod spec.
 
 <CodeExample filePath="dagster-plus/deployment/hybrid/agents/kubernetes/resource_request_pipes.py" language="python" title="Request resources for a k8s pipes asset" />
 

@@ -33,10 +33,10 @@ As a data orchestrator, Dagster helps with data ingestion as it can:
 
 Dagster currently integrates with the following data ingestion tools, enabling you to sync diverse data sources into data warehouse tables using pre-built connectors:
 
-- [Airbyte](/integrations/airbyte)
-- [dlt](/integrations/dlt)
-- [Fivetran](/integrations/fivetran)
-- [Sling](/integrations/sling)
+- [Airbyte](/integrations/libraries/dagster-supported/airbyte)
+- [dlt](/integrations/libraries/dagster-supported/dlt)
+- [Fivetran](/integrations/libraries/dagster-supported/fivetran)
+- [Sling](/integrations/libraries/dagster-supported/sling)
 
 ## Writing custom data ingestion pipelines
 

@@ -33,7 +33,7 @@ To get started with asset checks, follow these general steps:
 ## Defining a single asset check \{#single-check}
 
 :::tip
-Dagster's dbt integration can model existing dbt tests as asset checks. Refer to the [dagster-dbt documentation](/integrations/dbt) for more information.
+Dagster's dbt integration can model existing dbt tests as asset checks. Refer to the [dagster-dbt documentation](/integrations/libraries/dagster-supported/dbt) for more information.
 :::
 
 A asset check is defined using the `@asset_check` decorator.

@@ -0,0 +1,5 @@
+---
+title: Creating a multi-asset integration
+---
+
+{/* TODO write this */}
@@ -1,8 +1,5 @@
 ---
-sidebar_class_name: hidden
 title: Integrations
 ---
 
-import DocCardList from '@theme/DocCardList';
-
-<DocCardList />
+You can integrate Dagster with external services using our libraries and libraries supported by the community.
@@ -14,9 +14,10 @@ categories:
   - ETL
 enabledBy:
 enables:
+tags: [dagster-supported, etl]
 ---
 
-### About this integration
+
 
 Using this integration, you can trigger Airbyte syncs and orchestrate your Airbyte connections from within Dagster, making it easy to chain an Airbyte sync with upstream or downstream steps in your workflow.
 

@@ -14,9 +14,10 @@ categories:
   - Storage
 enabledBy:
 enables:
+tags: [dagster-supported, storage]
 ---
 
-### About this integration
+
 
 This integration allows you to connect to AWS Athena, a serverless interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. Using this integration, you can issue queries to Athena, fetch results, and handle query execution states within your Dagster pipelines.
 

@@ -14,9 +14,10 @@ categories:
   - Monitoring
 enabledBy:
 enables:
+tags: [dagster-supported, monitoring]
 ---
 
-### About this integration
+
 
 This integration allows you to send Dagster logs to AWS CloudWatch, enabling centralized logging and monitoring of your Dagster jobs. By using AWS CloudWatch, you can take advantage of its powerful log management features, such as real-time log monitoring, log retention policies, and alerting capabilities.
 

@@ -14,9 +14,10 @@ categories:
   - Other
 enabledBy:
 enables:
+tags: [dagster-supported]
 ---
 
-### About this integration
+
 
 This integration allows you to connect to AWS Elastic Container Registry (ECR). It provides resources to interact with AWS ECR, enabling you to manage your container images.
 

@@ -14,9 +14,10 @@ categories:
   - Compute
 enabledBy:
 enables:
+tags: [dagster-supported, compute]
 ---
 
-### About this integration
+
 
 The `dagster-aws` integration provides ways orchestrating data pipelines that leverage AWS services, including AWS EMR (Elastic MapReduce). This integration allows you to run and scale big data workloads using open source tools such as Apache Spark, Hive, Presto, and more.
 

@@ -14,9 +14,10 @@ categories:
   - Compute
 enabledBy:
 enables:
+tags: [dagster-supported, compute]
 ---
 
-### About this integration
+
 
 The `dagster-aws` integration library provides the `PipesGlueClient` resource, enabling you to launch AWS Glue jobs directly from Dagster assets and ops. This integration allows you to pass parameters to Glue code while Dagster receives real-time events, such as logs, asset checks, and asset materializations, from the initiated jobs. With minimal code changes required on the job side, this integration is both efficient and easy to implement.
 

@@ -4,4 +4,4 @@ title: AWS
 
 import DocCardList from '@theme/DocCardList';
 
-<DocCardList />
+<DocCardList />
@@ -14,9 +14,10 @@ categories:
   - Compute
 enabledBy:
 enables:
+tags: [dagster-supported, compute]
 ---
 
-### About this integration
+
 
 Using this integration, you can leverage AWS Lambda to execute external code as part of your Dagster pipelines. This is particularly useful for running serverless functions that can scale automatically and handle various workloads without the need for managing infrastructure. The `PipesLambdaClient` class allows you to invoke AWS Lambda functions and stream logs and structured metadata back to Dagster's UI and tools.
 

@@ -14,9 +14,10 @@ categories:
   - Storage
 enabledBy:
 enables:
+tags: [dagster-supported, storage]
 ---
 
-### About this integration
+
 
 Using this integration, you can connect to an AWS Redshift cluster and issue queries against it directly from your Dagster assets. This allows you to seamlessly integrate Redshift into your data pipelines, leveraging the power of Redshift's data warehousing capabilities within your Dagster workflows.
 

@@ -14,9 +14,10 @@ categories:
   - Storage
 enabledBy:
 enables:
+tags: [dagster-supported, storage]
 ---
 
-### About this integration
+
 
 The AWS S3 integration allows data engineers to easily read, and write objects to the durable AWS S3 storage -- enabling engineers to a resilient storage layer when constructing their pipelines.
 

@@ -14,9 +14,10 @@ categories:
   - Other
 enabledBy:
 enables:
+tags: [dagster-supported]
 ---
 
-### About this integration
+
 
 This integration allows you to manage, retrieve, and rotate credentials, API keys, and other secrets using [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/).
 

@@ -14,9 +14,10 @@ categories:
   - Other
 enabledBy:
 enables:
+tags: [dagster-supported]
 ---
 
-### About this integration
+
 
 The Dagster AWS Systems Manager (SSM) Parameter Store integration allows you to manage and retrieve parameters stored in AWS SSM Parameter Store directly within your Dagster pipelines. This integration provides resources to fetch parameters by name, tags, or paths, and optionally set them as environment variables for your operations.
 

@@ -14,9 +14,10 @@ categories:
   - Storage
 enabledBy:
 enables:
+tags: [dagster-supported, storage]
 ---
 
-### About this integration
+
 
 Dagster helps you use Azure Storage Accounts as part of your data pipeline. Azure Data Lake Storage Gen 2 (ADLS2) is our primary focus but we also provide utilities for Azure Blob Storage.
 

@@ -14,9 +14,10 @@ categories:
   - ETL
 enabledBy:
 enables:
+tags: [community-supported, etl]
 ---
 
-### About this integration
+
 
 With the `dagster-census` integration you can execute a Census sync and poll until that sync completes, raising an error if it's unsuccessful.
 

@@ -14,9 +14,10 @@ categories:
   - Other
 enabledBy:
 enables:
+tags: [community-supported]
 ---
 
-### About this integration
+
 
 With the `dagster_cube` integration you can setup Cube and Dagster to work together so that Dagster can push changes from upstream data sources to Cube using its integration API.
 

@@ -14,9 +14,10 @@ categories:
   - Compute
 enabledBy:
 enables:
+tags: [dagster-supported, compute]
 ---
 
-### About this integration
+
 
 The `dagster-databricks` integration library provides the `PipesDatabricksClient` resource, enabling you to launch Databricks jobs directly from Dagster assets and ops. This integration allows you to pass parameters to Databricks code while Dagster receives real-time events, such as logs, asset checks, and asset materializations, from the initiated jobs. With minimal code changes required on the job side, this integration is both efficient and easy to implement.
 

@@ -14,9 +14,10 @@ categories:
   - Monitoring
 enabledBy:
 enables:
+tags: [dagster-supported, monitoring]
 ---
 
-### About this integration
+
 
 While Dagster provides comprehensive monitoring and observability of the pipelines it orchestrates, many teams look to centralize all their monitoring across apps, processes and infrastructure using Datadog's 'Cloud Monitoring as a Service'. The `dagster-datadog` integration allows you to publish metrics to Datadog from within Dagster ops.
 

@@ -14,9 +14,10 @@ categories:
   - ETL
 enabledBy:
 enables:
+tags: [dagster-supported, etl]
 ---
 
-### About this integration
+
 
 Dagster allows you to run dbt Cloud jobs alongside other technologies. You can schedule them to run as a step in a larger pipeline and manage them as a data asset.
 

@@ -14,9 +14,10 @@ categories:
   - ETL
 enabledBy:
 enables:
+tags: [dagster-supported, etl]
 ---
 
-### About this integration
+
 
 Dagster orchestrates dbt alongside other technologies, so you can schedule dbt with Spark, Python, etc. in a single data pipeline.
 

@@ -15,9 +15,10 @@ categories:
   - Storage
 enabledBy:
 enables:
+tags: [community-supported, storage]
 ---
 
-### About this integration
+
 
 Delta Lake is a great storage format for Dagster workflows. With this integration, you can use the Delta Lake I/O Manager to read and write your Dagster assets.
 

@@ -14,9 +14,10 @@ categories:
   - ETL
 enabledBy:
 enables:
+tags: [dagster-supported, etl]
 ---
 
-### About this integration
+
 
 This integration allows you to use [dlt](https://dlthub.com/) to easily ingest and replicate data between systems through Dagster.
 

@@ -14,9 +14,10 @@ categories:
   - Compute
 enabledBy:
 enables:
+tags: [dagster-supported, compute]
 ---
 
-### About this integration
+
 
 The `dagster-docker` integration library provides the `PipesDockerClient` resource, enabling you to launch Docker containers and execute external code directly from Dagster assets and ops. This integration allows you to pass parameters to Docker containers while Dagster receives real-time events, such as logs, asset checks, and asset materializations, from the initiated jobs. With minimal code changes required on the job side, this integration is both efficient and easy to implement.
 

@@ -14,9 +14,10 @@ categories:
   - Storage
 enabledBy:
 enables:
+tags: [dagster-supported, storage]
 ---
 
-### About this integration
+
 
 This library provides an integration with the DuckDB database, and allows for an out-of-the-box [I/O Manager](https://docs.dagster.io/concepts/io-management/io-managers) so that you can make DuckDB your storage of choice.
 

@@ -14,9 +14,10 @@ categories:
   - ETL
 enabledBy:
 enables:
+tags: [dagster-supported, etl]
 ---
 
-### About this integration
+
 
 The Dagster-Fivetran integration enables you to orchestrate data ingestion as part of a larger pipeline. Programmatically interact with the Fivetran REST API to initiate syncs and monitor their progress.
 

@@ -14,9 +14,10 @@ categories:
   - Storage
 enabledBy:
 enables:
+tags: [dagster-supported, storage]
 ---
 
-### About this integration
+
 
 The Google Cloud Platform BigQuery integration allows data engineers to easily query and store data in the BigQuery data warehouse through the use of the `BigQueryResource`.
 

@@ -14,9 +14,10 @@ categories:
   - Compute
 enabledBy:
 enables:
+tags: [dagster-supported, compute]
 ---
 
-### About this integration
+
 
 Using this integration, you can manage and interact with Google Cloud Platform's Dataproc service directly from Dagster. This integration allows you to create, manage, and delete Dataproc clusters, and submit and monitor jobs on these clusters.
 

@@ -14,9 +14,10 @@ categories:
   - Storage
 enabledBy:
 enables:
+tags: [dagster-supported, storage]
 ---
 
-### About this integration
+
 
 This integration allows you to interact with Google Cloud Storage (GCS) using Dagster. It provides resources, I/O Managers, and utilities to manage and store data in GCS, making it easier to integrate GCS into your data pipelines.
 

@@ -14,9 +14,10 @@ categories:
   - Other
 enabledBy:
 enables:
+tags: [dagster-supported]
 ---
 
-### About this integration
+
 
 This library provides an integration with _[GitHub Apps](https://docs.github.com/en/developers/apps/getting-started-with-apps/about-apps)_ by providing a thin wrapper on the GitHub v4 GraphQL API. This allows for automating operations within your GitHub repositories and with the tighter permissions scopes that GitHub Apps allow for vs using a personal token.
 

@@ -15,9 +15,10 @@ categories:
   - Other
 enabledBy:
 enables:
+tags: [community-supported]
 ---
 
-### About this integration
+
 
 Package for integrating HashiCorp Vault into Dagster so that you can securely manage tokens and passwords.
Original file line number	Diff line number	Diff line change
Expand Up		@@ -4,4 +4,4 @@ title: AWS

		import DocCardList from '@theme/DocCardList';

		<DocCardList />
		<DocCardList />