Skip to content

Commit

Permalink
[docs-beta] swizzle DocCardList for better integration index (#27035)
Browse files Browse the repository at this point in the history
## Summary & Motivation

**Before**
<img width="1874" alt="image"
src="https://github.com/user-attachments/assets/81d6b8df-7e3f-44b8-9fab-a294da3ebd54"
/>

**After**
<img width="1868" alt="image"
src="https://github.com/user-attachments/assets/1ee9ccef-3744-4258-a60f-1ee07e631219"
/>


## How I Tested These Changes

## Changelog

NOCHANGELOG
  • Loading branch information
cmpadden authored Jan 12, 2025
1 parent 012a428 commit fcc708f
Show file tree
Hide file tree
Showing 135 changed files with 1,042 additions and 136 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ categories:
enabledBy:
enables:
tags: [dagster-supported, etl]
sidebar_custom_props:
logo: images/integrations/airbyte.svg
---

This guide provides instructions for using Dagster with Airbyte Cloud using the `dagster-airbyte` library. Your Airbyte Cloud connection tables can be represented as assets in the Dagster asset graph, allowing you to track lineage and dependencies between Airbyte Cloud assets and data assets you are already modeling in Dagster. You can also use Dagster to orchestrate Airbyte Cloud connections, allowing you to trigger syncs for these on a cadence or based on upstream data changes.
Expand Down
5 changes: 2 additions & 3 deletions docs/docs-beta/docs/integrations/libraries/airbyte/airbyte.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,16 +9,15 @@ date: 2022-11-07
apireflink: https://docs.dagster.io/_apidocs/libraries/dagster-airbyte
docslink: https://docs.dagster.io/integrations/airbyte
partnerlink: https://airbyte.com/tutorials/orchestrate-data-ingestion-and-transformation-pipelines
logo: /integrations/airbyte.svg
categories:
- ETL
enabledBy:
enables:
tags: [dagster-supported, etl]
sidebar_custom_props:
logo: images/integrations/airbyte.svg
---



Using this integration, you can trigger Airbyte syncs and orchestrate your Airbyte connections from within Dagster, making it easy to chain an Airbyte sync with upstream or downstream steps in your workflow.

### Installation
Expand Down
5 changes: 2 additions & 3 deletions docs/docs-beta/docs/integrations/libraries/aws/athena.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,16 +9,15 @@ date: 2024-06-21
apireflink: https://docs.dagster.io/_apidocs/libraries/dagster-aws
docslink:
partnerlink: https://aws.amazon.com/
logo: /integrations/aws-athena.svg
categories:
- Storage
enabledBy:
enables:
tags: [dagster-supported, storage]
sidebar_custom_props:
logo: images/integrations/aws-athena.svg
---



This integration allows you to connect to AWS Athena, a serverless interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. Using this integration, you can issue queries to Athena, fetch results, and handle query execution states within your Dagster pipelines.

### Installation
Expand Down
5 changes: 2 additions & 3 deletions docs/docs-beta/docs/integrations/libraries/aws/cloudwatch.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,16 +9,15 @@ date: 2024-06-21
apireflink: https://docs.dagster.io/_apidocs/libraries/dagster-aws
docslink:
partnerlink: https://aws.amazon.com/
logo: /integrations/aws-cloudwatch.svg
categories:
- Monitoring
enabledBy:
enables:
tags: [dagster-supported, monitoring]
sidebar_custom_props:
logo: images/integrations/aws-cloudwatch.svg
---



This integration allows you to send Dagster logs to AWS CloudWatch, enabling centralized logging and monitoring of your Dagster jobs. By using AWS CloudWatch, you can take advantage of its powerful log management features, such as real-time log monitoring, log retention policies, and alerting capabilities.

Using this integration, you can configure your Dagster jobs to log directly to AWS CloudWatch, making it easier to track and debug your workflows. This is particularly useful for production environments where centralized logging is essential for maintaining observability and operational efficiency.
Expand Down
5 changes: 2 additions & 3 deletions docs/docs-beta/docs/integrations/libraries/aws/ecr.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,16 +9,15 @@ date: 2024-06-21
apireflink: https://docs.dagster.io/_apidocs/libraries/dagster-aws
docslink:
partnerlink: https://aws.amazon.com/
logo: /integrations/aws-ecr.svg
categories:
- Other
enabledBy:
enables:
tags: [dagster-supported]
sidebar_custom_props:
logo: images/integrations/aws-ecr.svg
---



This integration allows you to connect to AWS Elastic Container Registry (ECR). It provides resources to interact with AWS ECR, enabling you to manage your container images.

Using this integration, you can seamlessly integrate AWS ECR into your Dagster pipelines, making it easier to manage and deploy containerized applications.
Expand Down
5 changes: 2 additions & 3 deletions docs/docs-beta/docs/integrations/libraries/aws/emr.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,16 +9,15 @@ date: 2024-06-21
apireflink: https://docs.dagster.io/_apidocs/libraries/dagster-aws
docslink:
partnerlink: https://aws.amazon.com/
logo: /integrations/aws-emr.svg
categories:
- Compute
enabledBy:
enables:
tags: [dagster-supported, compute]
sidebar_custom_props:
logo: images/integrations/aws-emr.svg
---



The `dagster-aws` integration provides ways orchestrating data pipelines that leverage AWS services, including AWS EMR (Elastic MapReduce). This integration allows you to run and scale big data workloads using open source tools such as Apache Spark, Hive, Presto, and more.

Using this integration, you can:
Expand Down
4 changes: 2 additions & 2 deletions docs/docs-beta/docs/integrations/libraries/aws/glue.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,10 @@ categories:
enabledBy:
enables:
tags: [dagster-supported, compute]
sidebar_custom_props:
logo: images/integrations/aws-glue.svg
---



The `dagster-aws` integration library provides the `PipesGlueClient` resource, enabling you to launch AWS Glue jobs directly from Dagster assets and ops. This integration allows you to pass parameters to Glue code while Dagster receives real-time events, such as logs, asset checks, and asset materializations, from the initiated jobs. With minimal code changes required on the job side, this integration is both efficient and easy to implement.

### Installation
Expand Down
4 changes: 3 additions & 1 deletion docs/docs-beta/docs/integrations/libraries/aws/index.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,9 @@
---
title: AWS
sidebar_custom_props:
logo: images/integrations/aws.svg
---

import DocCardList from '@theme/DocCardList';

<DocCardList />
<DocCardList />
4 changes: 2 additions & 2 deletions docs/docs-beta/docs/integrations/libraries/aws/lambda.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,10 @@ categories:
enabledBy:
enables:
tags: [dagster-supported, compute]
sidebar_custom_props:
logo: images/integrations/aws-lambda.svg
---



Using this integration, you can leverage AWS Lambda to execute external code as part of your Dagster pipelines. This is particularly useful for running serverless functions that can scale automatically and handle various workloads without the need for managing infrastructure. The `PipesLambdaClient` class allows you to invoke AWS Lambda functions and stream logs and structured metadata back to Dagster's UI and tools.

### Installation
Expand Down
4 changes: 2 additions & 2 deletions docs/docs-beta/docs/integrations/libraries/aws/redshift.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,10 @@ categories:
enabledBy:
enables:
tags: [dagster-supported, storage]
sidebar_custom_props:
logo: images/integrations/aws-redshift.svg
---



Using this integration, you can connect to an AWS Redshift cluster and issue queries against it directly from your Dagster assets. This allows you to seamlessly integrate Redshift into your data pipelines, leveraging the power of Redshift's data warehousing capabilities within your Dagster workflows.

### Installation
Expand Down
4 changes: 2 additions & 2 deletions docs/docs-beta/docs/integrations/libraries/aws/s3.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,10 @@ categories:
enabledBy:
enables:
tags: [dagster-supported, storage]
sidebar_custom_props:
logo: images/integrations/aws-s3.svg
---



The AWS S3 integration allows data engineers to easily read, and write objects to the durable AWS S3 storage -- enabling engineers to a resilient storage layer when constructing their pipelines.

### Installation
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,16 +9,15 @@ date: 2024-06-21
apireflink: https://docs.dagster.io/_apidocs/libraries/dagster-aws
docslink:
partnerlink: https://aws.amazon.com/
logo: /integrations/aws-secretsmanager.svg
categories:
- Other
enabledBy:
enables:
tags: [dagster-supported]
sidebar_custom_props:
logo: images/integrations/aws-secretsmanager.svg
---



This integration allows you to manage, retrieve, and rotate credentials, API keys, and other secrets using [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/).

### Installation
Expand Down
5 changes: 2 additions & 3 deletions docs/docs-beta/docs/integrations/libraries/aws/ssm.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,16 +9,15 @@ date: 2024-06-21
apireflink: https://docs.dagster.io/_apidocs/libraries/dagster-aws
docslink:
partnerlink: https://aws.amazon.com/
logo: /integrations/aws-ssm.svg
categories:
- Other
enabledBy:
enables:
tags: [dagster-supported]
sidebar_custom_props:
logo: images/integrations/aws-ssm.svg
---



The Dagster AWS Systems Manager (SSM) Parameter Store integration allows you to manage and retrieve parameters stored in AWS SSM Parameter Store directly within your Dagster pipelines. This integration provides resources to fetch parameters by name, tags, or paths, and optionally set them as environment variables for your operations.

### Installation
Expand Down
9 changes: 4 additions & 5 deletions docs/docs-beta/docs/integrations/libraries/azure-adls2.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,8 @@
---
layout: Integration
status: published
name: Azure Data Lake Storage Gen 2 (ADLS2)
title: Dagster & Azure Data Lake Storage Gen 2 (ADLS2)
sidebar_label: Azure Data Lake Storage Gen 2 (ADLS2)
title: Dagster & Azure Data Lake Storage Gen 2
sidebar_label: Azure Data Lake Storage Gen 2
excerpt: Get utilities for ADLS2 and Blob Storage.
date: 2022-11-07
apireflink: https://docs.dagster.io/_apidocs/libraries/dagster-azure
Expand All @@ -15,10 +14,10 @@ categories:
enabledBy:
enables:
tags: [dagster-supported, storage]
sidebar_custom_props:
logo: images/integrations/azure.svg
---



Dagster helps you use Azure Storage Accounts as part of your data pipeline. Azure Data Lake Storage Gen 2 (ADLS2) is our primary focus but we also provide utilities for Azure Blob Storage.

### Installation
Expand Down
4 changes: 2 additions & 2 deletions docs/docs-beta/docs/integrations/libraries/census.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,10 @@ categories:
enabledBy:
enables:
tags: [community-supported, etl]
sidebar_custom_props:
logo: images/integrations/census.svg
---



With the `dagster-census` integration you can execute a Census sync and poll until that sync completes, raising an error if it's unsuccessful.

### Installation
Expand Down
4 changes: 2 additions & 2 deletions docs/docs-beta/docs/integrations/libraries/cube.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,10 @@ categories:
enabledBy:
enables:
tags: [community-supported]
sidebar_custom_props:
logo: images/integrations/cube.svg
---



With the `dagster_cube` integration you can setup Cube and Dagster to work together so that Dagster can push changes from upstream data sources to Cube using its integration API.

### Installation
Expand Down
4 changes: 2 additions & 2 deletions docs/docs-beta/docs/integrations/libraries/databricks.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,10 @@ categories:
enabledBy:
enables:
tags: [dagster-supported, compute]
sidebar_custom_props:
logo: images/integrations/databricks.svg
---



The `dagster-databricks` integration library provides the `PipesDatabricksClient` resource, enabling you to launch Databricks jobs directly from Dagster assets and ops. This integration allows you to pass parameters to Databricks code while Dagster receives real-time events, such as logs, asset checks, and asset materializations, from the initiated jobs. With minimal code changes required on the job side, this integration is both efficient and easy to implement.

### Installation
Expand Down
4 changes: 2 additions & 2 deletions docs/docs-beta/docs/integrations/libraries/datadog.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,10 @@ categories:
enabledBy:
enables:
tags: [dagster-supported, monitoring]
sidebar_custom_props:
logo: images/integrations/datadog.svg
---



While Dagster provides comprehensive monitoring and observability of the pipelines it orchestrates, many teams look to centralize all their monitoring across apps, processes and infrastructure using Datadog's 'Cloud Monitoring as a Service'. The `dagster-datadog` integration allows you to publish metrics to Datadog from within Dagster ops.

### Installation
Expand Down
5 changes: 2 additions & 3 deletions docs/docs-beta/docs/integrations/libraries/dbt-cloud.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,16 +9,15 @@ date: 2022-11-07
apireflink: https://docs.dagster.io/_apidocs/libraries/dagster-dbt#assets-dbt-cloud
docslink: https://docs.dagster.io/integrations/dbt_cloud
partnerlink:
logo: /integrations/dbt.svg
categories:
- ETL
enabledBy:
enables:
tags: [dagster-supported, etl]
sidebar_custom_props:
logo: images/integrations/dbt.svg
---



Dagster allows you to run dbt Cloud jobs alongside other technologies. You can schedule them to run as a step in a larger pipeline and manage them as a data asset.

### Installation
Expand Down
4 changes: 2 additions & 2 deletions docs/docs-beta/docs/integrations/libraries/dbt/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,10 @@ categories:
enabledBy:
enables:
tags: [dagster-supported, etl]
sidebar_custom_props:
logo: images/integrations/dbt.svg
---



Dagster orchestrates dbt alongside other technologies, so you can schedule dbt with Spark, Python, etc. in a single data pipeline.

Dagster assets understand dbt at the level of individual dbt models. This means that you can:
Expand Down
5 changes: 2 additions & 3 deletions docs/docs-beta/docs/integrations/libraries/deltalake.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,16 +10,15 @@ communityIntegration: true
apireflink: https://delta-io.github.io/delta-rs/integrations/delta-lake-dagster/
docslink:
partnerlink: https://delta.io/
logo: /integrations/DeltaLake.svg
categories:
- Storage
enabledBy:
enables:
tags: [community-supported, storage]
sidebar_custom_props:
logo: images/integrations/deltalake.svg
---



Delta Lake is a great storage format for Dagster workflows. With this integration, you can use the Delta Lake I/O Manager to read and write your Dagster assets.

Here are some of the benefits that Delta Lake provides Dagster users:
Expand Down
4 changes: 2 additions & 2 deletions docs/docs-beta/docs/integrations/libraries/dlt.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,10 @@ categories:
enabledBy:
enables:
tags: [dagster-supported, etl]
sidebar_custom_props:
logo: images/integrations/dlthub.jpeg
---



This integration allows you to use [dlt](https://dlthub.com/) to easily ingest and replicate data between systems through Dagster.

### Installation
Expand Down
4 changes: 2 additions & 2 deletions docs/docs-beta/docs/integrations/libraries/docker.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,10 +15,10 @@ categories:
enabledBy:
enables:
tags: [dagster-supported, compute]
sidebar_custom_props:
logo: images/integrations/docker.svg
---



The `dagster-docker` integration library provides the `PipesDockerClient` resource, enabling you to launch Docker containers and execute external code directly from Dagster assets and ops. This integration allows you to pass parameters to Docker containers while Dagster receives real-time events, such as logs, asset checks, and asset materializations, from the initiated jobs. With minimal code changes required on the job side, this integration is both efficient and easy to implement.

### Installation
Expand Down
4 changes: 2 additions & 2 deletions docs/docs-beta/docs/integrations/libraries/duckdb.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,15 +9,15 @@ date: 2022-11-07
apireflink: https://docs.dagster.io/_apidocs/libraries/dagster-duckdb
docslink: https://dagster.io/blog/duckdb-data-lake
partnerlink: https://duckdb.org/
logo: /integrations/Duckdb.svg
categories:
- Storage
enabledBy:
enables:
tags: [dagster-supported, storage]
sidebar_custom_props:
logo: images/integrations/duckdb.svg
---

{/* This library provides an integration with the DuckDB database, and allows for an out-of-the-box [I/O Manager](/concepts/io-management/io-managers) so that you can make DuckDB your storage of choice. */}
This library provides an integration with the DuckDB database, and allows for an out-of-the-box [I/O Manager](/todo) so that you can make DuckDB your storage of choice.

### Installation
Expand Down
2 changes: 2 additions & 0 deletions docs/docs-beta/docs/integrations/libraries/fivetran.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ categories:
enabledBy:
enables:
tags: [dagster-supported, etl]
sidebar_custom_props:
logo: images/integrations/fivetran.svg
---

This guide provides instructions for using Dagster with Fivetran using the `dagster-fivetran` library. Your Fivetran connector tables can be represented as assets in the Dagster asset graph, allowing you to track lineage and dependencies between Fivetran assets and data assets you are already modeling in Dagster. You can also use Dagster to orchestrate Fivetran connectors, allowing you to trigger syncs for these on a cadence or based on upstream data changes.
Expand Down
Loading

1 comment on commit fcc708f

@github-actions
Copy link

@github-actions github-actions bot commented on fcc708f Jan 12, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deploy preview for dagster-docs-beta ready!

✅ Preview
https://dagster-docs-beta-dspsaby87-elementl.vercel.app

Built with commit fcc708f.
This pull request is being automatically deployed with vercel-action

Please sign in to comment.