Skip to content

Commit

Permalink
more links
Browse files Browse the repository at this point in the history
Signed-off-by: nikki everett <[email protected]>
  • Loading branch information
neverett committed Dec 19, 2024
1 parent 59419ac commit b731556
Show file tree
Hide file tree
Showing 11 changed files with 16 additions and 16 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,11 @@ The default I/O manager cannot be used if you are a Serverless user who:
- Are otherwise working with data subject to GDPR or other such regulations
:::

In Serverless, code that uses the default [I/O manager](/guides/build/configure/io-managers) is automatically adjusted to save data in Dagster+ managed storage. This automatic change is useful because the Serverless filesystem is ephemeral, which means the default I/O manager wouldn't work as expected.
In Serverless, code that uses the default [I/O manager](/guides/build/connect-to-external-systems/io-managers) is automatically adjusted to save data in Dagster+ managed storage. This automatic change is useful because the Serverless filesystem is ephemeral, which means the default I/O manager wouldn't work as expected.

However, this automatic change also means potentially sensitive data could be **stored** and not just processed or orchestrated by Dagster+.

To prevent this, you can use [another I/O manager](/guides/build/configure/io-managers#built-in) that stores data in your infrastructure or [adapt your code to avoid using an I/O manager](/guides/build/configure/io-managers#before-you-begin).
To prevent this, you can use [another I/O manager](/guides/build/connect-to-external-systems/io-managers#built-in) that stores data in your infrastructure or [adapt your code to avoid using an I/O manager](/guides/build/connect-to-external-systems/io-managers#before-you-begin).

:::note
You must have [boto3](https://pypi.org/project/boto3/) or `dagster-cloud[serverless]` installed as a project dependency otherwise the Dagster+ managed storage can fail and silently fall back to using the default I/O manager.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ When building a data pipeline, you'll likely need to connect to several external
To follow the steps in this guide, you'll need:

- Familiarity with [Assets](/guides/build/create-asset-pipelines/assets-concepts/index.mdx
- Familiarity with [Resources](/guides/build/configure/resources)
- Familiarity with [Resources](/guides/build/connect-to-external-systems/resources)
- To install the `requests` library:
```bash
pip install requests
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -57,5 +57,5 @@ You can also fetch environment variables using the `os` library. Dagster treats

## Next steps

- Explore how to use resources for [Connecting to APIs](/guides/build/configure/apis)
- Go deeper into [Understanding Resources](/guides/build/configure/resources)
- Explore how to use resources for [Connecting to APIs](/guides/build/connect-to-external-systems/apis)
- Go deeper into [Understanding Resources](/guides/build/connect-to-external-systems/resources)
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ Using an asset factory reduces complexity and creates a pluggable entry point to
<details>
<summary>Prerequisites</summary>

This guide builds upon the concepts in the [asset factories](/guides/build/configure/asset-factories) tutorial.
This guide builds upon the concepts in the [asset factories](/guides/build/connect-to-external-systems/asset-factories) tutorial.
</details>

---
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,8 +19,8 @@ The asset factory pattern can solve both of these problems.
To follow the steps in this guide, you'll need:

- Familiarity with:
- [Assets](/guides/build/create-asset-pipelinea/data-assets)
- [Resources](/guides/build/configure/resources)
- [Assets](/guides/build/create-asset-pipelines/data-assets)
- [Resources](/guides/build/connect-to-external-systems/resources)
- SQL, YAML and Amazon Web Services (AWS) S3
- [Pydantic](https://docs.pydantic.dev/latest/) and [Jinja2](https://jinja.palletsprojects.com/en/3.1.x/)
- A Python virtual environment with the following dependencies installed:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -45,4 +45,4 @@ This will open the Launchpad, where you can scaffold the config, customize its v
## Next steps

- Learn more about Dagster [assets](/guides/build/create-asset-pipelines/assets-concepts/index.mdx
- Connect to external [APIs](/guides/build/configure/apis) and [databases](/guides/build/configure/databases) with resources
- Connect to external [APIs](/guides/build/connect-to-external-systems/apis) and [databases](/guides/build/connect-to-external-systems/databases) with resources
Original file line number Diff line number Diff line change
Expand Up @@ -73,4 +73,4 @@ flowchart LR

- Learn to create [dependencies between assets](/guides/build/create-asset-pipelines/assets-concepts/asset-dependencies)
- Enrich Dagster's built-in data catalog with [asset metadata](/guides/build/create-asset-pipelines/metadata)
- Learn to use a [factory pattern](/guides/build/configure/asset-factories) to create multiple, similar assets
- Learn to use a [factory pattern](/guides/build/connect-to-external-systems/asset-factories) to create multiple, similar assets
Original file line number Diff line number Diff line change
Expand Up @@ -134,7 +134,7 @@ Dagster can automatically attach code references to assets during local developm

### Customizing code references \{#custom-references}

If you want to customize how code references are attached - such as when you are building [domain-specific languages with asset factories](/guides/build/configure/asset-factories) - you can manually add the `dagster/code_references` metadata to asset definitions:
If you want to customize how code references are attached - such as when you are building [domain-specific languages with asset factories](/guides/build/connect-to-external-systems/asset-factories) - you can manually add the `dagster/code_references` metadata to asset definitions:

<CodeExample filePath="guides/data-modeling/metadata/custom-local-references.py" language="python" />

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -48,5 +48,5 @@ For example, imagine there's a CSV file of counties on the internet and you want

## Next steps

- Transform data using [Dagster's dbt integration](/guides/build/integrate/transform-dbt)
- Transform data using [Dagster's dbt integration](/guides/build/integrate-with-external-systems/transform-dbt)
- Use asset checks [to test data quality](/guides/test/asset-checks) and [freshness](/guides/test/data-freshness-testing)
6 changes: 3 additions & 3 deletions docs/docs-beta/docs/guides/operate/io-managers.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ For assets that follow this pattern, an I/O manager can streamline the code that
To follow the steps in this guide, you'll need familiarity with:

- [Assets](/guides/build/create-asset-pipelines/assets-concepts/index.mdx
- [Resources](/guides/build/configure/resources)
- [Resources](/guides/build/connect-to-external-systems/resources)
</details>

## Before you begin
Expand Down Expand Up @@ -82,5 +82,5 @@ Dagster offers built-in library implementations for I/O managers for popular dat

## Next steps

- Learn to [connect databases](/guides/build/configure/databases) with resources
- Learn to [connect APIs](/guides/build/configure/apis) with resources
- Learn to [connect databases](/guides/build/connect-to-external-systems/databases) with resources
- Learn to [connect APIs](/guides/build/connect-to-external-systems/apis) with resources
Original file line number Diff line number Diff line change
Expand Up @@ -108,4 +108,4 @@ If an asset or op has multiple parameters, it's recommended to use keyword argum

- Learn more about assets in [Understanding Assets](/guides/build/create-asset-pipelines/assets-concepts/index.mdx
- Learn more about ops in [Understanding Ops](/guides/build/ops-jobs)
- Learn more about resources in [Resources](/guides/build/configure/resources)
- Learn more about resources in [Resources](/guides/build/connect-to-external-systems/resources)

0 comments on commit b731556

Please sign in to comment.