Skip to content

Commit

Permalink
Add serverless io manager appropriate usage for baa (#22544)
Browse files Browse the repository at this point in the history
## Summary & Motivation

Explain how the default filesystem IO manager is replaced in Serverless
which could surprise users. Also address that this replacement is not
appropriate for sensitive data, specifically warning certain customers
(like those with a BAA) that appropriate use of the system forgoes using
the default serverless io manager.

## How I Tested These Changes
- reviewed the docs preview and link

---------

Co-authored-by: Erin Cochran <[email protected]>
  • Loading branch information
slopp and erinkcochran87 authored Jun 14, 2024
1 parent 8dcec5d commit 842f19a
Showing 1 changed file with 21 additions and 1 deletion.
22 changes: 21 additions & 1 deletion docs/content/dagster-plus/deployment/serverless.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -306,11 +306,31 @@ After changing the deployment type, you will need to update your code locations'

Unlike Hybrid, Serverless Deployments on Dagster+ require direct access to your data, secrets and source code.

- Dagster+ Serverless does not provide persistent storage. Ephemeral storage is deleted when a run concludes.
- Secrets and source code are built into the image directly. Images are stored in a per-customer container registry with restricted access.
- User code is securely sandboxed using modern container sandboxing techniques.
- All production access is governed by industry-standard best practices which are regularly audited.

### I/O management on Serverless

<Note>
The default I/O manager cannot be used if you are a Serverless user who:
<ul>
<li>Works with personally identifiable information (PII)</li>
<li>Works with private health information (PHI)</li>
<li>Has signed a business association agreement (BAA), or</li>
<li>
Are otherwise working with data subject to GDPR or other such regulations
</li>
</ul>
</Note>{" "}

In Serverless, code that uses the default [I/O manager](/concepts/io-management/io-managers#built-in-io-managers) is automatically adjusted to save data in Dagster+ managed storage. This automatic change is useful because the default file system in Serverless is ephemeral, which means the default I/O manager wouldn't work as expected. However, this automatic change means potentially sensitive data is being stored, not just processed or orchestrated, by Dagster+.

To avoid this behavior, you can:

- Use an I/O manager that stores data in your infrastructure
- Write code that doesn't use an I/O manager

---

## Whitelisting Dagster's IP addresses
Expand Down

1 comment on commit 842f19a

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Deploy preview for dagster-docs ready!

✅ Preview
https://dagster-docs-moay0sgqa-elementl.vercel.app
https://master.dagster.dagster-docs.io

Built with commit 842f19a.
This pull request is being automatically deployed with vercel-action

Please sign in to comment.