-
Notifications
You must be signed in to change notification settings - Fork 1.6k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[docs-beta] Refresh Dagster+ Serverless guide (#25563)
## Summary & Motivation [DOCS-545](https://linear.app/dagster-labs/issue/DOC-545/[docs]-refresh-dagster-serverless-guide) - Ported missing content from original Serverless docs - Extended content in some pages/sections based on feedback from users and support cases ## How I Tested These Changes 👀 --------- Co-authored-by: colton <[email protected]>
- Loading branch information
1 parent
910f5c5
commit 6a46878
Showing
6 changed files
with
295 additions
and
38 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,8 +1,43 @@ | ||
--- | ||
title: "Serverless deployment" | ||
displayed_sidebar: "dagsterPlus" | ||
title: 'Serverless deployment' | ||
displayed_sidebar: 'dagsterPlus' | ||
sidebar_position: 1 | ||
unlisted: true | ||
--- | ||
|
||
# Serverless deployment | ||
|
||
Dagster+ Serverless is a fully managed version of Dagster+ and is the easiest way to get started with Dagster. With a Serverless deployment, you can run your Dagster jobs without spinning up any infrastructure yourself. | ||
|
||
--- | ||
|
||
## When to choose Serverless {#when-to-choose-serverless} | ||
|
||
Serverless works best with workloads that primarily orchestrate other services or perform light computation. Most workloads fit into this category, especially those that orchestrate third-party SaaS products like cloud data warehouses and ETL tools. | ||
|
||
If any of the following are applicable, you should select [Hybrid deployment](/dagster-plus/deployment/hybrid): | ||
|
||
- You require substantial computational resources. For example, training a large machine learning (ML) model in-process | ||
- Your dataset is too large to fit in memory. For example, training a large ML model in-process on a terabyte of data | ||
- You need to distribute computation across many nodes for a single run. Dagster+ runs currently execute on a single node with 4 CPUs | ||
- You don't want to add Dagster Labs as a data processor | ||
|
||
--- | ||
|
||
## Limitations {#limitations} | ||
|
||
Serverless is subject to the following limitations: | ||
|
||
- Maximum of 100 GB of bandwidth per day | ||
- Maximum of 4500 step-minutes per day | ||
- Runs receive 4 vCPU cores, 16 GB of RAM, and 128 GB of ephemeral disk | ||
- Code locations receive 0.25 vCPU cores and 1 GB of RAM | ||
- All Serverless jobs run in the United States | ||
- Infrastructure cannot be customized or extended, such as using additional containers | ||
|
||
Dagster+ Pro customers may request a quota increase by [contacting Sales](https://dagster.io/contact). | ||
|
||
--- | ||
|
||
## Next steps | ||
|
||
To start using Dagster+ Serverless, follow our [Getting started with Dagster+](/dagster-plus/getting-started) guide. |
90 changes: 90 additions & 0 deletions
90
docs/docs-beta/docs/dagster-plus/deployment/serverless/ci-cd-in-serverless.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,90 @@ | ||
--- | ||
title: 'CI/CD in Serverless ' | ||
displayed_sidebar: 'dagsterPlus' | ||
sidebar_position: 10 | ||
--- | ||
|
||
# CI/CD in Serverless | ||
|
||
If you're a GitHub or GitLab user, you can use our predefined workflows to seamlessly deploy and synchronize your code to Dagster+. You can also use other Git providers or a local Git repository with our [dagster-cloud CLI](/dagster-plus/deployment/branch-deployments/dagster-cloud-cli) to run your own CI/CD process. | ||
|
||
<Tabs groupId="method"> | ||
<TabItem value="GitHub" label="With GitHub"> | ||
|
||
If you're a GitHub user, with a single click our GitHub app with GitHub Actions can set up a repo containing skeleton code and configuration for you consistent with Dagster+'s best practices. Pushing your code changes to the `main` branch will automatically deploy them to your `prod` Serverless deployment. Pull requests will spin up ephemeral [branch deployments](/dagster-plus/deployment/branch-deployments) that you can view in the Dagster+ UI for previewing and testing. | ||
|
||
:::note | ||
**If you are importing a Dagster project that's in an existing GitHub repo:** | ||
|
||
- The repo will need to allow the [Workflow permission](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository) for `Read and write permissions`. Workflow permissions settings can be found in GitHub's `Settings` > `Actions` > `General` > `Workflow permissions`. In GitHub Enterprise, these permissions [are controlled at the Organization level](https://github.com/orgs/community/discussions/57244). | ||
|
||
- An initial commit will need to be able to be merged directly to the repo's `main` branch to automatically add the GitHub Actions workflow files. If [branch protection rules](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches#about-protected-branches) require changes be done through a pull request, it will prevent the automatic setup from completing. | ||
|
||
- You can temporarily disable the branch protection rules and then re-enable them after the automatic setup completes. Alternatively, you can manually set up the GitHub Actions workflows. You can use our [dagster-cloud-serverless-quickstart repo](https://github.com/dagster-io/dagster-cloud-serverless-quickstart) as a template and its [README](https://github.com/dagster-io/dagster-cloud-serverless-quickstart/blob/main/README.md) as a guide. | ||
|
||
::: | ||
|
||
</TabItem> | ||
|
||
<TabItem value="GitLab" label="With GitLab"> | ||
|
||
If you're a GitLab user, with a single click our GitLab app can set up a repo containing skeleton code and CI/CD configuration for you consistent with Dagster+'s best practices. Pushing your code changes to the `main` branch will automatically deploy them to your `prod` Serverless deployment. Pull requests will spin up ephemeral [branch deployments](/dagster-plus/deployment/branch-deployments) that you can view in the Dagster+ UI for previewing and testing. | ||
|
||
</TabItem> | ||
|
||
<TabItem value="Other" label="Other Git providers or local development"> | ||
|
||
If you don't want to use our automated GitHub/GitLab process, we offer [the powerful `dagster-cloud` command-line interface (CLI)](/dagster-plus/deployment/branch-deployments/dagster-cloud-cli) that you can use in another CI environment or locally. | ||
|
||
First, [create a new project](/getting-started/quickstart) with the Dagster open source CLI. | ||
|
||
The example below uses our [quickstart_etl example project](https://github.com/dagster-io/dagster/tree/master/examples/quickstart_etl). For more info about the examples, visit the [Dagster GitHub repository](https://github.com/dagster-io/dagster/tree/master/examples). | ||
|
||
```shell | ||
pip install dagster | ||
dagster project from-example \ | ||
--name my-dagster-project \ | ||
--example quickstart_etl | ||
``` | ||
|
||
:::note | ||
If using a different project, ensure that `dagster-cloud` is included as a dependency in your `setup.py` or `requirements.txt` file. | ||
|
||
For example, in `my-dagster-project/setup.py`: | ||
|
||
```python | ||
install_requires=[ | ||
"dagster", | ||
"dagster-cloud", # add this line | ||
... | ||
] | ||
``` | ||
|
||
::: | ||
|
||
Next, install the [`dagster-cloud` CLI](/dagster-plus/deployment/branch-deployments/dagster-cloud-cli) and use its `configure` command to authenticate it to your Dagster+ organization. | ||
|
||
**Note:** The CLI requires a recent version of Python 3 and Docker. | ||
|
||
```shell | ||
pip install dagster-cloud | ||
dagster-cloud configure | ||
``` | ||
|
||
You can also configure the `dagster-cloud` tool non-interactively; see [the CLI docs](/dagster-plus/deployment/branch-deployments/dagster-cloud-cli) for more information. | ||
|
||
Finally, deploy your project to Dagster+ using the `serverless` command: | ||
|
||
```shell | ||
dagster-cloud serverless deploy-python-executable ./my-dagster-project \ | ||
--location-name example \ | ||
--package-name quickstart_etl \ | ||
--python-version 3.12 | ||
``` | ||
|
||
**Note:** Windows users should use the `deploy` command instead of `deploy-python-executable`. | ||
|
||
</TabItem> | ||
</Tabs> | ||
|
||
--- |
31 changes: 26 additions & 5 deletions
31
docs/docs-beta/docs/dagster-plus/deployment/serverless/dagster-ips.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,8 +1,29 @@ | ||
--- | ||
title: "Dagster+ IP addresses" | ||
displayed_sidebar: "dagsterPlus" | ||
sidebar_position: 20 | ||
unlisted: true | ||
title: 'IP addresses' | ||
displayed_sidebar: 'dagsterPlus' | ||
sidebar_position: 50 | ||
--- | ||
|
||
# Dagster+ IP addresses | ||
# Dagster+ Serverless IP addresses | ||
|
||
Serverless code will make requests from one of the following IP addresses. You may need to whitelist / allowlist them for services your code interacts with. | ||
|
||
```plain | ||
34.216.9.66 | ||
35.162.181.243 | ||
35.83.14.215 | ||
44.230.239.14 | ||
44.240.64.133 | ||
52.34.41.163 | ||
52.36.97.173 | ||
52.37.188.218 | ||
52.38.102.213 | ||
52.39.253.102 | ||
52.40.171.60 | ||
52.89.191.177 | ||
54.201.195.80 | ||
54.68.25.27 | ||
54.71.18.84 | ||
``` | ||
|
||
**Note**: Additional IP addresses may be added over time. This list was last updated on **October 24, 2024**. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.