Skip to content

Commit

Permalink
[docs-beta] Refresh Dagster+ Serverless guide (#25563)
Browse files Browse the repository at this point in the history
## Summary & Motivation

[DOCS-545](https://linear.app/dagster-labs/issue/DOC-545/[docs]-refresh-dagster-serverless-guide)

- Ported missing content from original Serverless docs
- Extended content in some pages/sections based on feedback from users
and support cases

## How I Tested These Changes
👀

---------

Co-authored-by: colton <[email protected]>
  • Loading branch information
garethbrickman and cmpadden authored Oct 29, 2024
1 parent 910f5c5 commit 6a46878
Show file tree
Hide file tree
Showing 6 changed files with 295 additions and 38 deletions.
41 changes: 38 additions & 3 deletions docs/docs-beta/docs/dagster-plus/deployment/serverless.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,43 @@
---
title: "Serverless deployment"
displayed_sidebar: "dagsterPlus"
title: 'Serverless deployment'
displayed_sidebar: 'dagsterPlus'
sidebar_position: 1
unlisted: true
---

# Serverless deployment

Dagster+ Serverless is a fully managed version of Dagster+ and is the easiest way to get started with Dagster. With a Serverless deployment, you can run your Dagster jobs without spinning up any infrastructure yourself.

---

## When to choose Serverless {#when-to-choose-serverless}

Serverless works best with workloads that primarily orchestrate other services or perform light computation. Most workloads fit into this category, especially those that orchestrate third-party SaaS products like cloud data warehouses and ETL tools.

If any of the following are applicable, you should select [Hybrid deployment](/dagster-plus/deployment/hybrid):

- You require substantial computational resources. For example, training a large machine learning (ML) model in-process
- Your dataset is too large to fit in memory. For example, training a large ML model in-process on a terabyte of data
- You need to distribute computation across many nodes for a single run. Dagster+ runs currently execute on a single node with 4 CPUs
- You don't want to add Dagster Labs as a data processor

---

## Limitations {#limitations}

Serverless is subject to the following limitations:

- Maximum of 100 GB of bandwidth per day
- Maximum of 4500 step-minutes per day
- Runs receive 4 vCPU cores, 16 GB of RAM, and 128 GB of ephemeral disk
- Code locations receive 0.25 vCPU cores and 1 GB of RAM
- All Serverless jobs run in the United States
- Infrastructure cannot be customized or extended, such as using additional containers

Dagster+ Pro customers may request a quota increase by [contacting Sales](https://dagster.io/contact).

---

## Next steps

To start using Dagster+ Serverless, follow our [Getting started with Dagster+](/dagster-plus/getting-started) guide.
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
---
title: 'CI/CD in Serverless '
displayed_sidebar: 'dagsterPlus'
sidebar_position: 10
---

# CI/CD in Serverless

If you're a GitHub or GitLab user, you can use our predefined workflows to seamlessly deploy and synchronize your code to Dagster+. You can also use other Git providers or a local Git repository with our [dagster-cloud CLI](/dagster-plus/deployment/branch-deployments/dagster-cloud-cli) to run your own CI/CD process.

<Tabs groupId="method">
<TabItem value="GitHub" label="With GitHub">

If you're a GitHub user, with a single click our GitHub app with GitHub Actions can set up a repo containing skeleton code and configuration for you consistent with Dagster+'s best practices. Pushing your code changes to the `main` branch will automatically deploy them to your `prod` Serverless deployment. Pull requests will spin up ephemeral [branch deployments](/dagster-plus/deployment/branch-deployments) that you can view in the Dagster+ UI for previewing and testing.

:::note
**If you are importing a Dagster project that's in an existing GitHub repo:**

- The repo will need to allow the [Workflow permission](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository) for `Read and write permissions`. Workflow permissions settings can be found in GitHub's `Settings` > `Actions` > `General` > `Workflow permissions`. In GitHub Enterprise, these permissions [are controlled at the Organization level](https://github.com/orgs/community/discussions/57244).

- An initial commit will need to be able to be merged directly to the repo's `main` branch to automatically add the GitHub Actions workflow files. If [branch protection rules](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches#about-protected-branches) require changes be done through a pull request, it will prevent the automatic setup from completing.

- You can temporarily disable the branch protection rules and then re-enable them after the automatic setup completes. Alternatively, you can manually set up the GitHub Actions workflows. You can use our [dagster-cloud-serverless-quickstart repo](https://github.com/dagster-io/dagster-cloud-serverless-quickstart) as a template and its [README](https://github.com/dagster-io/dagster-cloud-serverless-quickstart/blob/main/README.md) as a guide.

:::

</TabItem>

<TabItem value="GitLab" label="With GitLab">

If you're a GitLab user, with a single click our GitLab app can set up a repo containing skeleton code and CI/CD configuration for you consistent with Dagster+'s best practices. Pushing your code changes to the `main` branch will automatically deploy them to your `prod` Serverless deployment. Pull requests will spin up ephemeral [branch deployments](/dagster-plus/deployment/branch-deployments) that you can view in the Dagster+ UI for previewing and testing.

</TabItem>

<TabItem value="Other" label="Other Git providers or local development">

If you don't want to use our automated GitHub/GitLab process, we offer [the powerful `dagster-cloud` command-line interface (CLI)](/dagster-plus/deployment/branch-deployments/dagster-cloud-cli) that you can use in another CI environment or locally.

First, [create a new project](/getting-started/quickstart) with the Dagster open source CLI.

The example below uses our [quickstart_etl example project](https://github.com/dagster-io/dagster/tree/master/examples/quickstart_etl). For more info about the examples, visit the [Dagster GitHub repository](https://github.com/dagster-io/dagster/tree/master/examples).

```shell
pip install dagster
dagster project from-example \
--name my-dagster-project \
--example quickstart_etl
```

:::note
If using a different project, ensure that `dagster-cloud` is included as a dependency in your `setup.py` or `requirements.txt` file.

For example, in `my-dagster-project/setup.py`:

```python
install_requires=[
"dagster",
"dagster-cloud", # add this line
...
]
```

:::

Next, install the [`dagster-cloud` CLI](/dagster-plus/deployment/branch-deployments/dagster-cloud-cli) and use its `configure` command to authenticate it to your Dagster+ organization.

**Note:** The CLI requires a recent version of Python 3 and Docker.

```shell
pip install dagster-cloud
dagster-cloud configure
```

You can also configure the `dagster-cloud` tool non-interactively; see [the CLI docs](/dagster-plus/deployment/branch-deployments/dagster-cloud-cli) for more information.

Finally, deploy your project to Dagster+ using the `serverless` command:

```shell
dagster-cloud serverless deploy-python-executable ./my-dagster-project \
--location-name example \
--package-name quickstart_etl \
--python-version 3.12
```

**Note:** Windows users should use the `deploy` command instead of `deploy-python-executable`.

</TabItem>
</Tabs>

---
Original file line number Diff line number Diff line change
@@ -1,8 +1,29 @@
---
title: "Dagster+ IP addresses"
displayed_sidebar: "dagsterPlus"
sidebar_position: 20
unlisted: true
title: 'IP addresses'
displayed_sidebar: 'dagsterPlus'
sidebar_position: 50
---

# Dagster+ IP addresses
# Dagster+ Serverless IP addresses

Serverless code will make requests from one of the following IP addresses. You may need to whitelist / allowlist them for services your code interacts with.

```plain
34.216.9.66
35.162.181.243
35.83.14.215
44.230.239.14
44.240.64.133
52.34.41.163
52.36.97.173
52.37.188.218
52.38.102.213
52.39.253.102
52.40.171.60
52.89.191.177
54.201.195.80
54.68.25.27
54.71.18.84
```

**Note**: Additional IP addresses may be added over time. This list was last updated on **October 24, 2024**.
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: 'Serverless run isolation'
displayed_sidebar: 'dagsterPlus'
sidebar_label: 'Run isolation'
sidebar_position: 40
sidebar_position: 35
---

# Serverless run isolation
Expand Down
Loading

0 comments on commit 6a46878

Please sign in to comment.