Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[docs-beta] migrate - managing multiple projects and teams doc #26501

Merged
merged 3 commits into from
Dec 19, 2024
Merged

[docs-beta] migrate - managing multiple projects and teams doc #26501

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4e54951
[docs-beta] migrate - managing multiple projects and teams doc
cmpadden Dec 15, 2024
17665d1
todo links
cmpadden Dec 19, 2024
39afdee
todo links
cmpadden Dec 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
391 changes: 389 additions & 2 deletions ...docs/dagster-plus/deployment/management/managing-multiple-projects-and-teams.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,393 @@
---
title: Managing multiple projects and teams
unlisted: true
---

{/* TODO move from https://docs.dagster.io/dagster-plus/best-practices/managing-multiple-projects-and-teams */}
In this guide, we'll cover some strategies for managing multiple projects/code bases and teams in a Dagster+ account.

## Separating code bases

:::note
In this section, repository refers to a version control system, such as Git or Mercurial.
:::

If you want to manage complexity or divide your work into areas of responsibility, consider isolating your code bases into multiple projects with:

- Multiple directories in a single repository, or
- Multiple repositories

Refer to the following table for more information, including the pros and cons of each approach.

<table
className="table"
style={{
width: "100%",
}}
>
<thead>
<tr>
<th
style={{
width: "14%",
}}
>
Approach
</th>
<th
style={{
width: "43%",
}}
>
Multiple directories in a single repository
</th>
<th>Multiple repositories</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<strong>How it works</strong>
</td>
<td>
You can use a single repository to manage multiple projects by placing
each project in a separate directory. Depending on your VCS, you may be
able to set{" "}
<a href="https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners">

Check warning on line 54 in docs/docs-beta/docs/dagster-plus/deployment/management/managing-multiple-projects-and-teams.md

View workflow job for this annotation

GitHub Actions / deploy 

Do not use an `<a>` element to navigate. Use the `<Link />` component from `@docusaurus/Link` instead. See: https://docusaurus.io/docs/docusaurus-core#link
code owners
</a>{" "}
to restrict who can modify each project.
</td>
<td>
For stronger isolation, you can use multiple repositories to manage
multiple projects.
</td>
</tr>
<tr>
<td>
<strong>Pros</strong>
</td>
<td>
<ul
style={{
marginTop: "0px",
}}
>
<li
style={{
marginTop: "0px",
}}
>
Simple to implement

Check failure on line 79 in docs/docs-beta/docs/dagster-plus/deployment/management/managing-multiple-projects-and-teams.md

View workflow job for this annotation

GitHub Actions / runner / vale 

[vale] reported by reviewdog 🐶
[Vale.Avoid] Avoid using 'Simple'.

Raw Output:
{"message": "[Vale.Avoid] Avoid using 'Simple'.", "location": {"path": "docs/docs-beta/docs/dagster-plus/deployment/management/managing-multiple-projects-and-teams.md", "range": {"start": {"line": 79, "column": 13}}}, "severity": "ERROR"}
</li>
<li>Facilitates code sharing between projects</li>
</ul>
</td>
<td>
<ul
style={{
marginTop: "0px",
}}
>
<li
style={{
marginTop: "0px",
}}
>
Stronger isolation between projects and teams
</li>
<li>
Each project has its own CI/CD pipeline and be deployed
independently
</li>
<li>Dependencies between projects can be managed independently</li>
</ul>
</td>
</tr>
<tr>
<td>
<strong>Cons</strong>
</td>
<td>
<ul
style={{
marginTop: "0px",
}}
>
<li
style={{
marginTop: "0px",
}}
>
All projects share the same CI/CD pipeline and cannot be deployed
independently
</li>
<li>
Shared dependencies between projects may cause conflicts and require
coordination between teams
</li>
</ul>
</td>
<td>
Code sharing between projects require additional coordination to publish
and reuse packages between projects
</td>
</tr>
</tbody>
</table>

### Deployment configuration

{/* Whether you use a single repository or multiple, you can use a [`dagster_cloud.yaml` file](/dagster-plus/managing-deployments/dagster-cloud-yaml) to define the code locations to deploy. For each repository, follow the [steps appropriate to your CI/CD provider](/dagster-plus/getting-started#step-4-configure-cicd-for-your-project) and include only the code locations that are relevant to the repository in your CI/CD workflow. */}
Whether you use a single repository or multiple, you can use a [`dagster_cloud.yaml` file](/todo) to define the code locations to deploy. For each repository, follow the [steps appropriate to your CI/CD provider](/todo) and include only the code locations that are relevant to the repository in your CI/CD workflow.

#### Example with GitHub CI/CD on Hybrid deployment

1. **For each repository**, use the CI/CD workflow provided in [Dagster+ Hybrid quickstart repository](https://github.com/dagster-io/dagster-cloud-hybrid-quickstart/blob/main/.github/workflows/dagster-cloud-deploy.yml).

{/* 2. **For each project in the repository**, configure a code location in the [`dagster_cloud.yaml` file](/dagster-plus/managing-deployments/dagster-cloud-yaml): */}
2. **For each project in the repository**, configure a code location in the [`dagster_cloud.yaml` file](/todo):

```yaml
# dagster_cloud.yml

locations:
- location_name: project_a
code_source:
package_name: project_a
build:
# ...
- location_name: project_b
code_source:
package_name: project_b
build:
# ...
```

3. In the repository's `dagster-cloud-deploy.yml` file, modify the CI/CD workflow to deploy all code locations for the repository:

```yaml
# .github/workflows/dagster-cloud-deploy.yml

jobs:
dagster-cloud-deploy:
# ...
steps:
- name: Update build session with image tag for "project_a" code location
id: ci-set-build-output-project-a
if: steps.prerun.outputs.result != 'skip'
uses: dagster-io/dagster-cloud-action/actions/utils/[email protected]
with:
command: "ci set-build-output --location-name=project_a --image-tag=$IMAGE_TAG"

- name: Update build session with image tag for "project_b" code location
id: ci-set-build-output-project-b
if: steps.prerun.outputs.result != 'skip'
uses: dagster-io/dagster-cloud-action/actions/utils/[email protected]
with:
command: "ci set-build-output --location-name=project_b --image-tag=$IMAGE_TAG"
# ...
```

---

## Isolating execution context between projects

Separating execution context between projects can have several motivations:

- Facilitating separation of duty between teams to prevent access to sensitive data
- Differing compute environments and requirements, such as different architecture, cloud provider, etc.
- Reducing impact on other projects. For example, a project with a large number of runs can impact the performance of other projects.

In order from least to most isolated, there are three levels of isolation:

- [Code location](#code-location-isolation)
- [Agent](#agent-isolation)
- [Deployment](#deployment-isolation)

### Code location isolation

If you have no specific requirements for isolation beyond the ability to deploy and run multiple projects, you can use a single agent and deployment to manage all your projects as individual code locations.

![Diagram of isolation at the code location level](/images/dagster-cloud/managing-deployments/isolation-level-code-locations.png)

<table
className="table"
style={{
width: "100%",
}}
>
<thead>
<tr>
<th
style={{
width: "50%",
}}
>
Pros
</th>
<th>Cons</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<ul
style={{
marginTop: "0px",
}}
>
<li
style={{
marginTop: "0px",
}}
>
Simplest and most cost-effective solution
</li>
<li>User access control can be set at the code location level</li>
<li>Single glass pane to view all assets</li>
</ul>
</td>
<td>
<ul
style={{
marginTop: "0px",
}}
>
<li
style={{
marginTop: "0px",
}}
>
No isolation between execution environments
</li>
</ul>
</td>
</tr>
</tbody>
</table>

### Agent isolation

:::note
Agent queues are a Dagster+ Pro feature available on hybrid deployment.
:::

{/* Using the [agent routing feature](/dagster-plus/deployment/agents/running-multiple-agents#routing-requests-to-specific-agents), you can effectively isolate execution environments between projects by using a separate agent for each project. */}
Using the [agent routing feature](/todo), you can effectively isolate execution environments between projects by using a separate agent for each project.

Motivations for utilizing this approach could include:

- Different compute requirements, such as different cloud providers or architectures
- Optimizing for locality or access, such as running the data processing closer or in environment with access to the storage locations

![Diagram of isolation at the agent level](/images/dagster-cloud/managing-deployments/isolation-level-agents.png)

<table
className="table"
style={{
width: "100%",
}}
>
<thead>
<tr>
<th
style={{
width: "50%",
}}
>
Pros
</th>
<th>Cons</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<ul
style={{
marginTop: "0px",
}}
>
<li
style={{
marginTop: "0px",
}}
>
Isolation between execution environments
</li>
<li>User access control can be set at the code location level</li>
<li>Single glass pane to view all assets</li>
</ul>
</td>
<td>Extra work to set up additional agents and agent queues</td>
</tr>
</tbody>
</table>

### Deployment isolation

:::note
Multiple deployments are only available in Dagster+ Pro.
:::

Of the approaches outlined in this guide, multiple deployments are the most isolated solution. The typical motivation for this isolation level is to separate production and non-production environments. It may be considered to satisfy other organization specific requirements.

![Diagram of isolation at the Dagster+ deployment level](/images/dagster-cloud/managing-deployments/isolation-level-deployments.png)

<table
className="table"
style={{
width: "100%",
}}
>
<thead>
<tr>
<th
style={{
width: "50%",
}}
>
Pros
</th>
<th>Cons</th>
</tr>
</thead>
<tbody>
<tr>
<td>
<ul
style={{
marginTop: "0px",
}}
>
<li
style={{
marginTop: "0px",
}}
>
Isolation between assets and execution environments
</li>
<li>
User access control can be set at the code location and deployment
level
</li>
</ul>
</td>
<td>
No single glass pane to view all assets (requires switching between
multiple deployments in the UI)
</td>
</tr>
</tbody>
</table>


{/*
## Related

- [Dagster+ Hybrid deployments](/dagster-plus/deployment/hybrid)
- [Dagster+ Hybrid agents](/dagster-plus/deployment/agents)
- [Managing deployments in Dagster+](/dagster-plus/managing-deployments)
(/* - [Running multiple Dagster+ Hybrid agents](/dagster-plus/deployment/agents/running-multiple-agents#routing-requests-to-specific-agents) */}
- [Running multiple Dagster+ Hybrid agents](/todo)
{/* - [dagster_cloud.yaml](/dagster-plus/managing-deployments/dagster-cloud-yaml) */}
- [dagster_cloud.yaml](/todo)
Binary file added ...eta/static/images/dagster-cloud/managing-deployments/isolation-level-agents.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added ...ic/images/dagster-cloud/managing-deployments/isolation-level-code-locations.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file added ...tatic/images/dagster-cloud/managing-deployments/isolation-level-deployments.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading