Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[docs-beta] format mdx with prettier #26491

Draft
wants to merge 3 commits into
base: master
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/docs-beta/.prettierignore
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
node_modules
.docusaurus
.yarn/
build/
docs/
node_modules
10 changes: 5 additions & 5 deletions docs/docs-beta/docs/about/community.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,11 +76,11 @@ Instances of abusive, harassing, or otherwise unacceptable behavior may be repor

The Dagster core team will follow these Community Impact Guidelines in determining the consequences for any action they deem in violation of this Code of Conduct:

| Level | Community impact | Consequence |
|---|----|----|
| Reminder | Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community. | A private, written reminder from the Dagster core team, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. |
| Warning | A violation through a single incident or series of actions. | A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces and external channels like social media. Violating these terms will lead to a permanent ban. |
| Permanent ban | Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals. | A permanent ban from any sort of public interaction within the community. |
| Level | Community impact | Consequence |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Reminder | Use of inappropriate language or other behavior deemed unprofessional or unwelcome in the community. | A private, written reminder from the Dagster core team, providing clarity around the nature of the violation and an explanation of why the behavior was inappropriate. |
| Warning | A violation through a single incident or series of actions. | A warning with consequences for continued behavior. No interaction with the people involved, including unsolicited interaction with those enforcing the Code of Conduct, for a specified period of time. This includes avoiding interactions in community spaces and external channels like social media. Violating these terms will lead to a permanent ban. |
| Permanent ban | Demonstrating a pattern of violation of community standards, including sustained inappropriate behavior, harassment of an individual, or aggression toward or disparagement of classes of individuals. | A permanent ban from any sort of public interaction within the community. |

### Attribution

Expand Down
23 changes: 14 additions & 9 deletions docs/docs-beta/docs/api/api-lifecycle.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,49 +10,54 @@ The lifecycle stages described below provide a clear framework for understanding

## API lifecycle stages

| Stage | Description | Lifetime |
|---------|------------------------------------------------------------------------------------------------------------|----------------------------------------------------------------------------------------|
| Preview | API in the design phase, can change significantly, or be removed completely. Not for production use. | Until design is complete, or implementation cancelled |
| Beta | Features that is still being tested and may change. More stable than Preview, but still subject to change. | At most, two 1.x releases before it is either considered stable or returned to preview |
| Generally Available (GA)| Ready for production use, with minimal risk of breaking changes. | Supported until at least 2.0
| Superseded | This API is still available, but is no longer the best practice. A better alternative is available. | Supported until at least 2.0
| Deprecated | API is still available but will be removed in the future; avoid new usage. | Will be removed in a minor release, the DeprecationWarning will indicate the next release that will remove the API.

| Stage | Description | Lifetime |
| ------------------------ | ---------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------- |
| Preview | API in the design phase, can change significantly, or be removed completely. Not for production use. | Until design is complete, or implementation cancelled |
| Beta | Features that is still being tested and may change. More stable than Preview, but still subject to change. | At most, two 1.x releases before it is either considered stable or returned to preview |
| Generally Available (GA) | Ready for production use, with minimal risk of breaking changes. | Supported until at least 2.0 |
| Superseded | This API is still available, but is no longer the best practice. A better alternative is available. | Supported until at least 2.0 |
| Deprecated | API is still available but will be removed in the future; avoid new usage. | Will be removed in a minor release, the DeprecationWarning will indicate the next release that will remove the API. |

## Understanding the stages

### Preview

- **Purpose**: For early testing and feedback
- **Stability**: Highly unstable, expect frequent changes
- **Usage**: Not recommended for production environments
- **Documentation**: Minimal, typically just a README or unlisted documentation

### Beta

- **Purpose**: Feature testing with wider audience
- **Stability**: More stable than Preview, but still subject to changes
- **Usage**: Can be used in non-critical production environments
- **Documentation**: How-to guides and API documentation available

### GA (General Availability)

- **Purpose**: Production-ready features
- **Stability**: Stable with minimal risk of breaking changes
- **Usage**: Recommended for all production environments
- **Documentation**: Comprehensive documentation available

### Superseded

- **Purpose**: Maintains backwards compatibility while promoting newer alternatives
- **Stability**: Stable but no longer recommended
- **Usage**: Existing implementations can continue, but new projects should use the recommended alternative
- **Documentation**: API docs remain, but usage is discouraged in favor of newer alternatives

### Deprecated

- **Purpose**: Signals upcoming removal of the API
- **Stability**: Stable but scheduled for removal
- **Usage**: Existing implementations should plan migration
- **Documentation**: API docs remain, with clear warnings about deprecation. Arguments may be removed from function signature

### Dead

- **Purpose**: Removed APIs
- **Stability**: N/A
- **Usage**: No longer available for use
- **Documentation**: Removed from all documentation
- **Documentation**: Removed from all documentation
2 changes: 1 addition & 1 deletion docs/docs-beta/docs/changelog.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: "Changelog"
title: 'Changelog'
unlisted: true
---

Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: "Code location history and rollbacks"
title: 'Code location history and rollbacks'
sidebar_position: 100
---

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@ sidebar_position: 200
unlisted: true
---

{/* TODO move content from https://docs.dagster.io/dagster-plus/managing-deployments/dagster-cloud-yaml */}
{/** TODO move content from https://docs.dagster.io/dagster-plus/managing-deployments/dagster-cloud-yaml **/}
149 changes: 76 additions & 73 deletions docs/docs-beta/docs/dagster-plus/deployment/code-locations/index.md
Original file line number Diff line number Diff line change
@@ -1,33 +1,34 @@
---
title: "Code locations"
title: 'Code locations'
sidebar_position: 40
---

Separate code locations allow you to deploy different projects that still roll up into a single Dagster+ deployment with one global lineage graph.

This guide will cover three options for adding a new code location:

- Adding a code location manually
- Adding a code location in a new Git repository
- Adding a new code location to an existing Git monorepo

<details>
<summary>Prerequisites</summary>
<summary>Prerequisites</summary>

1. An existing Dagster project. Refer to the [recommended project structure](/tutorial/create-new-project) and [code requirements](/dagster-plus/deployment/code-requirements) pages for more information.

2. Editor, Admin, or Organization Admin permissions in Dagster+.

3. To know your Dagster+ deployment type. An administrator can help find this information, or you can locate it by clicking *Deployment > Agents tab* in the Dagster UI. Dagster+ Serverless organizations will have a *Managed by Dagster+* label next to the agent.
3. To know your Dagster+ deployment type. An administrator can help find this information, or you can locate it by clicking _Deployment > Agents tab_ in the Dagster UI. Dagster+ Serverless organizations will have a _Managed by Dagster+_ label next to the agent.

</details>

Adding a code location follows two steps:

- For Dagster+ Hybrid, ensuring the Dagster code is in a place accessible by your agent, usually by building a Docker image with your code that's pushed to a registry. For Dagster+ Serverless you can skip this step.
- Notifying Dagster+ of the new or updated code location. This will be done by using the Dagster+ Python client.

Often these two steps are handled by CI/CD connected to your Git repository.


## Add a new code location manually

Start by installing the `dagster-cloud` Python client:
Expand Down Expand Up @@ -62,6 +63,7 @@ Now add the code location. The following example assumes you are running the com
```

The commands below take two main arguments:

- `module_name` is determined by your code structure
- `location_name` is the unique label for this code location used in Dagster+

Expand All @@ -74,7 +76,8 @@ If you are using Dagster+ Serverless, run the following command to add a code lo
dagster-cloud serverless deploy-python-executable --deployment prod --location-name quickstart --module-name quickstart
```

Running the command multiple times with the same location name will *update* the code location. Running the command with a new location name will *add* a code location.
Running the command multiple times with the same location name will _update_ the code location. Running the command with a new location name will _add_ a code location.

</TabItem>
<TabItem value="hybrid" label="Dagster+ Hybrid">

Expand All @@ -83,10 +86,11 @@ If you are using Dagster+ Hybrid, make sure you have deployed the code appropria
```bash
dagster-cloud deployment add-location --deployment prod --location-name quickstart --module-name quickstart --image 764506304434.dkr.ecr.us-west-2.amazonaws.com/hooli-data-science-prod:latest
```

</TabItem>
</Tabs>

After running the command you can verify the code location was deployed by navigating to the *Deployments* tab on Dagster+.
After running the command you can verify the code location was deployed by navigating to the _Deployments_ tab on Dagster+.

## Adding a code location in a new Git repository

Expand All @@ -99,82 +103,81 @@ To get started, review the appropriate example repository and then create your G
- [GitLab CI/CD for Dagster+ Serverless](https://github.com/dagster-io/dagster-cloud-action/blob/main/gitlab/serverless-ci.yml)
- [GitLab CI/CD for Dagster+ Hybrid](https://github.com/dagster-io/dagster-cloud-action/blob/main/gitlab/hybrid-ci.yml)


Overall, the Git repository should contain:

1. Your Dagster code, structured as a Python module. For Dagter+ Hybrid you may need a [Dockerfile](https://github.com/dagster-io/dagster-cloud-hybrid-quickstart/blob/main/Dockerfile) as well. The repository might look like this:

```bash
README.md
dagster_cloud.yaml
Dockerfile
/.github
/workflows
dagster-cloud-deploy.yml
setup.py
pyproject.toml
/quickstart
__init__.py
definitions.py
/assets
...
/resources
...
```
```bash
README.md
dagster_cloud.yaml
Dockerfile
/.github
/workflows
dagster-cloud-deploy.yml
setup.py
pyproject.toml
/quickstart
__init__.py
definitions.py
/assets
...
/resources
...
```

2. A [`dagster_cloud.yaml` file](/todo) with the settings for your code location. Here is an example:

```yaml title="dagster_cloud.yaml
locations:
- location_name: quickstart
code_source:
package_name: quickstart
```
```yaml title="dagster_cloud.yaml
locations:
- location_name: quickstart
code_source:
package_name: quickstart
```

3. A CI/CD workflow file that contains the steps for adding your code location. These are the same steps outlined in the preceding section. Here is a minimal example workflow file for a Dagster+ Hybrid organization based on [this GitLab template](https://github.com/dagster-io/dagster-cloud-action/blob/main/gitlab/hybrid-ci.yml).

```yaml
variables:
DAGSTER_CLOUD_ORGANIZATION: <organinization-name>
DAGSTER_PROJECT_DIR: .
IMAGE_REGISTRY: <account-id>.dkr.ecr.us-west-2.amazonaws.com/<image-name>
IMAGE_TAG: $CI_COMMIT_SHORT_SHA-$CI_PIPELINE_ID

stages:
- build
- deploy
```yaml
variables:
DAGSTER_CLOUD_ORGANIZATION: <organinization-name>
DAGSTER_PROJECT_DIR: .
IMAGE_REGISTRY: <account-id>.dkr.ecr.us-west-2.amazonaws.com/<image-name>
IMAGE_TAG: $CI_COMMIT_SHORT_SHA-$CI_PIPELINE_ID

stages:
- build
- deploy

build:
stage: build
image: docker:latest
services:
- docker:dind
before_script:
# # For Gitlab Container Registry
# - echo $CI_JOB_TOKEN | docker login --username $CI_REGISTRY_USER --password-stdin $REGISTRY_URL
# # For DockerHub
# - echo $DOCKERHUB_TOKEN | docker login --username $DOCKERHUB_USERNAME --password-stdin $REGISTRY_URL
# # For AWS Elastic Container Registry (ECR)
# - apk add --no-cache curl jq python3 py3-pip
# - pip install awscli
# - echo $AWS_ECR_PASSWORD | docker login --username AWS --password-stdin $IMAGE_REGISTRY
# # For Google Container Registry (GCR)
# - echo $GCR_JSON_KEY | docker login --username _json_key --password-stdin $REGISTRY_URL
script:
- docker build . -t $IMAGE_REGISTRY:$IMAGE_TAG
- docker push $IMAGE_REGISTRY:$IMAGE_TAG

deploy:
stage: deploy
dependencies:
- build
image: ghcr.io/dagster-io/dagster-cloud-action:0.1.43
script:
- dagster-cloud deployment add-location --deployment prod --image
$IMAGE_REGISTRY:$IMAGE_TAG --location-name quickstart --package-name quickstart
```

build:
stage: build
image: docker:latest
services:
- docker:dind
before_script:
# # For Gitlab Container Registry
# - echo $CI_JOB_TOKEN | docker login --username $CI_REGISTRY_USER --password-stdin $REGISTRY_URL
# # For DockerHub
# - echo $DOCKERHUB_TOKEN | docker login --username $DOCKERHUB_USERNAME --password-stdin $REGISTRY_URL
# # For AWS Elastic Container Registry (ECR)
# - apk add --no-cache curl jq python3 py3-pip
# - pip install awscli
# - echo $AWS_ECR_PASSWORD | docker login --username AWS --password-stdin $IMAGE_REGISTRY
# # For Google Container Registry (GCR)
# - echo $GCR_JSON_KEY | docker login --username _json_key --password-stdin $REGISTRY_URL
script:
- docker build . -t $IMAGE_REGISTRY:$IMAGE_TAG
- docker push $IMAGE_REGISTRY:$IMAGE_TAG

deploy:
stage: deploy
dependencies:
- build
image: ghcr.io/dagster-io/dagster-cloud-action:0.1.43
script:
- dagster-cloud deployment add-location --deployment prod --image
$IMAGE_REGISTRY:$IMAGE_TAG --location-name quickstart --package-name quickstart
```

Once your Git repository has this structure, you will want to run your CI/CD process. The CI/CD process will add the code location to Dagster+ which can be verified by viewing the *Deployments* tab.
Once your Git repository has this structure, you will want to run your CI/CD process. The CI/CD process will add the code location to Dagster+ which can be verified by viewing the _Deployments_ tab.

## Adding a new code location to a Git monorepo

Expand Down Expand Up @@ -220,7 +223,7 @@ locations:
package_name: core
build:
directory: ./core
registry: your-registry/image # eg 764506304434.dkr.ecr.us-west-2.amazonaws.com/core
registry: your-registry/image # eg 764506304434.dkr.ecr.us-west-2.amazonaws.com/core
- location_name: new
code_source:
package_name: new
Expand All @@ -229,7 +232,7 @@ locations:
registry: your-registry/image # eg 764506304434.dkr.ecr.us-west-2.amazonaws.com/new
```

The monorepo should have CI/CD configured to deploy your changes and add or update your new code location. After adding your code and updating the `dagster_cloud.yaml` file, trigger the CI/CD process to add your code location to Dagster+. Navigate to the *Deployments* tab in Dagster+ to confirm your code location was added.
The monorepo should have CI/CD configured to deploy your changes and add or update your new code location. After adding your code and updating the `dagster_cloud.yaml` file, trigger the CI/CD process to add your code location to Dagster+. Navigate to the _Deployments_ tab in Dagster+ to confirm your code location was added.

## Next steps

Expand Down
Loading
Loading