Skip to content

Commit

Permalink
add yarn
Browse files Browse the repository at this point in the history
  • Loading branch information
PedramNavid committed Aug 21, 2024
1 parent 853a03d commit 83f260a
Show file tree
Hide file tree
Showing 65 changed files with 34,619 additions and 11,610 deletions.
8 changes: 8 additions & 0 deletions docs/docs-beta/.gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -19,3 +19,11 @@ npm-debug.log*
yarn-debug.log*
yarn-error.log*
*.sqlite

.pnp.*
.yarn/*
!.yarn/patches
!.yarn/plugins
!.yarn/releases
!.yarn/sdks
!.yarn/versions
Binary file added docs/docs-beta/.yarn/install-state.gz
Binary file not shown.
925 changes: 925 additions & 0 deletions docs/docs-beta/.yarn/releases/yarn-4.4.0.cjs

Large diffs are not rendered by default.

3 changes: 3 additions & 0 deletions docs/docs-beta/.yarnrc.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
nodeLinker: node-modules

yarnPath: .yarn/releases/yarn-4.4.0.cjs
36 changes: 21 additions & 15 deletions docs/docs-beta/content-templates/concept.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,47 +5,49 @@ description: ''

# [TOPIC]

This section is an intro that includes:
<!-- This section is an intro that includes:
- A brief description of what the topic is,
- An example of how it could be used in the real-world
- What it can do in the UI
-->

---

## Benefits

This section lists the benefits of using the topic, whatever it is. The items listed here should be solutions to real-world problems that the user cares about, ex:
<!-- This section lists the benefits of using the topic, whatever it is. The items listed here should be solutions to real-world problems that the user cares about, ex:
Using schedules helps you:
- Predictably process and deliver data to stakeholders and business-critical applications
- Consistently run data pipelines without the need for manual intervention
- Optimize resource usage by scheduling pipelines to run during off-peak hours
-->

Using [TOPIC] helps you:

- A benefit of the thing
<!-- - A benefit of the thing
- Another benefit
- And one more
- And one more -->

---

## Prerequisites

This section lists the prerequisites users must complete before they should/can proceed. For concepts, we should list the other concepts they should be familiar with first.
<!-- This section lists the prerequisites users must complete before they should/can proceed. For concepts, we should list the other concepts they should be familiar with first. -->

Before continuing, you should be familiar with:

- Ex: To use asset checks, users should understand Asset definitions first
<!-- - Ex: To use asset checks, users should understand Asset definitions first
- Another one
- One more
- One more -->

---

## How it works

This section provides a high-level overview of how the concept works without getting too into the technical details. Code can be shown here, but this section shouldn't focus on it. The goal is to help the user generally understand how the thing works and what they need to do to get it working without overwhelming them with details.
<!-- This section provides a high-level overview of how the concept works without getting too into the technical details. Code can be shown here, but this section shouldn't focus on it. The goal is to help the user generally understand how the thing works and what they need to do to get it working without overwhelming them with details.
For example, this is the How it works for Schedules:
Expand All @@ -55,36 +57,40 @@ Schedules run jobs at fixed time intervals and have two main components:
- A cron expression, which defines when the schedule runs. Simple and complex schedules are supported, allowing you to have fine-grained control over when runs are executed. With cron syntax, you can:
- Create custom schedules like Every hour from 9:00AM - 5:00PM with cron expressions (0 9-17 \* \* \*)
- Quickly create basic schedules like Every day at midnight with predefined cron definitions (@daily, @midnight)
- Create custom schedules like Every hour from 9:00AM - 5:00PM with cron expressions (0 9-17 * * *)
- Quickly create basic schedules like Every day at midnight with predefined cron definitions (@daily, @midnight)
To make creating cron expressions easier, you can use an online tool like Crontab Guru. This tool allows you to create and describe cron expressions in a human-readable format and test the execution dates produced by the expression. Note: While this tool is useful for general cron expression testing, always remember to test your schedules in Dagster to ensure the results are as expected.
To make creating cron expressions easier, you can use an online tool like Crontab Guru. This tool allows you to create and describe cron expressions in a human-readable format and test the execution dates produced by the expression. Note: While this tool is useful for general cron expression testing, always remember to test your schedules in Dagster to ensure the results are as expected.
For a schedule to run, it must be turned on and an active dagster-daemon process must be running. If you used dagster dev to start the Dagster UI/webserver, the daemon process will be automatically launched alongside the webserver.
After these criteria are met, the schedule will run at the interval specified in the cron expression. Schedules will execute in UTC by default, but you can specify a custom timezone.
-->

---

## Getting started

This section is a list of guides / links to pages to help the user get started using the topic.
<!-- This section is a list of guides / links to pages to help the user get started using the topic. -->

Check out these guides to get started with [CONCEPT]:

From here, you can:

<!-- A list of things the user can do once they've got the basics down. These could be links to additional guides, ex:
- Construct schedules to run partitioned jobs
- Execute jobs in specific timezones
- Learn to test your schedules
- Identify and resolve common issues with our troubleshooting guide
- Identify and resolve common issues with our troubleshooting guide -->

### Limitations [and notes]

This section should describe any known limitations that could impact the user, ex: "Schedules will execute in UTC unless a timezone is specified"
<!-- This section should describe any known limitations that could impact the user, ex: "Schedules will execute in UTC unless a timezone is specified" -->

---

## Related

A list of related links and resources
<!-- A list of related links and resources -->
8 changes: 7 additions & 1 deletion docs/docs-beta/content-templates/example-reference.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,13 @@ This reference contains a variety of examples using Dagster [TOPIC]. Each exampl
- Links to relevant documentation
- A list of the APIs used in the example

---

## [Title of example]

[This example demonstrates [description of what the example accomplishes]

Example: This example demonstrates how to use resources in schedules. To specify a resource dependency, annotate the resource as a parameter to the schedule's function.
<!-- Example: This example demonstrates how to use resources in schedules. To specify a resource dependency, annotate the resource as a parameter to the schedule's function. -->

```python title="my_schedule.py"
@schedule(job=my_job, cron_schedule="* * * * *")
Expand All @@ -25,6 +27,8 @@ def logs_then_skips(context):
return SkipReason("Nothing to do")
```

<!-- We need to fix the base table implemenatation before launch. This is a must. -->

| | |
| -------------------- | --- |
| Notes | |
Expand All @@ -33,6 +37,8 @@ def logs_then_skips(context):

---

<!-- This section lists a few additional sources of inspiration, such as DOP and GitHub discussions. You shouldn't need to change anything here. -->

import InspirationList from '../partials/\_InspirationList.md';

<InspirationList />
8 changes: 4 additions & 4 deletions docs/docs-beta/content-templates/guide-with-steps.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,20 +23,20 @@ To follow the steps in this guide, you'll need:

</details>

## Step 1: Title that describes what this step will do
## Step 1: Title that describes what this step will do {#step-1}

For section / step headings:

- Titles should describe an action, ex: "Generate a token"
- Don't use gerunds (-ing) in titles, as it can cause issues with translation + SEO
- Each section heading should have an identifier that includes the word 'step' and the number of the step
- Each section heading should have an identifier that includes the word 'step' and the number of the step, ex: {#step-1}

### Step 1.1: Title that describes a substep
### Step 1.1: Title that describes a substep {#step-1-1}

If a step would benefit by being broken into smaller steps, follow this section's formatting
Each substep should get an H3 and start with Step N., followed by the number of the substep

## Step 2: Another step
## Step 2: Another step {#step-2}

## Next steps

Expand Down
2 changes: 1 addition & 1 deletion docs/docs-beta/docs/concepts/assets.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,6 @@ title: Assets

# Assets

## Assets and Ops
## Assets and ops

Assets and ops are two different concepts in Dagster.
4 changes: 2 additions & 2 deletions docs/docs-beta/docs/concepts/assets/thinking-in-assets.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: "Thinking in assets"
title: "Think in assets"
sidebar_position: 10
---

# Thinking in assets
# Think in assets
8 changes: 3 additions & 5 deletions docs/docs-beta/docs/concepts/io-managers.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
## =======

## title: "I/O managers"

# I/O managers
---
title: "I/O managers"
---
2 changes: 1 addition & 1 deletion docs/docs-beta/docs/concepts/understanding-assets.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
---
title: Understanding Assets
title: About assets
description: Understanding the concept of assets in Dagster
last_update:
date: 2024-08-11
Expand Down
4 changes: 2 additions & 2 deletions docs/docs-beta/docs/dagster-plus/access/rbac/teams.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: "Managing teams"
title: "Team management"
displayed_sidebar: "dagsterPlus"
sidebar_position: 2
---

# Managing teams
# Team management in Dagster+
4 changes: 2 additions & 2 deletions docs/docs-beta/docs/dagster-plus/access/rbac/users.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: "Managing users"
title: "User management"
displayed_sidebar: "dagsterPlus"
sidebar_position: 1
---

# Managing users
# User management in Dagster+
2 changes: 1 addition & 1 deletion docs/docs-beta/docs/dagster-plus/deployment/alerts.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@
title: "Dagster+ alerts"
---

# DAgster+ alerts
# Dagster+ alerts
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
---
title: "Branch Deployment Change Tracking"
title: "Change Tracking in Branch Deployments"
displayed_sidebar: "dagsterPlus"
sidebar_position: 4
sidebar_label: "Change Tracking"
---

# Using Change Tracking
# Change Tracking in Branch Deployments
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,4 @@ sidebar_position: 3
sidebar_label: "dagster-cloud CLI"
---

# Using Branch Deployments with the dagster-cloud CLI
# Use Branch Deployments with the dagster-cloud CLI
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,4 @@ sidebar_position: 1
sidebar_label: "GitHub"
---

# Using Branch Deployments with GitHub
# Use Branch Deployments with GitHub
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,4 @@ sidebar_position: 2
sidebar_label: "Gitlab"
---

# Using Branch Deployments with Gitlab
# Use Branch Deployments with Gitlab
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
---
title: "Setting environment variables using agent config"
title: "Set environment variables using agent config"
displayed_sidebar: "dagsterPlus"
sidebar_position: 2
sidebar_label: "Set with agent config"
---

# Setting environment variables using agent config
# Set environment variables using agent config
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
---
title: "Setting environment variables with the Dagster+ UI"
title: "Set environment variables with the Dagster+ UI"
displayed_sidebar: "dagsterPlus"
sidebar_position: 1
sidebar_label: "Set with Dagster+ UI"
---

# Setting environment variables with the Dagster+ UI
# Set environment variables with the Dagster+ UI
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@ displayed_sidebar: "dagsterPlus"
sidebar_position: 50
---

# Using multiple agents
# Use multiple agents with Dagster+
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@ displayed_sidebar: "dagsterPlus"
sidebar_position: 50
---

# Transitioning from Serverless to Hybrid
# Transition from Serverless to Hybrid
2 changes: 1 addition & 1 deletion docs/docs-beta/docs/dagster-plus/getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,4 @@ title: "Getting started with Dagster+"
displayed_sidebar: "dagsterPlus"
---

# Getting started with Dagster+
# Get started with Dagster+
4 changes: 2 additions & 2 deletions docs/docs-beta/docs/dagster-plus/insights/asset-metadata.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: "Integrating asset metadata into Dagster+ Insights"
title: "Integrate asset metadata into Dagster+ Insights"
sidebar_label: "Integrate asset metadata"
sidebar_position: 1
---

# Integrating asset metadata into Dagster+ Insights
# Integrate asset metadata into Dagster+ Insights
4 changes: 2 additions & 2 deletions docs/docs-beta/docs/dagster-plus/insights/export-metrics.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
---
title: "Exporting metrics from Dagster+ Insights"
title: "Export metrics from Dagster+ Insights"
sidebar_label: "Export metrics"
sidebar_position: 6
---

# xporting metrics from Dagster+ Insights
# Export metrics from Dagster+ Insights
3 changes: 1 addition & 2 deletions docs/docs-beta/docs/guides/automation.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,6 @@ You must provide a function that the sensor will use to determine if it should t

Like schedules, sensors operate on a selection of assets, known as [Jobs](/concepts/ops-jobs) and can either start a pipeline through a Run or log a reason for not starting a pipeline using a SkipReason.


### When to use sensors

- You need event-driven automation
Expand All @@ -78,7 +77,7 @@ Asset Sensors trigger jobs when specified assets are materialized, allowing you

For more examples of how to create asset sensors, see the [How-To Use Asset Sensors](/guides/automation/asset-sensors) guide.

## Declarative automation
## Declarative Automation

TODO: add content

Expand Down
6 changes: 2 additions & 4 deletions docs/docs-beta/docs/guides/automation/asset-sensors.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,8 @@
---
title: Triggering jobs with Asset Sensors
sidebar_label: Triggering jobs with Asset Sensors
title: Trigger jobs with Asset Sensors
sidebar_position: 30
---


Asset sensors in Dagster provide a powerful mechanism for monitoring asset materializations and triggering downstream computations or notifications based on those events.

This guide covers the most common use cases for asset sensors such as defining cross-job and cross-code location dependencies.
Expand Down Expand Up @@ -74,7 +72,7 @@ otherwise it skips the run.

## Trigger a job with configuration

By providing a configuration to the `RunRequest` object, you can trigger a job with a specific configuration. This is useful when you want to trigger a job with custom parameters based on custom logic you define. For example, you might use a sensor to trigger a job when an asset is materialized, but also pass metadata about that materialization to the job.
By providing a configuration to the `RunRequest` object, you can trigger a job with a specific configuration. This is useful when you want to trigger a job with custom parameters based on custom logic you define. For example, you might use a sensor to trigger a job when an asset is materialized, but also pass metadata about that materialization to the job.

<CodeExample filePath="guides/automation/asset-sensor-with-config.py" language="python" title="Asset Sensor with Config" />

Expand Down
Loading

0 comments on commit 83f260a

Please sign in to comment.