Automation Part 2 #23723
Automation Part 2 #23723
Conversation
|
Deploy preview for dagster-docs-next ready! ✅ Preview Built with commit be0daa6. |
|
Deploy preview for dagster-docs ready! Preview available at https://dagster-docs-go5t5tpsf-elementl.vercel.app Direct link to changed pages: |
✅ Deploy Preview for dagsterapidocs canceled.
|
| more complex system that uses conditions on the assets to determine when to execute. | ||
|
|
||
| ## About Schedules |
There was a problem hiding this comment.
With this being here, what will the Schedules concept page look like?
There was a problem hiding this comment.
I wonder if we really need a 'Schedules' concept page or if Automation is enough?
We could then link to <concept/automation#Schedules>
There was a problem hiding this comment.
It might be fine for Schedules - it is something that everyone is probably familiar with. Do you think it will be a weird experience if Schedules doesn't have one, but something like Sensors does?
| - A `cron` string, which describes the schedule. | ||
|
|
||
| ### Defining a schedule using `ScheduleDefinition` |
There was a problem hiding this comment.
I don't think we should include code in concept pages - it's easy to fall into the 'everything but the kitchen sink' approach we have now. What exactly does it add here?
I especially don't think we should do that on this concept page, since the plan is to also add sections for sensors and eventually DA.
There was a problem hiding this comment.
I don't think I did a great job with this page, but my thinking was:
- how-to guides should be code-complete examples driven by a task, that you can copy-paste and run, which cover 70% of what people do
- Concept Explanations explain a topic, and code can be used to explain that topic, but there is no expectation that a code block is self-contained or runnable.
For example, in this section, I wanted to explain the difference between using @schedule and ScheduleDefinition, and using code is part of that explanation.
I did not fully write out the explanation for each, but that would be the idea., otherwise, I think we risk polluting our guides.
Share your concern about it becoming a kitchen sink though. I think maybe we have grades of review/input on content possibly? Maybe how-to guides anyone can start and write, but concept pages need more involvement from docs?
There was a problem hiding this comment.
there is no expectation that a code block is self-contained or runnable.
We might get flak for this, but I get what you mean.
And yes, docs should absolutely be involved in concept pages, including whether a topic needs one or not.
| ### Defining a schedule using `@schedule` | ||
|
|
||
| If you want more control over the schedule, you can use the `@schedule` decorator. In doing so, you are then responsible for either |
There was a problem hiding this comment.
"More control" meaning what, exactly? What additional capabilities does this approach give me?
There was a problem hiding this comment.
What do you think about something like this?
Define a schedule using @schedule
For more advanced scheduling needs, you can use the @schedule decorator. This allows you to implement custom logic to determine whether a schedule should run or not. When using this decorator, you need to return either a RunRequest to execute the job or a SkipReason to skip the execution. Additionally, you can log messages that will be displayed in the Dagster UI's schedule tick history, providing more context about the schedule's behavior.
@schedule(cron_schedule="15 5 * * 1-5")
def ecommerce_schedule(context):
if context.execution_time.hour < 10:
context.log.info("Not running ecommerce job at this time.")
return SkipReason("Not running ecommerce job at this time.")
else:
context.log.info("This log message will be visible in the Dagster UI.")
return RunRequest()| @@ -1,10 +1,80 @@ | |||
| --- | |||
| title: Asset Sensors | |||
| sidebar_position: 50 | |||
| title: Scheduling pipelines across projects and jobs with Asset Sensors | |||
There was a problem hiding this comment.
This page covers more than one type of asset sensor (@asset_sensor and @multi_asset_sensor). We should include some kind of 'rollup' of the different types of asset sensors and when to use each one before jumping into the individual sections.
There was a problem hiding this comment.
Ive been thinking through this and have struggled to find the right way to bring it up in this page.
I think it would be better in a concept section. If we think about the tasks people want to do, then either the @asset_sensor or @multi_asset_sensor is the right tool for that job.
But if people want to know about the different types of asset sensors, then that goes into a concept/explainer page.
I could add maybe a sentence under the Monitor multiple assets section.
What do you think of this:
The previous examples showed how to use a single asset sensor to monitor a single asset and trigger a job when it is materialized. In this example, we'll show how to use a multi-asset sensor to monitor multiple assets and trigger a job when any of the monitored assets are materialized.
docs/docs-beta/docs/code_examples/guides/automation/asset-sensor-custom-eval.py
Show resolved
Hide resolved
This is another attempt the Automation page.
The Asset sensor page is my attempt at a 'canonical example of a how-to guide'. I think it strikes the right balance between too much and not enough information. It's also a topic that is not as simple as 'Schedules' for example.
It also introduces diagrams to help clarify confusing concepts.
The Automation page is the 'entrypoint' into learning about the 3 types of automation.
The concept page is not complete, but its the first attempt at a 'concept page'. Right now it is only Schedules. But I imagine either 1) it will be split into multiple pages or 2) contain all automation concepts and not have sub-pages.