Skip to content

Commit

Permalink
sensors docs: clarify run_key behavior in a single sensor evaluation
Browse files Browse the repository at this point in the history
  • Loading branch information
Gw1p committed Dec 11, 2024
1 parent 5b628db commit 3b4d9aa
Showing 1 changed file with 2 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,8 @@ In the example, a <PyObject object="RunRequest"/> is requested for each file dur

Run keys allow you to write sensor evaluation functions that declaratively describe what runs should exist, and helps you avoid the need for more complex logic that manages state. However, when dealing with high-volume external events, some state-tracking optimizations might be necessary.

NOTE: if you yield multiple <PyObject object="RunRequest"/> with the same `run_key` in the same sensor evaluation, dagster will not skip any of them. That is, all `RunRequests` in a single sensor evaluation will be launched (even if the run key is the same on some or all of them).

### Sensor optimizations using cursors

When writing a sensor that deals with high-volume events, it might not be feasible to yield a <PyObject object="RunRequest"/> during every sensor evaluation. For example, you may have an `s3` storage bucket that contains thousands of files.
Expand Down

0 comments on commit 3b4d9aa

Please sign in to comment.