Frequent 1 second polling in Sensor. Does it have pitfalls?

[4l1fe]

I guess i handle a typical scenario of polling the RabbitMQ queue for messages to run a job in a sensor. For the scenario, a listening consumer would be more efficient, but it's not provided by Dagster, only polling with Sensor.

Due to that i neeed more frequent timeout than the default 60 seconds, let it be 1 second, to get close to listening as dozens messages can appear in a queue in a short period of time, and to increase the whole system responsivity.

• What are potenial problems i could meet? I see the issue that is related to the sensor ticks run_status_sensor (failure & success) easily overwhelmed in larger deployments and become useless. Recent code changes removed ability to fix via hidden cursor. #19224
• Is Sensor designed for high frequency workloads? Cursor and Tick are entities in the MetadataDb and i think some issues can derive from there...

[garethbrickman]

This section about sensor evaluation interval is important to note:

By default, the Dagster daemon runs a sensor 30 seconds after that sensor's previous evaluation finishes executing. You can configure the interval using the minimum_interval_seconds argument on the @sensor decorator.

It's important to note that this interval represents a minimum interval between runs of the sensor and not the exact frequency the sensor runs. If you have a sensor that takes 30 seconds to complete, but the minimum_interval_seconds is 5 seconds, the fastest Dagster daemon will run the sensor is every 35 seconds. The minimum_interval_seconds only guarantees that the sensor is not evaluated more frequently than the given interval.

[4l1fe]

Thx for the reminder, i remember that. It describes the different issue that the sensor runs within an interval, not at an exact moment.

Would be great if you share how Sensor works on the database level...