Scheduler taking 3 minutes between launching each run in a schedule

[NiallRees]

(Dagster version 1.6.6, Helm on GKE)

Perhaps related to #20078, we're experiencing very slow scheduler cycle times. The scheduler is taking 3 minutes between launching each run in our ~120 run schedule. CPU and memory all look fine. The maximum concurrency we've set is 26 - but it's hovering at around 13. Any help would be greatly appreciated.

Logs showing big gaps between launches:

CPU and memory:

Log dump from the daemon for a 20 minute period:
downloaded-logs-20240227-104432.json

Number of runs launched over time (18 are launched initially in quick succession, then it starts to slow down considerably. Perhaps the monitoring daemon is blocking the scheduler somehow?)

Also interesting is that we added two very basic schedules on the 24th of Feb. These are extremely simple - they run once per day, launch a single run each, lasting around 5 minutes. However, ever since this, the rate of launching runs has dropped significantly. Previously 120 runs would launch immediately, now it's considerably fewer.

We're also seeing lots of query timeouts in the UI

Operation name: ScheduleRootQuery

Message: (psycopg2.errors.QueryCanceled) canceling statement due to statement timeout

[SQL: SELECT runs.id, runs.run_body, runs.status, runs.create_timestamp, runs.update_timestamp, runs.start_time, runs.end_time 
FROM runs 
WHERE runs.run_id IN (SELECT run_tags.run_id 
FROM run_tags 
WHERE run_tags.key = %(key_1)s AND run_tags.value = %(value_1)s OR run_tags.key = %(key_2)s AND run_tags.value = %(value_2)s GROUP BY run_tags.run_id 
HAVING count(DISTINCT run_tags.key) = %(count_1)s) ORDER BY runs.id DESC 
 LIMIT %(param_1)s]
[parameters: {'key_1': 'dagster/schedule_name', 'value_1': 'hourly', 'key_2': '.dagster/repository', 'value_2': '__repository__@replicator-code', 'count_1': 2, 'param_1': 1}]
(Background on this error at: https://sqlalche.me/e/20/e3q8)

Path: ["scheduleOrError","scheduleState","runs"]

Locations: [{"line":82,"column":3}]

[nsteins]

Hi Niall, were you able to find a resolution to this issue? I am seeing very similar issues.

[NiallRees]

Dagster is incredibly chatty with its database, I had to bump the database instance size and it solved it