How do I get alerted when a scheduled run or backfill fails?

[NiallRees]

This morning, our scheduled run failed mid-run. The error message was dagster._core.errors.DagsterCodeLocationLoadError: Failure loading replicator-code: dagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server. gRPC Error code: UNAVAILABLE.

Two questions:

1. How do we make sure Dagster is resilient to the code server being down for a couple of minutes at any point in time (pod gets rescheduled)
2. How do we get alerted when this happens in the future

Thanks.

[prha]

Hi @NiallRees...

Was it the run that failed, or the schedule evaluation that failed?

If it was the run itself that failed, are you using the default run launcher? You can make runs resilient to code server downtime by launching runs as independent containers. The ecs / k8s / docker run launchers are examples of run launchers that isolate runs as separate containers. For run failures, you can set up retries or run failure sensor alerting.

For schedule failures, the scheduler daemon process should retry recently failed ticks, so it should tolerate ephemeral failures due to code server redeploys.

[NiallRees]

Hi @prha - got some more screenshots. It was the schedule that failed. I'm using the K8s helm chart. There was no retry unfortunately - it started 20 of the total partitions and then just stopped. Thanks for the help

[NiallRees]

Here are the logs from the daemon. Schedule starts at 06:00. It starts 20 runs/partitions out of over a hundred, then doesn't recover after the code server returns.

[32m2023-11-16 05:59:00 +0000[0m - dagster.daemon.SchedulerDaemon - [34mWARNING[0m - Using UTC as the timezone for hourly as it did not specify an execution_timezone in its definition.
[32m2023-11-16 05:59:00 +0000[0m - dagster.daemon.SchedulerDaemon - [34mINFO[0m - No new tick times to evaluate for hourly
[32m2023-11-16 05:59:06 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 05:59:06 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 05:59:11 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 05:59:12 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 05:59:16 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 05:59:16 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 05:59:17 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: update_snowflake_account_partitions
[32m2023-11-16 05:59:17 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor update_snowflake_account_partitions skipped: No account IDs added or removed
[32m2023-11-16 05:59:26 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 05:59:26 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 05:59:31 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 05:59:32 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 05:59:41 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 05:59:42 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 05:59:46 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 05:59:46 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 05:59:51 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 05:59:52 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: update_snowflake_account_partitions
[32m2023-11-16 05:59:52 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 05:59:52 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor update_snowflake_account_partitions skipped: No account IDs added or removed
[32m2023-11-16 05:59:56 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 05:59:56 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:00:00 +0000[0m - dagster.daemon.SchedulerDaemon - [34mINFO[0m - Evaluating schedule `hourly` at 2023-11-16 06:00:00 +0000
[32m2023-11-16 06:00:02 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:00:02 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:00:07 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:00:07 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:00:08 +0000[0m - dagster.daemon.SchedulerDaemon - [34mINFO[0m - Completed scheduled launch of run 36d78448-0d32-4b17-ba05-d2d6d1f470ce for hourly
[32m2023-11-16 06:00:12 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:00:12 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:00:13 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Retrieved 1 queued runs, checking limits.
[32m2023-11-16 06:00:14 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Launched 1 runs.
[32m2023-11-16 06:00:16 +0000[0m - dagster.daemon.SchedulerDaemon - [34mINFO[0m - Completed scheduled launch of run fb9fb24a-86c4-4829-a23d-d9244e817eba for hourly
[32m2023-11-16 06:00:17 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:00:17 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:00:18 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Retrieved 1 queued runs, checking limits.
[32m2023-11-16 06:00:19 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Launched 1 runs.
[32m2023-11-16 06:00:23 +0000[0m - dagster.daemon.SchedulerDaemon - [34mINFO[0m - Completed scheduled launch of run 03008cd9-707d-4b59-8283-e6c31aedd1b1 for hourly
[32m2023-11-16 06:00:24 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Retrieved 1 queued runs, checking limits.
[32m2023-11-16 06:00:24 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Launched 1 runs.
[32m2023-11-16 06:00:27 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:00:27 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:00:27 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: update_snowflake_account_partitions
[32m2023-11-16 06:00:28 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor update_snowflake_account_partitions skipped: No account IDs added or removed
[32m2023-11-16 06:00:31 +0000[0m - dagster.daemon.SchedulerDaemon - [34mINFO[0m - Completed scheduled launch of run 8a28f9ce-36f8-48ff-b5d3-aee98686b0ab for hourly
[32m2023-11-16 06:00:32 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:00:32 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:00:34 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Retrieved 1 queued runs, checking limits.
[32m2023-11-16 06:00:35 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Launched 1 runs.
[32m2023-11-16 06:00:37 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:00:37 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:00:40 +0000[0m - dagster.daemon.SchedulerDaemon - [34mINFO[0m - Completed scheduled launch of run d79f38c7-4394-4abe-b4bd-2293361ef961 for hourly
[32m2023-11-16 06:00:44 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Retrieved 1 queued runs, checking limits.
[32m2023-11-16 06:00:45 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Launched 1 runs.
[32m2023-11-16 06:00:47 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:00:47 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:00:48 +0000[0m - dagster.daemon.SchedulerDaemon - [34mINFO[0m - Completed scheduled launch of run 1a3e76db-e4c6-44a2-92a4-8d7a4ad06b82 for hourly
[32m2023-11-16 06:00:50 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Retrieved 1 queued runs, checking limits.
[32m2023-11-16 06:00:50 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Launched 1 runs.
[32m2023-11-16 06:00:54 +0000[0m - dagster.daemon.SchedulerDaemon - [34mINFO[0m - Completed scheduled launch of run 338e84d1-51cc-4df6-9c99-17eb0b9b54f5 for hourly
[32m2023-11-16 06:00:55 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Retrieved 1 queued runs, checking limits.
[32m2023-11-16 06:00:56 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Launched 1 runs.
[32m2023-11-16 06:00:57 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:00:57 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:01:00 +0000[0m - dagster.daemon.SchedulerDaemon - [34mINFO[0m - Checking for new runs for the following schedules: hourly
[32m2023-11-16 06:01:02 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: update_snowflake_account_partitions
[32m2023-11-16 06:01:02 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor update_snowflake_account_partitions skipped: No account IDs added or removed
[32m2023-11-16 06:01:03 +0000[0m - dagster.daemon.SchedulerDaemon - [34mINFO[0m - Completed scheduled launch of run e98bfed1-5355-44c1-9fab-b3c2cd53ccc6 for hourly
[32m2023-11-16 06:01:05 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Retrieved 1 queued runs, checking limits.
[32m2023-11-16 06:01:06 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Launched 1 runs.
[32m2023-11-16 06:01:07 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:01:07 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:01:10 +0000[0m - dagster.daemon.SchedulerDaemon - [34mINFO[0m - Completed scheduled launch of run 1abb3b6d-dcd4-4999-88df-aa74c620d1ea for hourly
[32m2023-11-16 06:01:11 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Retrieved 1 queued runs, checking limits.
[32m2023-11-16 06:01:11 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Launched 1 runs.
[32m2023-11-16 06:01:12 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:01:12 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:01:17 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:01:17 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:01:19 +0000[0m - dagster.daemon.SchedulerDaemon - [34mINFO[0m - Completed scheduled launch of run e26b1109-b0af-42a3-b97d-32e2aaaa18a5 for hourly
[32m2023-11-16 06:01:21 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Retrieved 1 queued runs, checking limits.
[32m2023-11-16 06:01:22 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Launched 1 runs.
[32m2023-11-16 06:01:22 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:01:22 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:01:26 +0000[0m - dagster.daemon.SchedulerDaemon - [34mINFO[0m - Completed scheduled launch of run 189c2372-370e-432d-a9f7-f4ca8690c9d2 for hourly
[32m2023-11-16 06:01:31 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Retrieved 1 queued runs, checking limits.
[32m2023-11-16 06:01:32 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:01:33 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Launched 1 runs.
[32m2023-11-16 06:01:33 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:01:33 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: update_snowflake_account_partitions
[32m2023-11-16 06:01:34 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor update_snowflake_account_partitions skipped: No account IDs added or removed
[32m2023-11-16 06:01:35 +0000[0m - dagster.daemon.SchedulerDaemon - [34mINFO[0m - Completed scheduled launch of run 2c022089-44e8-4108-a34e-02114df3595c for hourly
[32m2023-11-16 06:01:37 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Retrieved 1 queued runs, checking limits.
[32m2023-11-16 06:01:37 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Launched 1 runs.
[32m2023-11-16 06:01:42 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:01:42 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:01:43 +0000[0m - dagster.daemon.SchedulerDaemon - [34mINFO[0m - Completed scheduled launch of run c0dd9e3e-abc8-474a-ae3d-0cd741e54cb8 for hourly
[32m2023-11-16 06:01:47 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:01:47 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Retrieved 1 queued runs, checking limits.
[32m2023-11-16 06:01:47 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:01:48 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Launched 1 runs.
[32m2023-11-16 06:01:51 +0000[0m - dagster.daemon.SchedulerDaemon - [34mINFO[0m - Completed scheduled launch of run 193469ed-067f-4810-bb61-de915ef69435 for hourly
[32m2023-11-16 06:01:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Collected 13 runs for monitoring
[32m2023-11-16 06:01:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Checking run c0dd9e3e-abc8-474a-ae3d-0cd741e54cb8
[32m2023-11-16 06:01:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Checking run 2c022089-44e8-4108-a34e-02114df3595c
[32m2023-11-16 06:01:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Checking run 189c2372-370e-432d-a9f7-f4ca8690c9d2
[32m2023-11-16 06:01:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Checking run e26b1109-b0af-42a3-b97d-32e2aaaa18a5
[32m2023-11-16 06:01:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Checking run 1abb3b6d-dcd4-4999-88df-aa74c620d1ea
[32m2023-11-16 06:01:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Checking run e98bfed1-5355-44c1-9fab-b3c2cd53ccc6
[32m2023-11-16 06:01:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Checking run 338e84d1-51cc-4df6-9c99-17eb0b9b54f5
[32m2023-11-16 06:01:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Checking run 1a3e76db-e4c6-44a2-92a4-8d7a4ad06b82
[32m2023-11-16 06:01:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Checking run d79f38c7-4394-4abe-b4bd-2293361ef961
[32m2023-11-16 06:01:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Checking run 8a28f9ce-36f8-48ff-b5d3-aee98686b0ab
[32m2023-11-16 06:01:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Checking run 03008cd9-707d-4b59-8283-e6c31aedd1b1
[32m2023-11-16 06:01:52 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Retrieved 1 queued runs, checking limits.
[32m2023-11-16 06:01:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Checking run fb9fb24a-86c4-4829-a23d-d9244e817eba
[32m2023-11-16 06:01:53 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Checking run 36d78448-0d32-4b17-ba05-d2d6d1f470ce
[32m2023-11-16 06:01:53 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Launched 1 runs.
[32m2023-11-16 06:01:57 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:01:57 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:01:59 +0000[0m - dagster.daemon.SchedulerDaemon - [34mINFO[0m - Completed scheduled launch of run d22098a7-dc64-4411-8784-e65298b889ee for hourly
[32m2023-11-16 06:02:02 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:02:02 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:02:03 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Retrieved 1 queued runs, checking limits.
[32m2023-11-16 06:02:04 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Launched 1 runs.
[32m2023-11-16 06:02:07 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: update_snowflake_account_partitions
[32m2023-11-16 06:02:07 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor update_snowflake_account_partitions skipped: No account IDs added or removed
[32m2023-11-16 06:02:08 +0000[0m - dagster.daemon.SchedulerDaemon - [34mINFO[0m - Completed scheduled launch of run 1d68e854-4487-43af-8192-8dc3e668896e for hourly
[32m2023-11-16 06:02:08 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Retrieved 1 queued runs, checking limits.
[32m2023-11-16 06:02:09 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Launched 1 runs.
[32m2023-11-16 06:02:12 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:02:12 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:02:16 +0000[0m - dagster.daemon.SchedulerDaemon - [34mINFO[0m - Completed scheduled launch of run ed759d72-966e-4737-8cbd-1ddfe2da518e for hourly
[32m2023-11-16 06:02:17 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:02:17 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:02:19 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Retrieved 1 queued runs, checking limits.
[32m2023-11-16 06:02:19 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Launched 1 runs.
[32m2023-11-16 06:02:24 +0000[0m - dagster.daemon.SchedulerDaemon - [34mINFO[0m - Completed scheduled launch of run 6bed9723-500a-4cf8-a7da-5bf9957d0f8d for hourly
[32m2023-11-16 06:02:24 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Retrieved 1 queued runs, checking limits.
[32m2023-11-16 06:02:25 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Launched 1 runs.
[32m2023-11-16 06:02:27 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:02:27 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:02:31 +0000[0m - dagster.daemon.SchedulerDaemon - [34mINFO[0m - Completed scheduled launch of run 64334995-95e8-4daf-9fa9-0b5ce6097d5c for hourly
[32m2023-11-16 06:02:35 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Retrieved 1 queued runs, checking limits.
[32m2023-11-16 06:02:35 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Launched 1 runs.
[32m2023-11-16 06:02:37 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:02:38 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: update_snowflake_account_partitions
[32m2023-11-16 06:02:38 +0000[0m - dagster.daemon.SensorDaemon - [34mERROR[0m - [31mSensor daemon caught an error for sensor slack_on_run_failure[0m
Traceback (most recent call last):\n  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 180, in _streaming_query\n    yield from self._get_streaming_response(\n  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 169, in _get_streaming_response\n    yield from getattr(stub, method)(request, metadata=self._metadata, timeout=timeout)\n  File \"/usr/local/lib/python3.10/site-packages/grpc/_channel.py\", line 541, in __next__\n    return self._next()\n  File \"/usr/local/lib/python3.10/site-packages/grpc/_channel.py\", line 967, in _next\n    raise self\ngrpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
\tstatus = StatusCode.UNAVAILABLE
\tdetails = \"failed to connect to all addresses; last error: UNKNOWN: ipv4:10.2.0.132:3030: Failed to connect to remote host: Connection refused\"
\tdebug_error_string = \"UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:10.2.0.132:3030: Failed to connect to remote host: Connection refused {created_time:\"2023-11-16T06:02:37.683504095+00:00\", grpc_status:14}\"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):\n  File \"/usr/local/lib/python3.10/site-packages/dagster/_daemon/sensor.py\", line 534, in _process_tick_generator\n    yield from _evaluate_sensor(\n  File \"/usr/local/lib/python3.10/site-packages/dagster/_daemon/sensor.py\", line 666, in _evaluate_sensor\n    sensor_runtime_data = code_location.get_external_sensor_execution_data(\n  File \"/usr/local/lib/python3.10/site-packages/dagster/_core/host_representation/code_location.py\", line 859, in get_external_sensor_execution_data\n    return sync_get_external_sensor_execution_data_grpc(\n  File \"/usr/local/lib/python3.10/site-packages/dagster/_api/snapshot_sensor.py\", line 63, in sync_get_external_sensor_execution_data_grpc\n    api_client.external_sensor_execution(\n  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 391, in external_sensor_execution\n    chunks = list(\n  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 184, in _streaming_query\n    self._raise_grpc_exception(\n  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 140, in _raise_grpc_exception\n    raise DagsterUserCodeUnreachableError(\ndagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server. gRPC Error code: UNAVAILABLE
/usr/local/lib/python3.10/site-packages/dagster/_core/workspace/context.py:613: UserWarning: Error loading repository location replicator-code:dagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server. gRPC Error code: UNAVAILABLE\n
Stack Trace:
  File \"/usr/local/lib/python3.10/site-packages/dagster/_core/workspace/context.py\", line 608, in _load_location
    origin.reload_location(self.instance) if reload else origin.create_location()
  File \"/usr/local/lib/python3.10/site-packages/dagster/_core/host_representation/origin.py\", line 368, in create_location
    return GrpcServerCodeLocation(self)
  File \"/usr/local/lib/python3.10/site-packages/dagster/_core/host_representation/code_location.py\", line 595, in __init__
    list_repositories_response = sync_list_repositories_grpc(self.client)
  File \"/usr/local/lib/python3.10/site-packages/dagster/_api/list_repositories.py\", line 20, in sync_list_repositories_grpc
    api_client.list_repositories(),
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 229, in list_repositories
    res = self._query(\"ListRepositories\", api_pb2.ListRepositoriesRequest)
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 157, in _query
    self._raise_grpc_exception(
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 140, in _raise_grpc_exception
    raise DagsterUserCodeUnreachableError(
The above exception was caused by the following exception:
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
\tstatus = StatusCode.UNAVAILABLE
\tdetails = \"failed to connect to all addresses; last error: UNKNOWN: ipv4:10.2.0.132:3030: Failed to connect to remote host: Connection refused\"
\tdebug_error_string = \"UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:10.2.0.132:3030: Failed to connect to remote host: Connection refused {created_time:\"2023-11-16T06:02:39.089517802+00:00\", grpc_status:14}\"
>
Stack Trace:
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 155, in _query
    return self._get_response(method, request=request_type(**kwargs), timeout=timeout)
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 130, in _get_response
    return getattr(stub, method)(request, metadata=self._metadata, timeout=timeout)
  File \"/usr/local/lib/python3.10/site-packages/grpc/_channel.py\", line 1161, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File \"/usr/local/lib/python3.10/site-packages/grpc/_channel.py\", line 1004, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
  warnings.warn(f\"Error loading repository location {location_name}:{error.to_string()}\")
[32m2023-11-16 06:02:39 +0000[0m - dagster.daemon.SensorDaemon - [34mERROR[0m - [31mSensor daemon caught an error for sensor update_snowflake_account_partitions[0m
Traceback (most recent call last):\n  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 180, in _streaming_query\n    yield from self._get_streaming_response(\n  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 169, in _get_streaming_response\n    yield from getattr(stub, method)(request, metadata=self._metadata, timeout=timeout)\n  File \"/usr/local/lib/python3.10/site-packages/grpc/_channel.py\", line 541, in __next__\n    return self._next()\n  File \"/usr/local/lib/python3.10/site-packages/grpc/_channel.py\", line 967, in _next\n    raise self\ngrpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
\tstatus = StatusCode.UNAVAILABLE
\tdetails = \"failed to connect to all addresses; last error: UNKNOWN: ipv4:10.2.0.132:3030: Failed to connect to remote host: Connection refused\"
\tdebug_error_string = \"UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:10.2.0.132:3030: Failed to connect to remote host: Connection refused {grpc_status:14, created_time:\"2023-11-16T06:02:38.283340576+00:00\"}\"
>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):\n  File \"/usr/local/lib/python3.10/site-packages/dagster/_daemon/sensor.py\", line 534, in _process_tick_generator\n    yield from _evaluate_sensor(\n  File \"/usr/local/lib/python3.10/site-packages/dagster/_daemon/sensor.py\", line 666, in _evaluate_sensor\n    sensor_runtime_data = code_location.get_external_sensor_execution_data(\n  File \"/usr/local/lib/python3.10/site-packages/dagster/_core/host_representation/code_location.py\", line 859, in get_external_sensor_execution_data\n    return sync_get_external_sensor_execution_data_grpc(\n  File \"/usr/local/lib/python3.10/site-packages/dagster/_api/snapshot_sensor.py\", line 63, in sync_get_external_sensor_execution_data_grpc\n    api_client.external_sensor_execution(\n  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 391, in external_sensor_execution\n    chunks = list(\n  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 184, in _streaming_query\n    self._raise_grpc_exception(\n  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 140, in _raise_grpc_exception\n    raise DagsterUserCodeUnreachableError(\ndagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server. gRPC Error code: UNAVAILABLE
[32m2023-11-16 06:02:40 +0000[0m - dagster.daemon.SchedulerDaemon - [34mINFO[0m - Completed scheduled launch of run 0773985c-b34a-4aee-95f1-90a7daec5ddd for hourly
[32m2023-11-16 06:02:40 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Retrieved 1 queued runs, checking limits.
[32m2023-11-16 06:02:40 +0000[0m - dagster.daemon.QueuedRunCoordinatorDaemon - [34mINFO[0m - Launched 1 runs.
[32m2023-11-16 06:02:51 +0000[0m - dagster.daemon.SensorDaemon - [34mWARNING[0m - Could not load location replicator-code to check for sensors due to the following error: dagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server. gRPC Error code: UNAVAILABLE\n
Stack Trace:
  File \"/usr/local/lib/python3.10/site-packages/dagster/_core/workspace/context.py\", line 608, in _load_location
    origin.reload_location(self.instance) if reload else origin.create_location()
  File \"/usr/local/lib/python3.10/site-packages/dagster/_core/host_representation/origin.py\", line 368, in create_location
    return GrpcServerCodeLocation(self)
  File \"/usr/local/lib/python3.10/site-packages/dagster/_core/host_representation/code_location.py\", line 595, in __init__
    list_repositories_response = sync_list_repositories_grpc(self.client)
  File \"/usr/local/lib/python3.10/site-packages/dagster/_api/list_repositories.py\", line 20, in sync_list_repositories_grpc
    api_client.list_repositories(),
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 229, in list_repositories
    res = self._query(\"ListRepositories\", api_pb2.ListRepositoriesRequest)
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 157, in _query
    self._raise_grpc_exception(
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 140, in _raise_grpc_exception
    raise DagsterUserCodeUnreachableError(
The above exception was caused by the following exception:
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
\tstatus = StatusCode.UNAVAILABLE
\tdetails = \"failed to connect to all addresses; last error: UNKNOWN: ipv4:10.2.0.132:3030: Failed to connect to remote host: Connection refused\"
\tdebug_error_string = \"UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:10.2.0.132:3030: Failed to connect to remote host: Connection refused {created_time:\"2023-11-16T06:02:39.089517802+00:00\", grpc_status:14}\"
>
Stack Trace:
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 155, in _query
    return self._get_response(method, request=request_type(**kwargs), timeout=timeout)
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 130, in _get_response
    return getattr(stub, method)(request, metadata=self._metadata, timeout=timeout)
  File \"/usr/local/lib/python3.10/site-packages/grpc/_channel.py\", line 1161, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File \"/usr/local/lib/python3.10/site-packages/grpc/_channel.py\", line 1004, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
[32m2023-11-16 06:02:51 +0000[0m - dagster.daemon.SensorDaemon - [34mWARNING[0m - Sensor update_snowflake_account_partitions was started from a location replicator-code that can no longer be found in the workspace. You can turn off this sensor in the Dagster UI from the Status tab.
[32m2023-11-16 06:02:51 +0000[0m - dagster.daemon.SensorDaemon - [34mWARNING[0m - Sensor slack_on_run_failure was started from a location replicator-code that can no longer be found in the workspace. You can turn off this sensor in the Dagster UI from the Status tab.
[32m2023-11-16 06:03:00 +0000[0m - dagster.daemon.SchedulerDaemon - [34mWARNING[0m - Could not load location replicator-code to check for schedules due to the following error: dagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server. gRPC Error code: UNAVAILABLE\n
Stack Trace:
  File \"/usr/local/lib/python3.10/site-packages/dagster/_core/workspace/context.py\", line 608, in _load_location
    origin.reload_location(self.instance) if reload else origin.create_location()
  File \"/usr/local/lib/python3.10/site-packages/dagster/_core/host_representation/origin.py\", line 368, in create_location
    return GrpcServerCodeLocation(self)
  File \"/usr/local/lib/python3.10/site-packages/dagster/_core/host_representation/code_location.py\", line 595, in __init__
    list_repositories_response = sync_list_repositories_grpc(self.client)
  File \"/usr/local/lib/python3.10/site-packages/dagster/_api/list_repositories.py\", line 20, in sync_list_repositories_grpc
    api_client.list_repositories(),
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 229, in list_repositories
    res = self._query(\"ListRepositories\", api_pb2.ListRepositoriesRequest)
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 157, in _query
    self._raise_grpc_exception(
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 140, in _raise_grpc_exception
    raise DagsterUserCodeUnreachableError(
The above exception was caused by the following exception:
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
\tstatus = StatusCode.UNAVAILABLE
\tdetails = \"failed to connect to all addresses; last error: UNKNOWN: ipv4:10.2.0.132:3030: Failed to connect to remote host: Connection refused\"
\tdebug_error_string = \"UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:10.2.0.132:3030: Failed to connect to remote host: Connection refused {created_time:\"2023-11-16T06:02:39.089517802+00:00\", grpc_status:14}\"
>
Stack Trace:
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 155, in _query
    return self._get_response(method, request=request_type(**kwargs), timeout=timeout)
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 130, in _get_response
    return getattr(stub, method)(request, metadata=self._metadata, timeout=timeout)
  File \"/usr/local/lib/python3.10/site-packages/grpc/_channel.py\", line 1161, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File \"/usr/local/lib/python3.10/site-packages/grpc/_channel.py\", line 1004, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
[32m2023-11-16 06:03:00 +0000[0m - dagster.daemon.SchedulerDaemon - [34mWARNING[0m - Schedule hourly was started from a location replicator-code that can no longer be found in the workspace. You can turn off this schedule in the Dagster UI from the Status tab.
[32m2023-11-16 06:03:03 +0000[0m - dagster-webserver - [34mINFO[0m - Received LocationStateChangeEventType.LOCATION_ERROR event for location replicator-code, refreshing
/usr/local/lib/python3.10/site-packages/dagster/_core/workspace/context.py:613: UserWarning: Error loading repository location replicator-code:dagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server. gRPC Error code: UNAVAILABLE\n
Stack Trace:
  File \"/usr/local/lib/python3.10/site-packages/dagster/_core/workspace/context.py\", line 608, in _load_location
    origin.reload_location(self.instance) if reload else origin.create_location()
  File \"/usr/local/lib/python3.10/site-packages/dagster/_core/host_representation/origin.py\", line 368, in create_location
    return GrpcServerCodeLocation(self)
  File \"/usr/local/lib/python3.10/site-packages/dagster/_core/host_representation/code_location.py\", line 595, in __init__
    list_repositories_response = sync_list_repositories_grpc(self.client)
  File \"/usr/local/lib/python3.10/site-packages/dagster/_api/list_repositories.py\", line 20, in sync_list_repositories_grpc
    api_client.list_repositories(),
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 229, in list_repositories
    res = self._query(\"ListRepositories\", api_pb2.ListRepositoriesRequest)
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 157, in _query
    self._raise_grpc_exception(
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 140, in _raise_grpc_exception
    raise DagsterUserCodeUnreachableError(
The above exception was caused by the following exception:
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
\tstatus = StatusCode.UNAVAILABLE
\tdetails = \"failed to connect to all addresses; last error: UNKNOWN: ipv4:10.2.0.132:3030: Failed to connect to remote host: FD Shutdown\"
\tdebug_error_string = \"UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:10.2.0.132:3030: Failed to connect to remote host: FD Shutdown {created_time:\"2023-11-16T06:03:23.138466147+00:00\", grpc_status:14}\"
>
Stack Trace:
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 155, in _query
    return self._get_response(method, request=request_type(**kwargs), timeout=timeout)
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 130, in _get_response
    return getattr(stub, method)(request, metadata=self._metadata, timeout=timeout)
  File \"/usr/local/lib/python3.10/site-packages/grpc/_channel.py\", line 1161, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File \"/usr/local/lib/python3.10/site-packages/grpc/_channel.py\", line 1004, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
The above exception occurred during handling of the following exception:
dagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server. gRPC Error code: UNAVAILABLE\n
Stack Trace:
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/server_watcher.py\", line 119, in watch_grpc_server_thread
    watch_for_changes()
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/server_watcher.py\", line 82, in watch_for_changes
    new_server_id = client.get_server_id(timeout=REQUEST_TIMEOUT)
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 214, in get_server_id
    res = self._query(\"GetServerId\", api_pb2.Empty, timeout=timeout)
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 157, in _query
    self._raise_grpc_exception(
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 140, in _raise_grpc_exception
    raise DagsterUserCodeUnreachableError(
The above exception was caused by the following exception:
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
\tstatus = StatusCode.UNAVAILABLE
\tdetails = \"failed to connect to all addresses; last error: UNKNOWN: ipv4:10.2.0.132:3030: Failed to connect to remote host: Connection refused\"
\tdebug_error_string = \"UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:10.2.0.132:3030: Failed to connect to remote host: Connection refused {created_time:\"2023-11-16T06:02:39.101581057+00:00\", grpc_status:14}\"
>
Stack Trace:
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 155, in _query
    return self._get_response(method, request=request_type(**kwargs), timeout=timeout)
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 130, in _get_response
    return getattr(stub, method)(request, metadata=self._metadata, timeout=timeout)
  File \"/usr/local/lib/python3.10/site-packages/grpc/_channel.py\", line 1161, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File \"/usr/local/lib/python3.10/site-packages/grpc/_channel.py\", line 1004, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
  warnings.warn(f\"Error loading repository location {location_name}:{error.to_string()}\")
[32m2023-11-16 06:03:52 +0000[0m - dagster.daemon.SensorDaemon - [34mWARNING[0m - Could not load location replicator-code to check for sensors due to the following error: dagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server. gRPC Error code: UNAVAILABLE\n
Stack Trace:
  File \"/usr/local/lib/python3.10/site-packages/dagster/_core/workspace/context.py\", line 608, in _load_location
    origin.reload_location(self.instance) if reload else origin.create_location()
  File \"/usr/local/lib/python3.10/site-packages/dagster/_core/host_representation/origin.py\", line 368, in create_location
    return GrpcServerCodeLocation(self)
  File \"/usr/local/lib/python3.10/site-packages/dagster/_core/host_representation/code_location.py\", line 595, in __init__
    list_repositories_response = sync_list_repositories_grpc(self.client)
  File \"/usr/local/lib/python3.10/site-packages/dagster/_api/list_repositories.py\", line 20, in sync_list_repositories_grpc
    api_client.list_repositories(),
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 229, in list_repositories
    res = self._query(\"ListRepositories\", api_pb2.ListRepositoriesRequest)
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 157, in _query
    self._raise_grpc_exception(
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 140, in _raise_grpc_exception
    raise DagsterUserCodeUnreachableError(
The above exception was caused by the following exception:
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
\tstatus = StatusCode.UNAVAILABLE
\tdetails = \"failed to connect to all addresses; last error: UNKNOWN: ipv4:10.2.0.132:3030: Failed to connect to remote host: FD Shutdown\"
\tdebug_error_string = \"UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:10.2.0.132:3030: Failed to connect to remote host: FD Shutdown {created_time:\"2023-11-16T06:03:23.138466147+00:00\", grpc_status:14}\"
>
Stack Trace:
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 155, in _query
    return self._get_response(method, request=request_type(**kwargs), timeout=timeout)
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 130, in _get_response
    return getattr(stub, method)(request, metadata=self._metadata, timeout=timeout)
  File \"/usr/local/lib/python3.10/site-packages/grpc/_channel.py\", line 1161, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File \"/usr/local/lib/python3.10/site-packages/grpc/_channel.py\", line 1004, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
The above exception occurred during handling of the following exception:
dagster._core.errors.DagsterUserCodeUnreachableError: Could not reach user code server. gRPC Error code: UNAVAILABLE\n
Stack Trace:
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/server_watcher.py\", line 119, in watch_grpc_server_thread
    watch_for_changes()
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/server_watcher.py\", line 82, in watch_for_changes
    new_server_id = client.get_server_id(timeout=REQUEST_TIMEOUT)
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 214, in get_server_id
    res = self._query(\"GetServerId\", api_pb2.Empty, timeout=timeout)
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 157, in _query
    self._raise_grpc_exception(
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 140, in _raise_grpc_exception
    raise DagsterUserCodeUnreachableError(
The above exception was caused by the following exception:
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
\tstatus = StatusCode.UNAVAILABLE
\tdetails = \"failed to connect to all addresses; last error: UNKNOWN: ipv4:10.2.0.132:3030: Failed to connect to remote host: Connection refused\"
\tdebug_error_string = \"UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:10.2.0.132:3030: Failed to connect to remote host: Connection refused {created_time:\"2023-11-16T06:02:39.101581057+00:00\", grpc_status:14}\"
>
Stack Trace:
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 155, in _query
    return self._get_response(method, request=request_type(**kwargs), timeout=timeout)
  File \"/usr/local/lib/python3.10/site-packages/dagster/_grpc/client.py\", line 130, in _get_response
    return getattr(stub, method)(request, metadata=self._metadata, timeout=timeout)
  File \"/usr/local/lib/python3.10/site-packages/grpc/_channel.py\", line 1161, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File \"/usr/local/lib/python3.10/site-packages/grpc/_channel.py\", line 1004, in _end_unary_response_blocking
    raise _InactiveRpcError(state)  # pytype: disable=not-instantiable
[32m2023-11-16 06:03:52 +0000[0m - dagster.daemon.SensorDaemon - [34mWARNING[0m - Sensor update_snowflake_account_partitions was started from a location replicator-code that can no longer be found in the workspace. You can turn off this sensor in the Dagster UI from the Status tab.
[32m2023-11-16 06:03:52 +0000[0m - dagster.daemon.SensorDaemon - [34mWARNING[0m - Sensor slack_on_run_failure was started from a location replicator-code that can no longer be found in the workspace. You can turn off this sensor in the Dagster UI from the Status tab.
[32m2023-11-16 06:03:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Collected 20 runs for monitoring
[32m2023-11-16 06:03:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Checking run 0773985c-b34a-4aee-95f1-90a7daec5ddd
[32m2023-11-16 06:03:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Checking run 64334995-95e8-4daf-9fa9-0b5ce6097d5c
[32m2023-11-16 06:03:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Checking run 6bed9723-500a-4cf8-a7da-5bf9957d0f8d
[32m2023-11-16 06:03:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Checking run ed759d72-966e-4737-8cbd-1ddfe2da518e
[32m2023-11-16 06:03:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Checking run 1d68e854-4487-43af-8192-8dc3e668896e
[32m2023-11-16 06:03:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Checking run d22098a7-dc64-4411-8784-e65298b889ee
[32m2023-11-16 06:03:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Checking run 193469ed-067f-4810-bb61-de915ef69435
[32m2023-11-16 06:03:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Checking run c0dd9e3e-abc8-474a-ae3d-0cd741e54cb8
[32m2023-11-16 06:03:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Checking run 2c022089-44e8-4108-a34e-02114df3595c
[32m2023-11-16 06:03:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Checking run 189c2372-370e-432d-a9f7-f4ca8690c9d2
[32m2023-11-16 06:03:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Checking run e26b1109-b0af-42a3-b97d-32e2aaaa18a5
[32m2023-11-16 06:03:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Checking run 1abb3b6d-dcd4-4999-88df-aa74c620d1ea
[32m2023-11-16 06:03:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Checking run e98bfed1-5355-44c1-9fab-b3c2cd53ccc6
[32m2023-11-16 06:03:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Checking run 338e84d1-51cc-4df6-9c99-17eb0b9b54f5
[32m2023-11-16 06:03:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Checking run 1a3e76db-e4c6-44a2-92a4-8d7a4ad06b82
[32m2023-11-16 06:03:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Checking run d79f38c7-4394-4abe-b4bd-2293361ef961
[32m2023-11-16 06:03:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Checking run 8a28f9ce-36f8-48ff-b5d3-aee98686b0ab
[32m2023-11-16 06:03:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Checking run 03008cd9-707d-4b59-8283-e6c31aedd1b1
[32m2023-11-16 06:03:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Checking run fb9fb24a-86c4-4829-a23d-d9244e817eba
[32m2023-11-16 06:03:52 +0000[0m - dagster.daemon.MonitoringDaemon - [34mINFO[0m - Checking run 36d78448-0d32-4b17-ba05-d2d6d1f470ce
[32m2023-11-16 06:03:54 +0000[0m - dagster-webserver - [34mINFO[0m - Received LocationStateChangeEventType.LOCATION_UPDATED event for location replicator-code, refreshing
[32m2023-11-16 06:03:57 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:03:57 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: update_snowflake_account_partitions
[32m2023-11-16 06:04:06 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:04:06 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor update_snowflake_account_partitions skipped: No account IDs added or removed
[32m2023-11-16 06:04:07 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:04:08 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:04:17 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:04:17 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:04:22 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:04:22 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:04:27 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:04:27 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:04:32 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: update_snowflake_account_partitions
[32m2023-11-16 06:04:33 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor update_snowflake_account_partitions skipped: No account IDs added or removed
[32m2023-11-16 06:04:37 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:04:38 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:04:47 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:04:47 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:04:52 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:04:52 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:04:58 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:04:58 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:05:00 +0000[0m - dagster.daemon.SchedulerDaemon - [34mINFO[0m - Checking for new runs for the following schedules: hourly
[32m2023-11-16 06:05:00 +0000[0m - dagster.daemon.SchedulerDaemon - [34mWARNING[0m - Using UTC as the timezone for hourly as it did not specify an execution_timezone in its definition.
[32m2023-11-16 06:05:00 +0000[0m - dagster.daemon.SchedulerDaemon - [34mINFO[0m - No new tick times to evaluate for hourly
[32m2023-11-16 06:05:02 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: update_snowflake_account_partitions
[32m2023-11-16 06:05:02 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor update_snowflake_account_partitions skipped: No account IDs added or removed
[32m2023-11-16 06:05:07 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:05:07 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:05:12 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:05:12 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:05:17 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:05:18 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result
[32m2023-11-16 06:05:22 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Checking for new runs for sensor: slack_on_run_failure
[32m2023-11-16 06:05:22 +0000[0m - dagster.daemon.SensorDaemon - [34mINFO[0m - Sensor slack_on_run_failure skipped: Sensor function returned an empty result

[prha]

I see... I think tick retries are a buried setting that is not exposed on the helm chart. I just merged #18094, which should allow you to enable tick retries once it's released (probably the week of the 26th).

[NiallRees]

Awesome tysm @prha - once released I’ll manually kill the code server pod and let you know if resolved

[NiallRees]

@prha - how do you work out what to set maxTickRetries to? How often is a tick attempted?

[NiallRees]

Ideally it would be possible to set something like maxScheduleRetryDuration which could be set to e.g. 5 minutes

[NiallRees]

Some more information - this was Dagster 1.5.6, by which point this PR had been released. cc @gibsondan