Detecting broken code location in Kubernetes deployment

[chrishiste]

We are running the open-source version of Dagster in a Kubernetes environment, with user deployments managed via a separate Helm chart.

We need to set up an alert system to notify us when the code location is broken. I'm open to configuring a custom Datadog monitor or setting up Kubernetes probes to detect these issues. However, I'm currently unsure how to extract the state of the code location from Dagster to trigger these alerts or probes.

[garethbrickman]

You could use the GraphQL API with the below query to check on the state / health of code locations:

query LocationStatusesQuery {
  workspaceOrError {
	  ... on Workspace{
      locationEntries {
        ... on WorkspaceLocationEntry {
          id
          loadStatus
          locationOrLoadError {
            __typename
          }
        }
      }
    }
  }
}

loadStatus will be LOADING or LOADED. You can use that additional locationOrLoadError field to distinguish between error and not. If the typename is PythonError, it loaded with an error state. If it's a RepositoryLocation it loaded in a working state.

Example response:

{'data': {'workspaceOrError': {'locationEntries': [{'id': 'code-server-1',
                                                    'loadStatus': 'LOADED',
                                                    'locationOrLoadError': {'__typename': 'RepositoryLocation'}},
                                                   {'id': 'code-server-2',
                                                    'loadStatus': 'LOADED',
                                                    'locationOrLoadError': {'__typename': 'RepositoryLocation'}},
                                                   {'id': 'code-server-3',
                                                    'loadStatus': 'LOADED',
                                                    'locationOrLoadError': {'__typename': 'PythonError'}}]}}}

(Worth noting that alerting when code locations fail to load due to an error is a native feature available in Dagster+ Alerts.)

[chrishiste]

Thanks should be all I need 👍

[chrishiste]

query {
  workspaceOrError {
    ... on Workspace {
      locationEntries {
        locationOrLoadError {
          ... on PythonError {
            message
          }
        }
      }
    }
  }
}

I ended using this