[K8sRunLauncher] Run was not terminated successfully; delete_job returned False

[DanilaBaklazhenkoWolt]

Hey!

We are running Dagster on K8s with k8s run launcher and multiprocess executor (single pod for the whole job run).
Everything runs smoothly, but the run termination from the UI is not working for some reason.

Every time we try to cancel something from UI it reports this:

RUN_CANCELLING   -  Sending run termination request.
ENGINE_EVENT  -  [K8sRunLauncher] Run was not terminated successfully; delete_job returned False

Looking into the delete_job function here it seems that the only way it can return False is when the Not Found error is raised somewhere in the python K8s client, but it obfuscates the error itself and it is impossible to debug it and understand what was the root cause.

Any ideas on how to approach it? Maybe there is a way to enable debug log level on the k8s client used by Dagster?
Or could it be a service account permission issue?