You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
SIGTERM is handled https://github.com/andreas-schroeder/kafka-health-check/blob/master/main.go#L31, but in our setup with k8s, the container doesn't shutdown within a 60 second grace period. The logs indicate the check is trying to delete the broker health check topic, but it just waits there until getting a SIGKILL 60 seconds later. This is with Kafka 2.0.1
In my case, it turned out the reason the health check was just sitting there was because the Kafka broker running in the same k8s pod was the topic leader, and it was responding to the pod sigterm quicker than the topic could be deleted. The topic leader needs to be alive to respond to the deletion request (via ZK). I fixed my setup by adding a 15 second "preStop" sleep to the Kafka broker in the pod.
While investigating, I did however find a bug during the shutdown process, preventing proper shutdown of the health heck, should be fixed in this PR: #32
Maybe It's my fault but seems that the process doesn't handle graceful process termination...
SIGTERM does nothing.... only sigkill works :(
The text was updated successfully, but these errors were encountered: