Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

sigterm not handled ? #29

Open
fvigotti opened this issue Sep 21, 2018 · 2 comments
Open

sigterm not handled ? #29

fvigotti opened this issue Sep 21, 2018 · 2 comments

Comments

@fvigotti
Copy link

Maybe It's my fault but seems that the process doesn't handle graceful process termination...
SIGTERM does nothing.... only sigkill works :(

@andpol
Copy link

andpol commented Jan 29, 2019

SIGTERM is handled https://github.com/andreas-schroeder/kafka-health-check/blob/master/main.go#L31, but in our setup with k8s, the container doesn't shutdown within a 60 second grace period. The logs indicate the check is trying to delete the broker health check topic, but it just waits there until getting a SIGKILL 60 seconds later. This is with Kafka 2.0.1

@andpol
Copy link

andpol commented Jan 29, 2019

In my case, it turned out the reason the health check was just sitting there was because the Kafka broker running in the same k8s pod was the topic leader, and it was responding to the pod sigterm quicker than the topic could be deleted. The topic leader needs to be alive to respond to the deletion request (via ZK). I fixed my setup by adding a 15 second "preStop" sleep to the Kafka broker in the pod.

While investigating, I did however find a bug during the shutdown process, preventing proper shutdown of the health heck, should be fixed in this PR: #32

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants