-
Notifications
You must be signed in to change notification settings - Fork 300
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Slack exceeding rate limit as SlackAPIRateLimitError is not caught #5204
Comments
Upgraded to latest v1.13.11 and we are still getting this issue:- �[2024-12-20 14:18:55,842 source=engine:celery worker=ForkPoolWorker-2 task_id=0dfed72e-00f1-4d68-8915-8bf1e15142e9 task_name=apps.slack.tasks.post_slack_rate_limit_message name=apps.slack.client level=ERROR Slack API call error! slack_team_identity=1 args=('chat.postMessage',) kwargs={'json': {'channel': 'C0765EM3Y3E', 'text': 'Delivering Alert Groups in Slack, for integration XXXXX is temporarily rate-limited (due to a Slack rate-limit). Meanwhile, you can still find new Alert Groups in the <http://oncall-grafana:80/a/grafana-oncall-app/alert-groups?integration=CKV6YCHXWVRHK&status=0|"Alert Groups" web page>'}} status=200 error=message_limit_exceeded response={'ok': False, 'error': 'message_limit_exceeded'}�[ The above exception was the direct cause of the following exception: Traceback (most recent call last): |
What went wrong?
What happened:
I stopped receiving messages in Slack workspace (Free). I the celery logs I get the following logs:-
`�[1;31m2024-10-23 12:47:46,985 source=engine:celery worker=ForkPoolWorker-19 task_id=b3e6c7d2-259f-4ed1-9dc7-c118390b2ddf task_name=apps.slack.tasks.post_slack_rate_limit_message name=apps.slack.client level=ERROR Slack API call error! slack_team_identity=1 args=('chat.postMessage',) kwargs={'json': {'channel': 'XXXXXXXXXX', 'text': 'Delivering and updating alert groups of integration xxxxxx in Slack is temporarily stopped due to rate limit. You could find new alert groups at <http://oncall-grafana:80/a/grafana-oncall-app/alert-groups?integration=XXXXXXXXXXX&status=0|web page "Alert Groups">'}} status=200 error=message_limit_exceeded response={'ok': False, 'error': 'message_limit_exceeded'}�[0m
�[1;33m2024-10-23 12:47:46,985 source=engine:celery worker=ForkPoolWorker-19 task_id=b3e6c7d2-259f-4ed1-9dc7-c118390b2ddf task_name=apps.slack.tasks.post_slack_rate_limit_message name=common.custom_celery_tasks.dedicated_queue_retry_task level=WARNING Retrying celery task
Traceback (most recent call last):
File "/etc/app/apps/slack/client.py", line 128, in api_call
response = super().api_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/slack_sdk/web/base_client.py", line 156, in api_call
return self._sync_send(api_url=api_url, req_args=req_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/slack_sdk/web/base_client.py", line 187, in _sync_send
return self._urllib_api_call(
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/slack_sdk/web/base_client.py", line 317, in _urllib_api_call
).validate()
^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/slack_sdk/web/slack_response.py", line 199, in validate
raise e.SlackApiError(message=msg, response=self)
slack_sdk.errors.SlackApiError: The request to the Slack API failed. (url: https://www.slack.com/api/chat.postMessage)
The server responded with: {'ok': False, 'error': 'message_limit_exceeded'}
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/celery/app/autoretry.py", line 38, in run
return task._orig_run(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/etc/app/apps/slack/tasks.py", line 314, in post_slack_rate_limit_message
post_message_to_channel(integration.organization, slack_channel, text)
File "/etc/app/apps/slack/utils.py", line 75, in post_message_to_channel
slack_client.chat_postMessage(channel=channel_id, text=text)
File "/usr/local/lib/python3.12/site-packages/slack_sdk/web/client.py", line 2220, in chat_postMessage
return self.api_call("chat.postMessage", json=kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/etc/app/apps/slack/client.py", line 153, in api_call
raise error_class(e.response) from e
apps.slack.errors.SlackAPIRatelimitError: Slack API error! Response: {'ok': False, 'error': 'message_limit_exceeded'}�[0m
2024-10-23 12:47:46,988 source=engine:celery worker=MainProcess task_id=??? task_name=??? name=celery.worker.strategy level=INFO Task apps.slack.tasks.post_slack_rate_limit_message[b3e6c7d2-259f-4ed1-9dc7-c118390b2ddf] received
2024-10-23 12:47:46,988 source=engine:celery worker=ForkPoolWorker-19 task_id=b3e6c7d2-259f-4ed1-9dc7-c118390b2ddf task_name=apps.slack.tasks.post_slack_rate_limit_message name=celery.app.trace level=INFO Task apps.slack.tasks.post_slack_rate_limit_message[b3e6c7d2-259f-4ed1-9dc7-c118390b2ddf] retry: Retry in 7s: SlackAPIRatelimitError("Slack API error! Response: {'ok': False, 'error': 'message_limit_exceeded'}")
2024-10-23 12:47:46,988 source=engine:celery worker=ForkPoolWorker-19 task_id=b3e6c7d2-259f-4ed1-9dc7-c118390b2ddf task_name=apps.slack.tasks.post_slack_rate_limit_message name=engine.celery level=INFO task finished args=[10]
2024-10-23 12:47:53,987 source=engine:celery worker=ForkPoolWorker-19 task_id=b3e6c7d2-259f-4ed1-9dc7-c118390b2ddf task_name=apps.slack.tasks.post_slack_rate_limit_message name=engine.celery level=INFO task started args=[10]
�[1;31m2024-10-23 12:47:54,149 source=engine:celery worker=ForkPoolWorker-19 task_id=b3e6c7d2-259f-4ed1-9dc7-c118390b2ddf task_name=apps.slack.tasks.post_slack_rate_limit_message name=apps.slack.client level=ERROR Slack API call error! slack_team_identity=1 args=('chat.postMessage',) kwargs={'json': {'channel': 'XXXXXXXXX', 'text': 'Delivering and updating alert groups of integration XXXXXXXX in Slack is temporarily stopped due to rate limit. You could find new alert groups at <http://oncall-grafana:80/a/grafana-oncall-app/alert-groups?integration=XXXXXXXXX&status=0|web page "Alert Groups">'}} status=200 error=message_limit_exceeded response={'ok': False, 'error': 'message_limit_exceeded'}�[0m
�[1;33m2024-10-23 12:47:54,149 source=engine:celery worker=ForkPoolWorker-19 task_id=b3e6c7d2-259f-4ed1-9dc7-c118390b2ddf task_name=apps.slack.tasks.post_slack_rate_limit_message name=common.custom_celery_tasks.dedicated_queue_retry_task level=WARNING Retrying celery task
Traceback (most recent call last):
File "/etc/app/apps/slack/client.py", line 128, in api_call
response = super().api_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/slack_sdk/web/base_client.py", line 156, in api_call
return self._sync_send(api_url=api_url, req_args=req_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/slack_sdk/web/base_client.py", line 187, in _sync_send
return self._urllib_api_call(
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/slack_sdk/web/base_client.py", line 317, in _urllib_api_call
).validate()
^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/slack_sdk/web/slack_response.py", line 199, in validate
raise e.SlackApiError(message=msg, response=self)
slack_sdk.errors.SlackApiError: The request to the Slack API failed. (url: https://www.slack.com/api/chat.postMessage)
The server responded with: {'ok': False, 'error': 'message_limit_exceeded'}
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/celery/app/autoretry.py", line 38, in run
return task._orig_run(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/etc/app/apps/slack/tasks.py", line 314, in post_slack_rate_limit_message
post_message_to_channel(integration.organization, slack_channel, text)
File "/etc/app/apps/slack/utils.py", line 75, in post_message_to_channel
slack_client.chat_postMessage(channel=channel_id, text=text)
File "/usr/local/lib/python3.12/site-packages/slack_sdk/web/client.py", line 2220, in chat_postMessage
return self.api_call("chat.postMessage", json=kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/etc/app/apps/slack/client.py", line 153, in api_call
raise error_class(e.response) from e
apps.slack.errors.SlackAPIRatelimitError: Slack API error! Response: {'ok': False, 'error': 'message_limit_exceeded'}�[0m
2024-10-23 12:47:54,152 source=engine:celery worker=MainProcess task_id=??? task_name=??? name=celery.worker.strategy level=INFO Task apps.slack.tasks.post_slack_rate_limit_message[b3e6c7d2-259f-4ed1-9dc7-c118390b2ddf] received
2024-10-23 12:47:54,153 source=engine:celery worker=ForkPoolWorker-19 task_id=b3e6c7d2-259f-4ed1-9dc7-c118390b2ddf task_name=apps.slack.tasks.post_slack_rate_limit_message name=celery.app.trace level=INFO Task apps.slack.tasks.post_slack_rate_limit_message[b3e6c7d2-259f-4ed1-9dc7-c118390b2ddf] retry: Retry in 11s: SlackAPIRatelimitError("Slack API error! Response: {'ok': False, 'error': 'message_limit_exceeded'}")
2024-10-23 12:47:54,154 source=engine:celery worker=ForkPoolWorker-19 task_id=b3e6c7d2-259f-4ed1-9dc7-c118390b2ddf task_name=apps.slack.tasks.post_slack_rate_limit_message name=engine.celery level=INFO task finished args=[10]
`
What did you expect to happen:
The code should be handling SlackAPIRateLimitError in File "/etc/app/apps/slack/utils.py", line 75 and backing off so that eventually Slack messages can resume again,.
How do we reproduce it?
Grafana OnCall Version
v1.11.3
Product Area
Chatops
Grafana OnCall Platform?
Kubernetes
User's Browser?
No response
Anything else to add?
No response
The text was updated successfully, but these errors were encountered: