Slack exceeding rate limit as SlackAPIRateLimitError is not caught

[AmiSMB]

What went wrong?

What happened:
I stopped receiving messages in Slack workspace (Free). I the celery logs I get the following logs:-

`�[1;31m2024-10-23 12:47:46,985 source=engine:celery worker=ForkPoolWorker-19 task_id=b3e6c7d2-259f-4ed1-9dc7-c118390b2ddf task_name=apps.slack.tasks.post_slack_rate_limit_message name=apps.slack.client level=ERROR Slack API call error! slack_team_identity=1 args=('chat.postMessage',) kwargs={'json': {'channel': 'XXXXXXXXXX', 'text': 'Delivering and updating alert groups of integration xxxxxx in Slack is temporarily stopped due to rate limit. You could find new alert groups at <http://oncall-grafana:80/a/grafana-oncall-app/alert-groups?integration=XXXXXXXXXXX&status=0|web page "Alert Groups">'}} status=200 error=message_limit_exceeded response={'ok': False, 'error': 'message_limit_exceeded'}�[0m
�[1;33m2024-10-23 12:47:46,985 source=engine:celery worker=ForkPoolWorker-19 task_id=b3e6c7d2-259f-4ed1-9dc7-c118390b2ddf task_name=apps.slack.tasks.post_slack_rate_limit_message name=common.custom_celery_tasks.dedicated_queue_retry_task level=WARNING Retrying celery task
Traceback (most recent call last):
File "/etc/app/apps/slack/client.py", line 128, in api_call
response = super().api_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/slack_sdk/web/base_client.py", line 156, in api_call
return self._sync_send(api_url=api_url, req_args=req_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/slack_sdk/web/base_client.py", line 187, in _sync_send
return self._urllib_api_call(
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/slack_sdk/web/base_client.py", line 317, in _urllib_api_call
).validate()
^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/slack_sdk/web/slack_response.py", line 199, in validate
raise e.SlackApiError(message=msg, response=self)
slack_sdk.errors.SlackApiError: The request to the Slack API failed. (url: https://www.slack.com/api/chat.postMessage)
The server responded with: {'ok': False, 'error': 'message_limit_exceeded'}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/celery/app/autoretry.py", line 38, in run
return task._orig_run(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/etc/app/apps/slack/tasks.py", line 314, in post_slack_rate_limit_message
post_message_to_channel(integration.organization, slack_channel, text)
File "/etc/app/apps/slack/utils.py", line 75, in post_message_to_channel
slack_client.chat_postMessage(channel=channel_id, text=text)
File "/usr/local/lib/python3.12/site-packages/slack_sdk/web/client.py", line 2220, in chat_postMessage
return self.api_call("chat.postMessage", json=kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/etc/app/apps/slack/client.py", line 153, in api_call
raise error_class(e.response) from e
apps.slack.errors.SlackAPIRatelimitError: Slack API error! Response: {'ok': False, 'error': 'message_limit_exceeded'}�[0m
2024-10-23 12:47:46,988 source=engine:celery worker=MainProcess task_id=??? task_name=??? name=celery.worker.strategy level=INFO Task apps.slack.tasks.post_slack_rate_limit_message[b3e6c7d2-259f-4ed1-9dc7-c118390b2ddf] received
2024-10-23 12:47:46,988 source=engine:celery worker=ForkPoolWorker-19 task_id=b3e6c7d2-259f-4ed1-9dc7-c118390b2ddf task_name=apps.slack.tasks.post_slack_rate_limit_message name=celery.app.trace level=INFO Task apps.slack.tasks.post_slack_rate_limit_message[b3e6c7d2-259f-4ed1-9dc7-c118390b2ddf] retry: Retry in 7s: SlackAPIRatelimitError("Slack API error! Response: {'ok': False, 'error': 'message_limit_exceeded'}")
2024-10-23 12:47:46,988 source=engine:celery worker=ForkPoolWorker-19 task_id=b3e6c7d2-259f-4ed1-9dc7-c118390b2ddf task_name=apps.slack.tasks.post_slack_rate_limit_message name=engine.celery level=INFO task finished args=[10]
2024-10-23 12:47:53,987 source=engine:celery worker=ForkPoolWorker-19 task_id=b3e6c7d2-259f-4ed1-9dc7-c118390b2ddf task_name=apps.slack.tasks.post_slack_rate_limit_message name=engine.celery level=INFO task started args=[10]
�[1;31m2024-10-23 12:47:54,149 source=engine:celery worker=ForkPoolWorker-19 task_id=b3e6c7d2-259f-4ed1-9dc7-c118390b2ddf task_name=apps.slack.tasks.post_slack_rate_limit_message name=apps.slack.client level=ERROR Slack API call error! slack_team_identity=1 args=('chat.postMessage',) kwargs={'json': {'channel': 'XXXXXXXXX', 'text': 'Delivering and updating alert groups of integration XXXXXXXX in Slack is temporarily stopped due to rate limit. You could find new alert groups at <http://oncall-grafana:80/a/grafana-oncall-app/alert-groups?integration=XXXXXXXXX&status=0|web page "Alert Groups">'}} status=200 error=message_limit_exceeded response={'ok': False, 'error': 'message_limit_exceeded'}�[0m
�[1;33m2024-10-23 12:47:54,149 source=engine:celery worker=ForkPoolWorker-19 task_id=b3e6c7d2-259f-4ed1-9dc7-c118390b2ddf task_name=apps.slack.tasks.post_slack_rate_limit_message name=common.custom_celery_tasks.dedicated_queue_retry_task level=WARNING Retrying celery task
Traceback (most recent call last):
File "/etc/app/apps/slack/client.py", line 128, in api_call
response = super().api_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/slack_sdk/web/base_client.py", line 156, in api_call
return self._sync_send(api_url=api_url, req_args=req_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/slack_sdk/web/base_client.py", line 187, in _sync_send
return self._urllib_api_call(
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/slack_sdk/web/base_client.py", line 317, in _urllib_api_call
).validate()
^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/slack_sdk/web/slack_response.py", line 199, in validate
raise e.SlackApiError(message=msg, response=self)
slack_sdk.errors.SlackApiError: The request to the Slack API failed. (url: https://www.slack.com/api/chat.postMessage)
The server responded with: {'ok': False, 'error': 'message_limit_exceeded'}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/celery/app/autoretry.py", line 38, in run
return task._orig_run(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/etc/app/apps/slack/tasks.py", line 314, in post_slack_rate_limit_message
post_message_to_channel(integration.organization, slack_channel, text)
File "/etc/app/apps/slack/utils.py", line 75, in post_message_to_channel
slack_client.chat_postMessage(channel=channel_id, text=text)
File "/usr/local/lib/python3.12/site-packages/slack_sdk/web/client.py", line 2220, in chat_postMessage
return self.api_call("chat.postMessage", json=kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/etc/app/apps/slack/client.py", line 153, in api_call
raise error_class(e.response) from e
apps.slack.errors.SlackAPIRatelimitError: Slack API error! Response: {'ok': False, 'error': 'message_limit_exceeded'}�[0m
2024-10-23 12:47:54,152 source=engine:celery worker=MainProcess task_id=??? task_name=??? name=celery.worker.strategy level=INFO Task apps.slack.tasks.post_slack_rate_limit_message[b3e6c7d2-259f-4ed1-9dc7-c118390b2ddf] received
2024-10-23 12:47:54,153 source=engine:celery worker=ForkPoolWorker-19 task_id=b3e6c7d2-259f-4ed1-9dc7-c118390b2ddf task_name=apps.slack.tasks.post_slack_rate_limit_message name=celery.app.trace level=INFO Task apps.slack.tasks.post_slack_rate_limit_message[b3e6c7d2-259f-4ed1-9dc7-c118390b2ddf] retry: Retry in 11s: SlackAPIRatelimitError("Slack API error! Response: {'ok': False, 'error': 'message_limit_exceeded'}")
2024-10-23 12:47:54,154 source=engine:celery worker=ForkPoolWorker-19 task_id=b3e6c7d2-259f-4ed1-9dc7-c118390b2ddf task_name=apps.slack.tasks.post_slack_rate_limit_message name=engine.celery level=INFO task finished args=[10]
`

What did you expect to happen:
The code should be handling SlackAPIRateLimitError in File "/etc/app/apps/slack/utils.py", line 75 and backing off so that eventually Slack messages can resume again,.

How do we reproduce it?

1. Allow many alerts to be posted to Slack to go over the API limit
2. Watch the logs and see errors but nothing in Slack

Grafana OnCall Version

v1.11.3

Product Area

Chatops

Grafana OnCall Platform?

Kubernetes

User's Browser?

No response

Anything else to add?

No response

[AmiSMB]

Upgraded to latest v1.13.11 and we are still getting this issue:-

�[2024-12-20 14:18:55,842 source=engine:celery worker=ForkPoolWorker-2 task_id=0dfed72e-00f1-4d68-8915-8bf1e15142e9 task_name=apps.slack.tasks.post_slack_rate_limit_message name=apps.slack.client level=ERROR Slack API call error! slack_team_identity=1 args=('chat.postMessage',) kwargs={'json': {'channel': 'C0765EM3Y3E', 'text': 'Delivering Alert Groups in Slack, for integration XXXXX is temporarily rate-limited (due to a Slack rate-limit). Meanwhile, you can still find new Alert Groups in the <http://oncall-grafana:80/a/grafana-oncall-app/alert-groups?integration=CKV6YCHXWVRHK&status=0|"Alert Groups" web page>'}} status=200 error=message_limit_exceeded response={'ok': False, 'error': 'message_limit_exceeded'}�[
�[2024-12-20 14:18:55,842 source=engine:celery worker=ForkPoolWorker-2 task_id=0dfed72e-00f1-4d68-8915-8bf1e15142e9 task_name=apps.slack.tasks.post_slack_rate_limit_message name=common.custom_celery_tasks.dedicated_queue_retry_task level=WARNING Retrying celery task
Traceback (most recent call last):
File "/etc/app/apps/slack/client.py", line 128, in api_call
response = super().api_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/slack_sdk/web/base_client.py", line 156, in api_call
return self._sync_send(api_url=api_url, req_args=req_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/slack_sdk/web/base_client.py", line 187, in _sync_send
return self._urllib_api_call(
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/slack_sdk/web/base_client.py", line 317, in _urllib_api_call
).validate()
^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/slack_sdk/web/slack_response.py", line 199, in validate
raise e.SlackApiError(message=msg, response=self)
slack_sdk.errors.SlackApiError: The request to the Slack API failed. (url: https://www.slack.com/api/chat.postMessage)
The server responded with: {'ok': False, 'error': 'message_limit_exceeded'}

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/celery/app/autoretry.py", line 38, in run
return task._orig_run(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/etc/app/apps/slack/tasks.py", line 378, in post_slack_rate_limit_message
post_message_to_channel(integration.organization, slack_channel.slack_id, text)
File "/etc/app/apps/slack/utils.py", line 75, in post_message_to_channel
slack_client.chat_postMessage(channel=channel_id, text=text)
File "/usr/local/lib/python3.12/site-packages/slack_sdk/web/client.py", line 2220, in chat_postMessage
return self.api_call("chat.postMessage", json=kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/etc/app/apps/slack/client.py", line 153, in api_call
raise error_class(e.response) from e
apps.slack.errors.SlackAPIRatelimitError: Slack API error! Response: {'ok': False, 'error': 'message_limit_exceeded'}
2024-12-20 14:18:55,848 source=engine:celery worker=ForkPoolWorker-2 task_id=0dfed72e-00f1-4d68-8915-8bf1e15142e9 task_name=apps.slack.tasks.post_slack_rate_limit_message name=celery.app.trace level=INFO Task apps.slack.tasks.post_slack_rate_limit_message[0dfed72e-00f1-4d68-8915-8bf1e15142e9] retry: Retry in 130s: SlackAPIRatelimitError("Slack API error! Response: {'ok': False, 'error': 'message_limit_exceeded'}")