Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug]: Error spam due to prometheus metrics trying to be updated for non-premium user #7817

Open
mikstew opened this issue Jan 16, 2025 · 3 comments
Labels
bug Something isn't working

Comments

@mikstew
Copy link
Contributor

mikstew commented Jan 16, 2025

What happened?

We are using a non-premium installation of LiteLLM proxy, which means the prometheus metrics are not created based on this code.
It looks like the issue is that this function call is trying to update metrics that don't exist, resulting in a bunch of error spam.

Relevant log output

{"message": "LiteLLM.LoggingError: [Non-Blocking] Exception occurred while success logging Traceback (most recent call last):\n  File \"/usr/local/lib/python3.13/site-packages/litellm/litellm_core_utils/litellm_logging.py\", line 1618, in async_success_handler\n    await callback.async_log_success_event(\n    ...<4 lines>...\n    )\n  File \"/usr/local/lib/python3.13/site-packages/litellm/integrations/prometheus.py\", line 396, in async_log_success_event\n    self._increment_top_level_request_and_spend_metrics(\n    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^\n        end_user_id=end_user_id,\n        ^^^^^^^^^^^^^^^^^^^^^^^^\n    ...<7 lines>...\n        enum_values=enum_values,\n        ^^^^^^^^^^^^^^^^^^^^^^^^\n    )\n    ^\n  File \"/usr/local/lib/python3.13/site-packages/litellm/integrations/prometheus.py\", line 575, in _increment_top_level_request_and_spend_metrics\n    self.litellm_requests_metric.labels(**_labels).inc()\n    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nAttributeError: 'PrometheusLogger' object has no attribute 'litellm_requests_metric'\n", "level": "ERROR", "timestamp": "2025-01-13T17:41:55.490887"}
{"message": "prometheus Layer Error(): Exception occured - 'PrometheusLogger' object has no attribute 'litellm_proxy_total_requests_metric'", "level": "ERROR", "timestamp": "2025-01-13T17:41:55.479797", "stacktrace": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.13/site-packages/litellm/integrations/prometheus.py\", line 812, in async_post_call_success_hook\n    self.litellm_proxy_total_requests_metric.labels(**_labels).inc()\n    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\nAttributeError: 'PrometheusLogger' object has no attribute 'litellm_proxy_total_requests_metric'"}

Are you a ML Ops Team?

No

What LiteLLM version are you on ?

v1.58.0

Twitter / LinkedIn details

No response

@mikstew mikstew added the bug Something isn't working label Jan 16, 2025
@krrishdholakia
Copy link
Contributor

@mikstew curious why prometheus is being added as a callback.

What does your config look like?

@mikstew
Copy link
Contributor Author

mikstew commented Jan 17, 2025

Yeah, I can't figure out why it's doing that. Here's our config:

litellm_settings: {
    cache: false,
    json_logs: true,
    success_callback: [
        'langfuse'
     ]
}

@krrishdholakia
Copy link
Contributor

able to repro

Image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants