ai-workspace-services/litellm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(proxy/hooks): populate llm_provider on internal rate-limit errors (#27707)

Browse Source 
* feat(proxy/hooks): add ProxyHTTPRateLimitError + provider resolver

Introduces a small helper layer used by every proxy-side rate-limit
hook so that the 429 they raise carries a populated llm_provider /
model — instead of an empty exception.llm_provider that downstream
loggers (Prometheus failure metric, observability callbacks) read as
'no provider attribution'.

ProxyHTTPRateLimitError inherits from both fastapi.HTTPException
(so the proxy server still renders it as a 429) and
litellm.exceptions.RateLimitError (so isinstance checks and
PrometheusLogger._get_exception_class_name pick up llm_provider).
We deliberately don't call RateLimitError.__init__ — it constructs
an httpx.Response we don't need and would just add failure surface;
attribute parity is what downstream consumers care about.

resolve_llm_provider_for_rate_limit() wraps litellm.get_llm_provider
defensively. Internal limiter hooks fire from async_pre_call_hook —
well before get_llm_provider runs anywhere else in the request
lifecycle — so we have to call it ourselves at raise time. If the
model is missing or unparseable (alias, router-only model) we fall
back to llm_provider='litellm_proxy' rather than letting a second
exception leak out and break the request path.

Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>

* fix(proxy/hooks): populate llm_provider on parallel-request 429s

Both v1 and v3 parallel-request limiters fired bare HTTPException(429)
from inside async_pre_call_hook. The downstream Prometheus failure
metric reads exception.llm_provider via _get_exception_class_name —
the empty value showed up as exception_class='HTTPException' and
left model_id='None' on the time series.

Threads requested_model through every raise site in:

* parallel_request_limiter.py:
  - check_key_in_limits (the per-key/per-model/per-user/per-team/
    per-customer over-limit path)
  - raise_rate_limit_error (zero-limit + global_max_parallel_requests
    paths) — now takes an optional requested_model kwarg
* parallel_request_limiter_v3.py:
  - _handle_rate_limit_error (the OVER_LIMIT translator), called
    from both the should_rate_limit pre-check and the TPM
    reservation path

Resolved via resolve_llm_provider_for_rate_limit so unknown / missing
models silently fall back to llm_provider='litellm_proxy' instead of
breaking the request path with a second exception.

Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>

* fix(proxy/hooks): populate llm_provider on dynamic-rate-limit 429s

Same plumbing change as the parallel limiters, applied to both
dynamic_rate_limiter (v1) and dynamic_rate_limiter_v3:

* v1: TPM-zero and RPM-zero paths in async_pre_call_hook now resolve
  data['model'] -> (model, llm_provider) once and pass it into both
  raises.
* v3: All three raise sites in _check_rate_limits — the
  model_saturation_check enforced raise, the priority_model
  enforced raise, and the fail-closed unknown-descriptor branch —
  now attribute the 429 to the actual provider.

Falls back to llm_provider='litellm_proxy' when the model can't be
resolved.

Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>

* fix(proxy/hooks): populate llm_provider on batch-rate-limit 429s

batch_rate_limiter._raise_rate_limit_error now takes a
requested_model kwarg threaded from data['model'] in
_check_and_increment_batch_counters. The batch-creation 429 is what
gets raised when the input file's tokens/requests count would push
the per-key TPM/RPM window over its limit.

Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>

* fix(proxy/hooks): populate llm_provider on budget/iterations 429s

Final batch of internal raise sites — the user/session-budget and
max-iterations hooks. Same pattern: resolve data['model'] once at
raise time, attach to ProxyHTTPRateLimitError so Prometheus and
observability callbacks can attribute the 429.

Hooks updated:
* max_budget_limiter (per-user max_budget exceeded)
* max_iterations_limiter (per-session agent iteration cap)
* max_budget_per_session_limiter (per-session dollar cap)

All three fall back to llm_provider='litellm_proxy' when data['model']
is missing or unparseable. Drops the now-unused HTTPException import
from each module.

Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>

* test(proxy/hooks): pin provider field on internal rate-limit 429s

Regression coverage for the 'provider field missing' bug across every
proxy-side rate-limit hook + the helper layer:

* ProxyHTTPRateLimitError class shape (HTTPException + RateLimitError,
  dict-detail stringification, None-provider normalization).
* resolve_llm_provider_for_rate_limit happy paths
  (gpt-4o-mini, anthropic/..., bedrock/...) plus all three fallback
  branches (None, '', unknown name) plus a 'get_llm_provider raises'
  case that asserts we swallow the secondary exception.
* For each limiter (parallel v1/v3, dynamic v1/v3, batch,
  max_budget, max_iterations, max_budget_per_session): assert the
  raised exception is a RateLimitError carrying the resolved
  model + llm_provider, and a sibling test that asserts the
  fallback path returns 'litellm_proxy' without leaking a second
  exception.
* Two PrometheusLogger._get_exception_class_name pins so the
  Prometheus failure metric label flips from 'HTTPException' to
  'Openai.ProxyHTTPRateLimitError' (or 'Litellm_proxy.*' on
  fallback) — that's what dashboards consume.

Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>

* perf(proxy/hooks): defer provider resolution to over-limit branches

* fix: use error_message in raise_rate_limit_error to avoid literal 'None' in detail

* Consolidate rate_limiter_utils imports in dynamic_rate_limiter

* fix(proxy): set num_retries/max_retries on ProxyHTTPRateLimitError

ProxyHTTPRateLimitError inherits from RateLimitError but did not call
RateLimitError.__init__, so num_retries/max_retries were never set.
When Starlette's HTTPException lacks __str__, MRO falls through to
RateLimitError.__str__, which unconditionally reads these attributes
and raises AttributeError during logging/traceback formatting.
Initialize them to None defensively.

* fix(mypy): silence base-class status_code conflict on ProxyHTTPRateLimitError

HTTPException declares 'status_code: int' while openai.RateLimitError
(via APIStatusError) declares 'status_code: Literal[429] = 429'. Mypy
flags the multi-base override as [misc] in CI lint. The runtime semantics
are fine (we set self.status_code in __init__), so silence the
class-level annotation conflict with a targeted ignore.

Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Mateo Wang <mateo-berri@users.noreply.github.com>
This commit is contained in:
Mateo Wang 2026-06-04 22:46:08 -07:00 committed by GitHub
parent 812a2217ca
commit df704d9016
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
10 changed files with 1186 additions and 29 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

14
litellm/proxy/hooks/batch_rate_limiter.py
View File

@ -32,6 +32,10 @@ from litellm.batches.batch_utils import (
)
from litellm.integrations.custom_logger import CustomLogger
from litellm.proxy._types import SpecialModelNames, UserAPIKeyAuth
from litellm.proxy.hooks.rate_limiter_utils import (
ProxyHTTPRateLimitError,
resolve_llm_provider_for_rate_limit,
)
if TYPE_CHECKING:
from opentelemetry.trace import Span as _Span
@ -375,6 +379,7 @@ class _PROXY_BatchRateLimiter(CustomLogger):
descriptors: List["RateLimitDescriptor"],
batch_usage: BatchFileUsage,
limit_type: str,
requested_model: Optional[str] = None,
) -> None:
"""Raise HTTPException for rate limit exceeded."""
from datetime import datetime
@ -419,7 +424,10 @@ class _PROXY_BatchRateLimiter(CustomLogger):
f"Limit resets at: {reset_time_formatted}"
)
raise HTTPException(
resolved_model, llm_provider = resolve_llm_provider_for_rate_limit(
requested_model
)
raise ProxyHTTPRateLimitError(
status_code=429,
detail=detail,
headers={
@ -427,6 +435,8 @@ class _PROXY_BatchRateLimiter(CustomLogger):
"rate_limit_type": limit_type,
"reset_at": reset_time_formatted,
},
model=resolved_model,
llm_provider=llm_provider,
)
async def _check_and_increment_batch_counters(
@ -470,6 +480,7 @@ class _PROXY_BatchRateLimiter(CustomLogger):
)
if rate_limit_response["overall_code"] == "OVER_LIMIT":
requested_model = data.get("model") if data else None
for status in rate_limit_response["statuses"]:
if status["code"] == "OVER_LIMIT":
self._raise_rate_limit_error(
@ -477,6 +488,7 @@ class _PROXY_BatchRateLimiter(CustomLogger):
descriptors,
batch_usage,
status["rate_limit_type"],
requested_model=requested_model,
)
async def count_input_file_usage(

23
litellm/proxy/hooks/dynamic_rate_limiter.py
View File

@ -6,20 +6,21 @@ import asyncio
import os
from typing import List, Optional, Tuple, Union
from fastapi import HTTPException
import litellm
from litellm import ModelResponse, Router
from litellm._logging import verbose_proxy_logger
from litellm.caching.caching import DualCache
from litellm.integrations.custom_logger import CustomLogger
from litellm.proxy._types import UserAPIKeyAuth
from litellm.proxy.hooks.rate_limiter_utils import (
ProxyHTTPRateLimitError,
convert_priority_to_percent,
resolve_llm_provider_for_rate_limit,
)
from litellm.types.router import ModelGroupInfo
from litellm.types.utils import CallTypesLiteral
from litellm.utils import get_utc_datetime
from .rate_limiter_utils import convert_priority_to_percent
class DynamicRateLimiterCache:
"""
@ -218,7 +219,10 @@ class _PROXY_DynamicRateLimitHandler(CustomLogger):
)
### CHECK TPM ###
if available_tpm is not None and available_tpm == 0:
raise HTTPException(
resolved_model, llm_provider = resolve_llm_provider_for_rate_limit(
data.get("model")
)
raise ProxyHTTPRateLimitError(
status_code=429,
detail={
"error": "Key={} over available TPM={}. Model TPM={}, Active keys={}".format(
@ -228,10 +232,15 @@ class _PROXY_DynamicRateLimitHandler(CustomLogger):
active_projects,
)
},
model=resolved_model,
llm_provider=llm_provider,
)
### CHECK RPM ###
elif available_rpm is not None and available_rpm == 0:
raise HTTPException(
resolved_model, llm_provider = resolve_llm_provider_for_rate_limit(
data.get("model")
)
raise ProxyHTTPRateLimitError(
status_code=429,
detail={
"error": "Key={} over available RPM={}. Model RPM={}, Active keys={}".format(
@ -241,6 +250,8 @@ class _PROXY_DynamicRateLimitHandler(CustomLogger):
active_projects,
)
},
model=resolved_model,
llm_provider=llm_provider,
)
elif available_rpm is not None or available_tpm is not None:
## UPDATE CACHE WITH ACTIVE PROJECT

19
litellm/proxy/hooks/dynamic_rate_limiter_v3.py
View File

@ -19,7 +19,11 @@ from litellm.proxy.hooks.parallel_request_limiter_v3 import (
RateLimitDescriptorRateLimitObject,
_PROXY_MaxParallelRequestsHandler_v3,
)
from litellm.proxy.hooks.rate_limiter_utils import convert_priority_to_percent
from litellm.proxy.hooks.rate_limiter_utils import (
ProxyHTTPRateLimitError,
convert_priority_to_percent,
resolve_llm_provider_for_rate_limit,
)
from litellm.proxy.utils import InternalUsageCache
from litellm.types.router import ModelGroupInfo
from litellm.types.utils import CallTypesLiteral
@ -487,12 +491,13 @@ class _PROXY_DynamicRateLimitHandlerV3(CustomLogger):
)
if atomic_response["overall_code"] == "OVER_LIMIT":
resolved_model, llm_provider = resolve_llm_provider_for_rate_limit(model)
for status in atomic_response["statuses"]:
if status["code"] != "OVER_LIMIT":
continue
descriptor_key = status["descriptor_key"]
if descriptor_key == "model_saturation_check":
raise HTTPException(
raise ProxyHTTPRateLimitError(
status_code=429,
detail={
"error": f"Model capacity reached for {model}. "
@ -507,13 +512,15 @@ class _PROXY_DynamicRateLimitHandlerV3(CustomLogger):
"rate_limit_type": str(status["rate_limit_type"]),
"x-litellm-priority": priority or "default",
},
model=resolved_model,
llm_provider=llm_provider,
)
if descriptor_key == "priority_model":
verbose_proxy_logger.debug(
f"Enforcing priority limits for {model}, saturation: {saturation:.1%}, "
f"priority: {priority}"
)
raise HTTPException(
raise ProxyHTTPRateLimitError(
status_code=429,
detail={
"error": f"Priority-based rate limit exceeded. "
@ -531,6 +538,8 @@ class _PROXY_DynamicRateLimitHandlerV3(CustomLogger):
"x-litellm-priority": priority or "default",
"x-litellm-saturation": f"{saturation:.2%}",
},
model=resolved_model,
llm_provider=llm_provider,
)
# Fail-closed guard: overall_code says OVER_LIMIT but no status
@ -547,7 +556,7 @@ class _PROXY_DynamicRateLimitHandlerV3(CustomLogger):
f"Dynamic rate limiter: OVER_LIMIT response with unknown "
f"descriptor_key(s) — refusing request. response={atomic_response}"
)
raise HTTPException(
raise ProxyHTTPRateLimitError(
status_code=429,
detail={
"error": "Rate limit exceeded",
@ -562,6 +571,8 @@ class _PROXY_DynamicRateLimitHandlerV3(CustomLogger):
"retry-after": str(self.v3_limiter.window_size),
"x-litellm-priority": priority or "default",
},
model=resolved_model,
llm_provider=llm_provider,
)
# If priority is NOT enforced (saturation below threshold) but

14
litellm/proxy/hooks/max_budget_limiter.py
View File

@ -5,6 +5,10 @@ from litellm._logging import verbose_proxy_logger
from litellm.caching.caching import DualCache
from litellm.integrations.custom_logger import CustomLogger
from litellm.proxy._types import UserAPIKeyAuth
from litellm.proxy.hooks.rate_limiter_utils import (
ProxyHTTPRateLimitError,
resolve_llm_provider_for_rate_limit,
)
class _PROXY_MaxBudgetLimiter(CustomLogger):
@ -63,7 +67,15 @@ class _PROXY_MaxBudgetLimiter(CustomLogger):
# CHECK IF REQUEST ALLOWED
if curr_spend >= max_budget:
raise HTTPException(status_code=429, detail="Max budget limit reached.")
resolved_model, llm_provider = resolve_llm_provider_for_rate_limit(
data.get("model") if data else None
)
raise ProxyHTTPRateLimitError(
status_code=429,
detail="Max budget limit reached.",
model=resolved_model,
llm_provider=llm_provider,
)
except HTTPException as e:
raise e
except Exception as e:

13
litellm/proxy/hooks/max_budget_per_session_limiter.py
View File

@ -17,12 +17,14 @@ Follows the same pattern as max_iterations_limiter.py.
import os
from typing import TYPE_CHECKING, Any, Optional, Union
from fastapi import HTTPException
from litellm import DualCache
from litellm._logging import verbose_proxy_logger
from litellm.integrations.custom_logger import CustomLogger
from litellm.proxy._types import UserAPIKeyAuth
from litellm.proxy.hooks.rate_limiter_utils import (
ProxyHTTPRateLimitError,
resolve_llm_provider_for_rate_limit,
)
if TYPE_CHECKING:
from litellm.proxy.utils import InternalUsageCache as _InternalUsageCache
@ -112,13 +114,18 @@ class _PROXY_MaxBudgetPerSessionHandler(CustomLogger):
)
if current_spend >= max_budget:
raise HTTPException(
resolved_model, llm_provider = resolve_llm_provider_for_rate_limit(
data.get("model") if data else None
)
raise ProxyHTTPRateLimitError(
status_code=429,
detail=(
f"Session budget exceeded for session {session_id}. "
f"Current spend: ${current_spend:.4f}, "
f"max_budget_per_session: ${max_budget:.2f}."
),
model=resolved_model,
llm_provider=llm_provider,
)
return None

13
litellm/proxy/hooks/max_iterations_limiter.py
View File

@ -13,12 +13,14 @@ Follows the same pattern as parallel_request_limiter_v3.py.
import os
from typing import TYPE_CHECKING, Any, Optional, Union
from fastapi import HTTPException
from litellm import DualCache
from litellm._logging import verbose_proxy_logger
from litellm.integrations.custom_logger import CustomLogger
from litellm.proxy._types import UserAPIKeyAuth
from litellm.proxy.hooks.rate_limiter_utils import (
ProxyHTTPRateLimitError,
resolve_llm_provider_for_rate_limit,
)
if TYPE_CHECKING:
from litellm.proxy.utils import InternalUsageCache as _InternalUsageCache
@ -116,12 +118,17 @@ class _PROXY_MaxIterationsHandler(CustomLogger):
current_count = await self._increment_and_get(cache_key)
if current_count > max_iterations:
raise HTTPException(
resolved_model, llm_provider = resolve_llm_provider_for_rate_limit(
data.get("model") if data else None
)
raise ProxyHTTPRateLimitError(
status_code=429,
detail=(
f"Max iterations exceeded for session {session_id}. "
f"Current count: {current_count}, max_iterations: {max_iterations}."
),
model=resolved_model,
llm_provider=llm_provider,
)
verbose_proxy_logger.debug(

39
litellm/proxy/hooks/parallel_request_limiter.py
View File

@ -17,6 +17,10 @@ from litellm.proxy.auth.auth_utils import (
get_key_model_rpm_limit,
get_key_model_tpm_limit,
)
from litellm.proxy.hooks.rate_limiter_utils import (
ProxyHTTPRateLimitError,
resolve_llm_provider_for_rate_limit,
)
if TYPE_CHECKING:
from opentelemetry.trace import Span as _Span
@ -73,7 +77,8 @@ class _PROXY_MaxParallelRequestsHandler(CustomLogger):
if max_parallel_requests == 0 or tpm_limit == 0 or rpm_limit == 0:
# base case
raise self.raise_rate_limit_error(
additional_details=f"{CommonProxyErrors.max_parallel_request_limit_reached.value}. Hit limit for {rate_limit_type}. Current limits: max_parallel_requests: {max_parallel_requests}, tpm_limit: {tpm_limit}, rpm_limit: {rpm_limit}"
additional_details=f"{CommonProxyErrors.max_parallel_request_limit_reached.value}. Hit limit for {rate_limit_type}. Current limits: max_parallel_requests: {max_parallel_requests}, tpm_limit: {tpm_limit}, rpm_limit: {rpm_limit}",
requested_model=data.get("model") if data else None,
)
new_val = {
"current_requests": 1,
@ -95,10 +100,16 @@ class _PROXY_MaxParallelRequestsHandler(CustomLogger):
values_to_update_in_cache.append((request_count_api_key, new_val))
else:
raise HTTPException(
requested_model = data.get("model") if data else None
resolved_model, llm_provider = resolve_llm_provider_for_rate_limit(
requested_model
)
raise ProxyHTTPRateLimitError(
status_code=429,
detail=f"LiteLLM Rate Limit Handler for rate limit type = {rate_limit_type}. {CommonProxyErrors.max_parallel_request_limit_reached.value}. current rpm: {current['current_rpm']}, rpm limit: {rpm_limit}, current tpm: {current['current_tpm']}, tpm limit: {tpm_limit}, current max_parallel_requests: {current['current_requests']}, max_parallel_requests: {max_parallel_requests}",
headers={"retry-after": str(self.time_to_next_minute())},
model=resolved_model,
llm_provider=llm_provider,
)
await self.internal_usage_cache.async_batch_set_cache(
@ -122,18 +133,31 @@ class _PROXY_MaxParallelRequestsHandler(CustomLogger):
return seconds_to_next_minute
def raise_rate_limit_error(
self, additional_details: Optional[str] = None
self,
additional_details: Optional[str] = None,
requested_model: Optional[str] = None,
) -> HTTPException:
"""
Raise an HTTPException with a 429 status code and a retry-after header
Raise an HTTPException with a 429 status code and a retry-after header.
``requested_model`` is resolved via :func:`get_llm_provider` so the
raised exception carries ``llm_provider`` for downstream loggers
(Prometheus failure metric, observability callbacks). Falls back to
``llm_provider="litellm_proxy"`` when the model is missing or
unparseable — see ``resolve_llm_provider_for_rate_limit``.
"""
error_message = "Max parallel request limit reached"
if additional_details is not None:
error_message = error_message + " " + additional_details
raise HTTPException(
resolved_model, llm_provider = resolve_llm_provider_for_rate_limit(
requested_model
)
raise ProxyHTTPRateLimitError(
status_code=429,
detail=f"Max parallel request limit reached {additional_details}",
detail=error_message,
headers={"retry-after": str(self.time_to_next_minute())},
model=resolved_model,
llm_provider=llm_provider,
)
async def get_all_cache_objects(
@ -225,7 +249,8 @@ class _PROXY_MaxParallelRequestsHandler(CustomLogger):
# if above -> raise error
if current_global_requests >= global_max_parallel_requests:
return self.raise_rate_limit_error(
additional_details=f"Hit Global Limit: Limit={global_max_parallel_requests}, current: {current_global_requests}"
additional_details=f"Hit Global Limit: Limit={global_max_parallel_requests}, current: {current_global_requests}",
requested_model=data.get("model") if data else None,
)
# if below -> increment
else:

16
litellm/proxy/hooks/parallel_request_limiter_v3.py
View File

@ -23,8 +23,6 @@ from typing import (
cast,
)
from fastapi import HTTPException
from litellm import DualCache
from litellm._logging import verbose_proxy_logger
from litellm.constants import DYNAMIC_RATE_LIMIT_ERROR_THRESHOLD_PER_MINUTE
@ -34,6 +32,10 @@ from litellm.litellm_core_utils.prompt_templates.common_utils import (
)
from litellm.proxy._types import UserAPIKeyAuth
from litellm.proxy.auth.auth_utils import get_model_rate_limit_from_metadata
from litellm.proxy.hooks.rate_limiter_utils import (
ProxyHTTPRateLimitError,
resolve_llm_provider_for_rate_limit,
)
from litellm.types.caching import RedisPipelineIncrementOperation
from litellm.types.llms.openai import BaseLiteLLMOpenAIResponseObject
from litellm.types.utils import CallTypes, ModelResponse, Usage
@ -1967,6 +1969,7 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
self,
response: RateLimitResponse,
descriptors: List[RateLimitDescriptor],
requested_model: Optional[str] = None,
) -> None:
"""Handle rate limit exceeded error by raising HTTPException."""
for status in response["statuses"]:
@ -1999,7 +2002,10 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
f"Limit resets at: {reset_time_formatted}"
)
raise HTTPException(
resolved_model, llm_provider = resolve_llm_provider_for_rate_limit(
requested_model
)
raise ProxyHTTPRateLimitError(
status_code=429,
detail=detail,
headers={
@ -2007,6 +2013,8 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
"rate_limit_type": str(status["rate_limit_type"]),
"reset_at": reset_time_formatted,
},
model=resolved_model,
llm_provider=llm_provider,
)
async def async_pre_call_hook(
@ -2115,6 +2123,7 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
self._handle_rate_limit_error(
response=response,
descriptors=descriptors,
requested_model=requested_model,
)
else:
# add descriptors to request headers
@ -2188,6 +2197,7 @@ class _PROXY_MaxParallelRequestsHandler_v3(CustomLogger):
self._handle_rate_limit_error(
response=tpm_response,
descriptors=descriptors,
requested_model=requested_model,
)
else:
self._stash_value_in_metadata_channels(

96
litellm/proxy/hooks/rate_limiter_utils.py
View File

@ -2,11 +2,105 @@
Shared utility functions for rate limiter hooks.
"""
from typing import Optional, Union
from typing import Any, Optional, Tuple, Union
from fastapi import HTTPException
import litellm
from litellm._logging import verbose_proxy_logger
from litellm.exceptions import RateLimitError
from litellm.types.router import ModelGroupInfo
from litellm.types.utils import PriorityReservationDict
PROXY_LLM_PROVIDER_FALLBACK = "litellm_proxy"
def resolve_llm_provider_for_rate_limit(
model: Optional[str],
) -> Tuple[str, str]:
"""
Resolve ``(model, llm_provider)`` for a request being rejected by an
internal proxy-side rate-limit hook.
These hooks fire from ``async_pre_call_hook`` — well before
:func:`litellm.get_llm_provider` is invoked anywhere else in the request
lifecycle — so the raised 429 would otherwise have an empty
``llm_provider`` field, making the resulting Prometheus
``litellm_proxy_failed_requests_metric`` show up with
``exception_class="RateLimitError"`` and no provider attribution.
Wrapped defensively: if ``model`` is missing, malformed, or
``get_llm_provider`` raises (unknown alias, router-only model, etc.) we
fall back to ``("", "litellm_proxy")`` so we never break the request path
by piling a second exception on top of the rate-limit one we're trying to
raise.
"""
if not model:
return "", PROXY_LLM_PROVIDER_FALLBACK
try:
resolved_model, custom_llm_provider, _, _ = litellm.get_llm_provider(
model=model,
)
return (
resolved_model or model,
custom_llm_provider or PROXY_LLM_PROVIDER_FALLBACK,
)
except Exception as e:
verbose_proxy_logger.debug(
"rate_limiter_utils.resolve_llm_provider_for_rate_limit: "
"could not resolve provider for model=%s, falling back to %s. err=%s",
model,
PROXY_LLM_PROVIDER_FALLBACK,
str(e),
)
return model, PROXY_LLM_PROVIDER_FALLBACK
class ProxyHTTPRateLimitError(HTTPException, RateLimitError): # type: ignore[misc]
"""
HTTPException raised by proxy-side rate-limit hooks that *also* exposes
``model`` and ``llm_provider`` attributes.
Why both base classes:
- The proxy server's exception handler keys off ``HTTPException`` to render
a 429 response, so we must remain an ``HTTPException``.
- Downstream loggers (Prometheus ``async_post_call_failure_hook``,
structured logging, observability callbacks) read ``exception.llm_provider``
via :meth:`litellm.integrations.prometheus.PrometheusLogger._get_exception_class_name`
and ``isinstance(exc, RateLimitError)`` for category routing. Inheriting
from :class:`litellm.exceptions.RateLimitError` keeps that wiring intact.
We intentionally do not call ``RateLimitError.__init__`` (which constructs
an httpx.Response) — it isn't needed here and just adds failure surface.
Attribute parity is what downstream consumers rely on.
"""
def __init__(
self,
status_code: int,
detail: Any = None,
headers: Optional[dict] = None,
*,
model: str = "",
llm_provider: str = PROXY_LLM_PROVIDER_FALLBACK,
) -> None:
HTTPException.__init__(
self, status_code=status_code, detail=detail, headers=headers
)
self.status_code = status_code
self.model = model or ""
self.llm_provider = llm_provider or PROXY_LLM_PROVIDER_FALLBACK
# `message` is what RateLimitError.__str__ would print and what some
# observability callbacks log. Keep it human-readable.
self.message = detail if isinstance(detail, str) else str(detail)
# `RateLimitError.__str__` (resolved via MRO since Starlette's
# HTTPException doesn't define `__str__`) unconditionally reads
# these attributes. Set them so `str(exc)` doesn't raise
# AttributeError from logging/traceback paths.
self.num_retries: Optional[int] = None
self.max_retries: Optional[int] = None
def convert_priority_to_percent(
value: Union[float, PriorityReservationDict], model_info: Optional[ModelGroupInfo]

968
tests/test_litellm/proxy/hooks/test_proxy_rate_limit_provider_field.py Normal file
View File

@ -0,0 +1,968 @@
"""
Regression tests for the "provider field missing" bug on proxy-side
rate-limit errors.
Background
----------
The proxy's internal rate-limit hooks (parallel_request_limiter,
parallel_request_limiter_v3, dynamic_rate_limiter, dynamic_rate_limiter_v3,
batch_rate_limiter, max_budget_limiter, max_iterations_limiter,
max_budget_per_session_limiter) all fire from ``async_pre_call_hook`` —
*before* :func:`litellm.get_llm_provider` runs anywhere else in the request
lifecycle.
Until now, those hooks raised a bare ``HTTPException(429, ...)`` which carries
no ``llm_provider`` / ``model`` attribute. Downstream:
- The Prometheus ``litellm_proxy_failed_requests_metric`` reads
``exception.llm_provider`` via ``_get_exception_class_name`` — it came back
empty, so dashboards showed ``exception_class="HTTPException"`` with no
provider attribution.
- Observability callbacks that ``isinstance(e, RateLimitError)`` for
category routing missed these entirely.
The fix wraps every internal raise site in
:class:`ProxyHTTPRateLimitError` (an ``HTTPException`` *and* a
``litellm.RateLimitError``), and resolves ``model`` / ``llm_provider`` from
``data["model"]`` via :func:`get_llm_provider`. When the model is missing or
unparseable we fall back to ``llm_provider="litellm_proxy"`` so we never break
the request path with a second exception.
These tests pin both the happy path (provider correctly resolved) and the
fallback path (unknown model, missing model) for every limiter.
"""
import sys
from unittest.mock import AsyncMock, MagicMock, patch
import pytest
from fastapi import HTTPException
import litellm
from litellm.caching.caching import DualCache
from litellm.exceptions import RateLimitError
from litellm.proxy._types import UserAPIKeyAuth
from litellm.proxy.hooks.batch_rate_limiter import (
BatchFileUsage,
_PROXY_BatchRateLimiter,
)
from litellm.proxy.hooks.dynamic_rate_limiter import _PROXY_DynamicRateLimitHandler
from litellm.proxy.hooks.dynamic_rate_limiter_v3 import (
_PROXY_DynamicRateLimitHandlerV3,
)
from litellm.proxy.hooks.max_budget_limiter import _PROXY_MaxBudgetLimiter
from litellm.proxy.hooks.max_budget_per_session_limiter import (
_PROXY_MaxBudgetPerSessionHandler,
)
from litellm.proxy.hooks.max_iterations_limiter import _PROXY_MaxIterationsHandler
from litellm.proxy.hooks.parallel_request_limiter import (
_PROXY_MaxParallelRequestsHandler,
)
from litellm.proxy.hooks.parallel_request_limiter_v3 import (
_PROXY_MaxParallelRequestsHandler_v3,
)
from litellm.proxy.hooks.rate_limiter_utils import (
PROXY_LLM_PROVIDER_FALLBACK,
ProxyHTTPRateLimitError,
resolve_llm_provider_for_rate_limit,
)
from litellm.proxy.utils import InternalUsageCache
from litellm.types.agents import AgentResponse
# ---------------------------------------------------------------------------
# Helper class itself
# ---------------------------------------------------------------------------
class TestProxyHTTPRateLimitErrorClass:
"""Pin the dual ``HTTPException`` + ``RateLimitError`` shape."""
def test_is_both_http_exception_and_rate_limit_error(self):
e = ProxyHTTPRateLimitError(
status_code=429,
detail="boom",
model="gpt-4o-mini",
llm_provider="openai",
)
# FastAPI handler keys off HTTPException to render the 429.
assert isinstance(e, HTTPException)
# Prometheus / observability key off RateLimitError + .llm_provider.
assert isinstance(e, RateLimitError)
assert e.status_code == 429
assert e.model == "gpt-4o-mini"
assert e.llm_provider == "openai"
assert e.message == "boom"
assert e.detail == "boom"
def test_dict_detail_is_stringified_for_message(self):
# Some hooks pass a dict detail (e.g. dynamic_rate_limiter v1) — the
# `message` attr (read by RateLimitError.__str__ and observability
# callbacks) must still be a string.
e = ProxyHTTPRateLimitError(
status_code=429,
detail={"error": "over rpm"},
model="claude-3-5-sonnet",
llm_provider="anthropic",
)
assert isinstance(e.message, str)
assert "over rpm" in e.message
def test_defaults_to_litellm_proxy_provider(self):
e = ProxyHTTPRateLimitError(status_code=429, detail="x")
assert e.llm_provider == PROXY_LLM_PROVIDER_FALLBACK
assert e.model == ""
def test_none_provider_normalized_to_fallback(self):
e = ProxyHTTPRateLimitError(
status_code=429,
detail="x",
model=None, # type: ignore[arg-type]
llm_provider=None, # type: ignore[arg-type]
)
assert e.llm_provider == PROXY_LLM_PROVIDER_FALLBACK
assert e.model == ""
class TestResolveLLMProviderForRateLimit:
@pytest.mark.parametrize(
"model, expected_provider",
[
("gpt-4o-mini", "openai"),
("anthropic/claude-3-5-sonnet", "anthropic"),
("bedrock/meta.llama3-1-70b-instruct-v1:0", "bedrock"),
],
)
def test_known_models_resolve_provider(self, model, expected_provider):
resolved_model, provider = resolve_llm_provider_for_rate_limit(model)
assert provider == expected_provider
assert resolved_model # non-empty
@pytest.mark.parametrize("model", [None, "", "totally-not-a-real-model-name"])
def test_missing_or_unknown_model_falls_back(self, model):
# Must never raise — the resolver wraps `get_llm_provider` defensively
# because raising here would mask the rate-limit error we're trying
# to surface to the user.
resolved_model, provider = resolve_llm_provider_for_rate_limit(model)
assert provider == PROXY_LLM_PROVIDER_FALLBACK
# Resolver returns the input model verbatim on the unknown branch so
# the `.model` attribute is never silently swapped to a different one.
if not model:
assert resolved_model == ""
else:
assert resolved_model == model
def test_get_llm_provider_raising_is_swallowed(self):
# If get_llm_provider itself blows up (unexpected error), we still
# fall back rather than letting the secondary exception escape.
with patch.object(
litellm,
"get_llm_provider",
side_effect=RuntimeError("boom"),
):
resolved_model, provider = resolve_llm_provider_for_rate_limit("anything")
assert provider == PROXY_LLM_PROVIDER_FALLBACK
assert resolved_model == "anything"
# ---------------------------------------------------------------------------
# parallel_request_limiter v1
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_parallel_request_limiter_v1_populates_provider_when_at_rpm_limit():
"""
Trip the per-key RPM cap and assert the raised exception carries
``model`` / ``llm_provider`` resolved from ``data["model"]``.
"""
handler = _PROXY_MaxParallelRequestsHandler(
internal_usage_cache=InternalUsageCache(DualCache())
)
user_api_key_dict = UserAPIKeyAuth(
api_key="sk-rl-test",
max_parallel_requests=10,
rpm_limit=1,
tpm_limit=10,
)
data = {"model": "gpt-4o-mini"}
# First request consumes the budget.
await handler.async_pre_call_hook(
user_api_key_dict=user_api_key_dict,
cache=DualCache(),
data=data,
call_type="completion",
)
with pytest.raises(HTTPException) as exc_info:
await handler.async_pre_call_hook(
user_api_key_dict=user_api_key_dict,
cache=DualCache(),
data=data,
call_type="completion",
)
exc = exc_info.value
assert exc.status_code == 429
assert isinstance(exc, RateLimitError)
assert exc.llm_provider == "openai"
assert exc.model == "gpt-4o-mini"
@pytest.mark.asyncio
async def test_parallel_request_limiter_v1_zero_limit_path_populates_provider():
"""
When tpm_limit / rpm_limit is 0 the limiter takes the
``raise_rate_limit_error`` path. That path receives ``requested_model``
via the call-site change and must pass it through.
"""
handler = _PROXY_MaxParallelRequestsHandler(
internal_usage_cache=InternalUsageCache(DualCache())
)
user_api_key_dict = UserAPIKeyAuth(
api_key="sk-rl-zero",
max_parallel_requests=0,
rpm_limit=10,
tpm_limit=10,
)
with pytest.raises(HTTPException) as exc_info:
await handler.async_pre_call_hook(
user_api_key_dict=user_api_key_dict,
cache=DualCache(),
data={"model": "anthropic/claude-3-5-sonnet"},
call_type="completion",
)
exc = exc_info.value
assert exc.status_code == 429
assert isinstance(exc, RateLimitError)
assert exc.llm_provider == "anthropic"
assert exc.model == "claude-3-5-sonnet"
@pytest.mark.asyncio
async def test_parallel_request_limiter_v1_global_limit_populates_provider():
"""global_max_parallel_requests path also threads the model through."""
handler = _PROXY_MaxParallelRequestsHandler(
internal_usage_cache=InternalUsageCache(DualCache())
)
user_api_key_dict = UserAPIKeyAuth(api_key="sk-global")
# Pre-fill the global counter so the next call exceeds it.
await handler.internal_usage_cache.async_set_cache(
key="global_max_parallel_requests",
value=5,
local_only=True,
litellm_parent_otel_span=None,
)
with pytest.raises(HTTPException) as exc_info:
await handler.async_pre_call_hook(
user_api_key_dict=user_api_key_dict,
cache=DualCache(),
data={
"model": "bedrock/meta.llama3-1-70b-instruct-v1:0",
"metadata": {"global_max_parallel_requests": 1},
},
call_type="completion",
)
exc = exc_info.value
assert exc.status_code == 429
assert exc.llm_provider == "bedrock"
assert exc.model == "meta.llama3-1-70b-instruct-v1:0"
@pytest.mark.asyncio
async def test_parallel_request_limiter_v1_unknown_model_falls_back():
"""
When ``data["model"]`` is unparseable, the resolver falls back to
``litellm_proxy`` — and crucially does *not* leak a secondary exception.
"""
handler = _PROXY_MaxParallelRequestsHandler(
internal_usage_cache=InternalUsageCache(DualCache())
)
user_api_key_dict = UserAPIKeyAuth(
api_key="sk-rl-unknown",
max_parallel_requests=10,
rpm_limit=1,
tpm_limit=10,
)
data = {"model": "totally-not-a-real-model"}
await handler.async_pre_call_hook(
user_api_key_dict=user_api_key_dict,
cache=DualCache(),
data=data,
call_type="completion",
)
with pytest.raises(HTTPException) as exc_info:
await handler.async_pre_call_hook(
user_api_key_dict=user_api_key_dict,
cache=DualCache(),
data=data,
call_type="completion",
)
exc = exc_info.value
assert exc.status_code == 429
assert exc.llm_provider == PROXY_LLM_PROVIDER_FALLBACK
# Resolver returns the input verbatim so we don't silently relabel the
# model in the user-facing 429 detail.
assert exc.model == "totally-not-a-real-model"
@pytest.mark.asyncio
async def test_parallel_request_limiter_v1_missing_model_falls_back():
handler = _PROXY_MaxParallelRequestsHandler(
internal_usage_cache=InternalUsageCache(DualCache())
)
user_api_key_dict = UserAPIKeyAuth(
api_key="sk-rl-no-model",
max_parallel_requests=10,
rpm_limit=1,
tpm_limit=10,
)
await handler.async_pre_call_hook(
user_api_key_dict=user_api_key_dict,
cache=DualCache(),
data={},
call_type="completion",
)
with pytest.raises(HTTPException) as exc_info:
await handler.async_pre_call_hook(
user_api_key_dict=user_api_key_dict,
cache=DualCache(),
data={},
call_type="completion",
)
exc = exc_info.value
assert exc.llm_provider == PROXY_LLM_PROVIDER_FALLBACK
assert exc.model == ""
# ---------------------------------------------------------------------------
# parallel_request_limiter v3
# ---------------------------------------------------------------------------
def _v3_over_limit_response(rate_limit_type: str = "rpm") -> dict:
return {
"overall_code": "OVER_LIMIT",
"statuses": [
{
"code": "OVER_LIMIT",
"descriptor_key": "key",
"current_limit": 1,
"limit_remaining": -1,
"rate_limit_type": rate_limit_type,
}
],
}
@pytest.mark.asyncio
@pytest.mark.parametrize(
"model, expected_provider",
[
("gpt-4o-mini", "openai"),
("anthropic/claude-3-5-sonnet", "anthropic"),
],
)
async def test_parallel_request_limiter_v3_populates_provider(model, expected_provider):
handler = _PROXY_MaxParallelRequestsHandler_v3(
internal_usage_cache=InternalUsageCache(DualCache())
)
descriptors = [{"key": "key", "value": "v", "rate_limit": {"requests_per_unit": 1}}]
over = _v3_over_limit_response()
with pytest.raises(HTTPException) as exc_info:
handler._handle_rate_limit_error(
response=over,
descriptors=descriptors,
requested_model=model,
)
exc = exc_info.value
assert exc.status_code == 429
assert isinstance(exc, RateLimitError)
assert exc.llm_provider == expected_provider
# v3 may strip the "anthropic/" prefix in the resolved model — accept
# either; we only care that the provider field is correct and the model
# is non-empty.
assert exc.model
@pytest.mark.asyncio
async def test_parallel_request_limiter_v3_unknown_model_falls_back():
handler = _PROXY_MaxParallelRequestsHandler_v3(
internal_usage_cache=InternalUsageCache(DualCache())
)
descriptors = [{"key": "key", "value": "v", "rate_limit": {"requests_per_unit": 1}}]
with pytest.raises(HTTPException) as exc_info:
handler._handle_rate_limit_error(
response=_v3_over_limit_response(),
descriptors=descriptors,
requested_model="totally-bogus",
)
assert exc_info.value.llm_provider == PROXY_LLM_PROVIDER_FALLBACK
assert exc_info.value.model == "totally-bogus"
@pytest.mark.asyncio
async def test_parallel_request_limiter_v3_missing_model_falls_back():
handler = _PROXY_MaxParallelRequestsHandler_v3(
internal_usage_cache=InternalUsageCache(DualCache())
)
descriptors = [{"key": "key", "value": "v", "rate_limit": {"requests_per_unit": 1}}]
with pytest.raises(HTTPException) as exc_info:
handler._handle_rate_limit_error(
response=_v3_over_limit_response(),
descriptors=descriptors,
requested_model=None,
)
assert exc_info.value.llm_provider == PROXY_LLM_PROVIDER_FALLBACK
assert exc_info.value.model == ""
# ---------------------------------------------------------------------------
# dynamic_rate_limiter v1
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_dynamic_rate_limiter_v1_tpm_zero_populates_provider():
handler = _PROXY_DynamicRateLimitHandler(internal_usage_cache=DualCache())
handler.check_available_usage = AsyncMock(return_value=(0, 5, 100, 5, 1))
user_api_key_dict = UserAPIKeyAuth(api_key="sk-dyn")
user_api_key_dict.metadata = {}
with pytest.raises(HTTPException) as exc_info:
await handler.async_pre_call_hook(
user_api_key_dict=user_api_key_dict,
cache=DualCache(),
data={"model": "gpt-4o-mini"},
call_type="completion",
)
exc = exc_info.value
assert exc.status_code == 429
assert isinstance(exc, RateLimitError)
assert exc.llm_provider == "openai"
assert exc.model == "gpt-4o-mini"
@pytest.mark.asyncio
async def test_dynamic_rate_limiter_v1_rpm_zero_populates_provider():
handler = _PROXY_DynamicRateLimitHandler(internal_usage_cache=DualCache())
handler.check_available_usage = AsyncMock(return_value=(5, 0, 5, 100, 1))
user_api_key_dict = UserAPIKeyAuth(api_key="sk-dyn")
user_api_key_dict.metadata = {}
with pytest.raises(HTTPException) as exc_info:
await handler.async_pre_call_hook(
user_api_key_dict=user_api_key_dict,
cache=DualCache(),
data={"model": "anthropic/claude-3-5-sonnet"},
call_type="completion",
)
exc = exc_info.value
assert exc.llm_provider == "anthropic"
assert exc.model == "claude-3-5-sonnet"
@pytest.mark.asyncio
async def test_dynamic_rate_limiter_v1_unknown_model_falls_back():
handler = _PROXY_DynamicRateLimitHandler(internal_usage_cache=DualCache())
handler.check_available_usage = AsyncMock(return_value=(0, 5, 100, 5, 1))
user_api_key_dict = UserAPIKeyAuth(api_key="sk-dyn")
user_api_key_dict.metadata = {}
with pytest.raises(HTTPException) as exc_info:
await handler.async_pre_call_hook(
user_api_key_dict=user_api_key_dict,
cache=DualCache(),
data={"model": "no-such-model"},
call_type="completion",
)
assert exc_info.value.llm_provider == PROXY_LLM_PROVIDER_FALLBACK
assert exc_info.value.model == "no-such-model"
# ---------------------------------------------------------------------------
# dynamic_rate_limiter v3 — exercise just the raise path via the helper, not
# the full Redis/Lua stack.
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_dynamic_rate_limiter_v3_model_capacity_path_populates_provider():
"""
The v3 dynamic limiter has three raise sites: model_saturation_check,
priority_model, and the fail-closed unknown-descriptor branch. We patch
the atomic increment to short-circuit straight into the model_saturation
path — that's the most common production trip — and confirm the
raised exception carries provider info.
"""
from litellm.types.router import ModelGroupInfo
handler = _PROXY_DynamicRateLimitHandlerV3(internal_usage_cache=DualCache())
handler.v3_limiter.atomic_check_and_increment_by_n = AsyncMock(
return_value={
"overall_code": "OVER_LIMIT",
"statuses": [
{
"code": "OVER_LIMIT",
"descriptor_key": "model_saturation_check",
"current_limit": 100,
"limit_remaining": 0,
"rate_limit_type": "rpm",
}
],
}
)
handler._create_priority_based_descriptors = MagicMock(return_value=[])
handler._create_model_tracking_descriptor = MagicMock(
return_value={
"key": "model_saturation_check",
"value": "gpt-4o-mini",
"rate_limit": {"requests_per_unit": 100},
}
)
user_api_key_dict = UserAPIKeyAuth(api_key="sk-dyn-v3")
user_api_key_dict.metadata = {}
model_info = ModelGroupInfo(model_group="gpt-4o-mini", providers=["openai"])
with pytest.raises(HTTPException) as exc_info:
await handler._check_rate_limits(
model="gpt-4o-mini",
model_group_info=model_info,
user_api_key_dict=user_api_key_dict,
priority="default",
saturation=1.0,
data={"model": "gpt-4o-mini"},
)
exc = exc_info.value
assert exc.status_code == 429
assert isinstance(exc, RateLimitError)
assert exc.llm_provider == "openai"
assert exc.model == "gpt-4o-mini"
@pytest.mark.asyncio
async def test_dynamic_rate_limiter_v3_unknown_descriptor_path_populates_provider():
"""Fail-closed unknown-descriptor branch must still attribute provider."""
from litellm.types.router import ModelGroupInfo
handler = _PROXY_DynamicRateLimitHandlerV3(internal_usage_cache=DualCache())
handler.v3_limiter.atomic_check_and_increment_by_n = AsyncMock(
return_value={
"overall_code": "OVER_LIMIT",
"statuses": [
{
"code": "OVER_LIMIT",
"descriptor_key": "something_we_dont_handle",
"current_limit": 1,
"limit_remaining": 0,
"rate_limit_type": "rpm",
}
],
}
)
handler._create_priority_based_descriptors = MagicMock(return_value=[])
handler._create_model_tracking_descriptor = MagicMock(
return_value={
"key": "model_saturation_check",
"value": "gpt-4o-mini",
"rate_limit": {"requests_per_unit": 1},
}
)
user_api_key_dict = UserAPIKeyAuth(api_key="sk-dyn-v3-unknown")
user_api_key_dict.metadata = {}
model_info = ModelGroupInfo(model_group="gpt-4o-mini", providers=["openai"])
with pytest.raises(HTTPException) as exc_info:
await handler._check_rate_limits(
model="gpt-4o-mini",
model_group_info=model_info,
user_api_key_dict=user_api_key_dict,
priority="default",
saturation=1.0,
data={"model": "gpt-4o-mini"},
)
assert exc_info.value.llm_provider == "openai"
# ---------------------------------------------------------------------------
# batch_rate_limiter
# ---------------------------------------------------------------------------
def _batch_over_limit_response() -> dict:
return {
"overall_code": "OVER_LIMIT",
"statuses": [
{
"code": "OVER_LIMIT",
"descriptor_key": "key",
"current_limit": 10,
"limit_remaining": -5,
"rate_limit_type": "requests",
}
],
}
@pytest.mark.asyncio
async def test_batch_rate_limiter_populates_provider():
"""
batch_rate_limiter trips when the file's request/token count exceeds the
remaining window. The raise must thread `data["model"]` through the
helper.
"""
parallel_limiter = MagicMock()
parallel_limiter.window_size = 60
parallel_limiter._create_rate_limit_descriptors = MagicMock(
return_value=[
{"key": "key", "value": "v", "rate_limit": {"requests_per_unit": 10}}
]
)
parallel_limiter.atomic_check_and_increment_by_n = AsyncMock(
return_value=_batch_over_limit_response()
)
handler = _PROXY_BatchRateLimiter(
internal_usage_cache=InternalUsageCache(DualCache()),
parallel_request_limiter=parallel_limiter,
)
with pytest.raises(HTTPException) as exc_info:
await handler._check_and_increment_batch_counters(
user_api_key_dict=UserAPIKeyAuth(api_key="sk-batch"),
data={"model": "gpt-4o-mini"},
batch_usage=BatchFileUsage(total_tokens=100, request_count=15),
)
exc = exc_info.value
assert exc.status_code == 429
assert isinstance(exc, RateLimitError)
assert exc.llm_provider == "openai"
assert exc.model == "gpt-4o-mini"
@pytest.mark.asyncio
async def test_batch_rate_limiter_unknown_model_falls_back():
parallel_limiter = MagicMock()
parallel_limiter.window_size = 60
parallel_limiter._create_rate_limit_descriptors = MagicMock(
return_value=[
{"key": "key", "value": "v", "rate_limit": {"requests_per_unit": 10}}
]
)
parallel_limiter.atomic_check_and_increment_by_n = AsyncMock(
return_value=_batch_over_limit_response()
)
handler = _PROXY_BatchRateLimiter(
internal_usage_cache=InternalUsageCache(DualCache()),
parallel_request_limiter=parallel_limiter,
)
with pytest.raises(HTTPException) as exc_info:
await handler._check_and_increment_batch_counters(
user_api_key_dict=UserAPIKeyAuth(api_key="sk-batch"),
data={"model": "fake-model-xyz"},
batch_usage=BatchFileUsage(total_tokens=100, request_count=15),
)
assert exc_info.value.llm_provider == PROXY_LLM_PROVIDER_FALLBACK
# ---------------------------------------------------------------------------
# max_budget_limiter
# ---------------------------------------------------------------------------
@pytest.mark.asyncio
async def test_max_budget_limiter_populates_provider():
handler = _PROXY_MaxBudgetLimiter()
user_api_key_dict = UserAPIKeyAuth(
api_key="sk-budget",
user_id="user-1",
user_max_budget=10.0,
)
with patch(
"litellm.proxy.proxy_server.get_current_spend",
new=AsyncMock(return_value=10.0),
):
with pytest.raises(HTTPException) as exc_info:
await handler.async_pre_call_hook(
user_api_key_dict=user_api_key_dict,
cache=DualCache(),
data={"model": "gpt-4o-mini"},
call_type="completion",
)
exc = exc_info.value
assert exc.status_code == 429
assert isinstance(exc, RateLimitError)
assert exc.llm_provider == "openai"
assert exc.model == "gpt-4o-mini"
@pytest.mark.asyncio
async def test_max_budget_limiter_no_model_falls_back():
handler = _PROXY_MaxBudgetLimiter()
user_api_key_dict = UserAPIKeyAuth(
api_key="sk-budget",
user_id="user-1",
user_max_budget=10.0,
)
with patch(
"litellm.proxy.proxy_server.get_current_spend",
new=AsyncMock(return_value=10.0),
):
with pytest.raises(HTTPException) as exc_info:
await handler.async_pre_call_hook(
user_api_key_dict=user_api_key_dict,
cache=DualCache(),
data={},
call_type="completion",
)
assert exc_info.value.llm_provider == PROXY_LLM_PROVIDER_FALLBACK
assert exc_info.value.model == ""
# ---------------------------------------------------------------------------
# max_iterations_limiter
# ---------------------------------------------------------------------------
def _make_iter_agent(max_iterations: int) -> AgentResponse:
return AgentResponse(
agent_id="agent-iter",
agent_name="iter-agent",
litellm_params={"max_iterations": max_iterations},
agent_card_params={"name": "iter-agent", "version": "1.0.0"},
)
@pytest.mark.asyncio
async def test_max_iterations_limiter_populates_provider():
local_cache = DualCache()
handler = _PROXY_MaxIterationsHandler(
internal_usage_cache=InternalUsageCache(local_cache)
)
user_api_key_dict = UserAPIKeyAuth(api_key="sk-iter", agent_id="agent-iter")
with patch(
"litellm.proxy.agent_endpoints.agent_registry.global_agent_registry"
) as mock_registry:
mock_registry.get_agent_by_id.return_value = _make_iter_agent(max_iterations=1)
await handler.async_pre_call_hook(
user_api_key_dict=user_api_key_dict,
cache=local_cache,
data={
"model": "gpt-4o-mini",
"metadata": {"session_id": "session-iter-1"},
},
call_type="completion",
)
with pytest.raises(HTTPException) as exc_info:
await handler.async_pre_call_hook(
user_api_key_dict=user_api_key_dict,
cache=local_cache,
data={
"model": "gpt-4o-mini",
"metadata": {"session_id": "session-iter-1"},
},
call_type="completion",
)
exc = exc_info.value
assert exc.status_code == 429
assert isinstance(exc, RateLimitError)
assert exc.llm_provider == "openai"
assert exc.model == "gpt-4o-mini"
@pytest.mark.asyncio
async def test_max_iterations_limiter_unknown_model_falls_back():
local_cache = DualCache()
handler = _PROXY_MaxIterationsHandler(
internal_usage_cache=InternalUsageCache(local_cache)
)
user_api_key_dict = UserAPIKeyAuth(api_key="sk-iter", agent_id="agent-iter")
with patch(
"litellm.proxy.agent_endpoints.agent_registry.global_agent_registry"
) as mock_registry:
mock_registry.get_agent_by_id.return_value = _make_iter_agent(max_iterations=1)
await handler.async_pre_call_hook(
user_api_key_dict=user_api_key_dict,
cache=local_cache,
data={
"model": "no-such-model",
"metadata": {"session_id": "session-iter-2"},
},
call_type="completion",
)
with pytest.raises(HTTPException) as exc_info:
await handler.async_pre_call_hook(
user_api_key_dict=user_api_key_dict,
cache=local_cache,
data={
"model": "no-such-model",
"metadata": {"session_id": "session-iter-2"},
},
call_type="completion",
)
assert exc_info.value.llm_provider == PROXY_LLM_PROVIDER_FALLBACK
# ---------------------------------------------------------------------------
# max_budget_per_session_limiter
# ---------------------------------------------------------------------------
def _make_session_budget_agent(max_budget: float) -> AgentResponse:
return AgentResponse(
agent_id="agent-session-budget",
agent_name="session-budget-agent",
litellm_params={"max_budget_per_session": max_budget},
agent_card_params={"name": "session-budget-agent", "version": "1.0.0"},
)
@pytest.mark.asyncio
async def test_max_budget_per_session_limiter_populates_provider():
handler = _PROXY_MaxBudgetPerSessionHandler(
internal_usage_cache=InternalUsageCache(DualCache())
)
user_api_key_dict = UserAPIKeyAuth(
api_key="sk-session-budget", agent_id="agent-session-budget"
)
with patch(
"litellm.proxy.agent_endpoints.agent_registry.global_agent_registry"
) as mock_registry:
mock_registry.get_agent_by_id.return_value = _make_session_budget_agent(
max_budget=1.0
)
with patch.object(
handler, "_get_current_spend", new=AsyncMock(return_value=5.0)
):
with pytest.raises(HTTPException) as exc_info:
await handler.async_pre_call_hook(
user_api_key_dict=user_api_key_dict,
cache=DualCache(),
data={
"model": "anthropic/claude-3-5-sonnet",
"metadata": {"session_id": "session-budget-1"},
},
call_type="completion",
)
exc = exc_info.value
assert exc.status_code == 429
assert isinstance(exc, RateLimitError)
assert exc.llm_provider == "anthropic"
@pytest.mark.asyncio
async def test_max_budget_per_session_limiter_unknown_model_falls_back():
handler = _PROXY_MaxBudgetPerSessionHandler(
internal_usage_cache=InternalUsageCache(DualCache())
)
user_api_key_dict = UserAPIKeyAuth(
api_key="sk-session-budget", agent_id="agent-session-budget"
)
with patch(
"litellm.proxy.agent_endpoints.agent_registry.global_agent_registry"
) as mock_registry:
mock_registry.get_agent_by_id.return_value = _make_session_budget_agent(
max_budget=1.0
)
with patch.object(
handler, "_get_current_spend", new=AsyncMock(return_value=5.0)
):
with pytest.raises(HTTPException) as exc_info:
await handler.async_pre_call_hook(
user_api_key_dict=user_api_key_dict,
cache=DualCache(),
data={
"model": "no-such-model",
"metadata": {"session_id": "session-budget-2"},
},
call_type="completion",
)
assert exc_info.value.llm_provider == PROXY_LLM_PROVIDER_FALLBACK
# ---------------------------------------------------------------------------
# Prometheus integration: failure metric reads exception.llm_provider
# via _get_exception_class_name. With the fix, this returns
# "Openai.RateLimitError" instead of plain "HTTPException" for proxy-side
# 429s on a known model. Pin that contract — that's what dashboards see.
# ---------------------------------------------------------------------------
def test_prometheus_exception_class_name_includes_provider():
from litellm.integrations.prometheus import PrometheusLogger
exc = ProxyHTTPRateLimitError(
status_code=429,
detail="over limit",
model="gpt-4o-mini",
llm_provider="openai",
)
name = PrometheusLogger._get_exception_class_name(exc)
# Format is "{Provider.}{ClassName}" per `_get_exception_class_name`.
assert name.startswith("Openai.")
# And specifically: it ends in our exception class. (We don't pin the
# full string to avoid coupling the test to PR #27687's parallel rename.)
assert name.endswith("ProxyHTTPRateLimitError")
def test_prometheus_exception_class_name_falls_back_when_no_model():
from litellm.integrations.prometheus import PrometheusLogger
exc = ProxyHTTPRateLimitError(status_code=429, detail="over limit")
name = PrometheusLogger._get_exception_class_name(exc)
# `litellm_proxy` -> `Litellm_proxy.` (capitalize first char only).
assert name.startswith("Litellm_proxy.")
if __name__ == "__main__":
sys.exit(pytest.main([__file__, "-vv", "-x"]))