* Cato Networks guardrail, based on Aim (#26597)
* Aim was acquired by Cato Networks, creating Cato Networks guardrail based on Aim
* Add more tests
* Move test so they are reached by codecov coverage
* base URL trailing slashes
* Support Lemonade runtime context metadata (#28135)
* Support Lemonade runtime context metadata
* Add provider hook for runtime model metadata
* Address provider model info review feedback
Keep the runtime model info hook duck-typed instead of extending the base model-info class, and avoid importing ModelInfoBase from Ollama common utilities to reduce CodeQL cyclic-import noise.
Co-authored-by: openhands <openhands@all-hands.dev>
* Fix CI after staging rebase
Relax the Ollama runtime metadata return annotation to match the provider-hook dict response and update the Google Interactions OpenAPI status expectation for the current live spec.
Co-authored-by: openhands <openhands@all-hands.dev>
* Normalize Lemonade runtime model metadata
* Avoid leaking Ollama metadata auth
* Avoid leaking Lemonade metadata auth
---------
Co-authored-by: Graham Neubig <398875+neubig@users.noreply.github.com>
Co-authored-by: openhands <openhands@all-hands.dev>
* fix(cato): address guardrail review feedback
Use proxy-authenticated user identity, forward moderation hook return values,
and ensure streaming sender tasks are cancelled and awaited on exit.
Co-authored-by: Cursor <cursoragent@cursor.com>
* fix(vertex_ai): route google/gemma-*-maas through partner-models OpenAI path - clone of #28010 (#28846)
* fix(vertex_ai): route google/gemma-*-maas through partner-models OpenAI path
Fixes#26083
vertex_ai/google/gemma-4-26b-a4b-it-maas previously fell through to the
NON_GEMINI route. Per owtaylor's plan on #26083: add the google/gemma-
prefix to PartnerModelPrefixes so is_vertex_partner_model picks it up
and should_use_openai_handler routes it to the OpenAI-compatible
/endpoints/openapi/chat/completions URL. No gemma-detection exclusion
needed (the "gemma/" check uses a slash, which google/gemma-... doesn't
match). No OpenAIGPTConfig subclass needed — works with the base handler.
* fix(vertex_ai): mark gemma-4-26b-a4b-it-maas as vision-capable (empirically verified)
* fix(vertex_ai): address greptile feedback — provider category, canonical URL, sync backup
* test(vertex_ai): add function-calling and vision pass-through tests for Gemma MaaS
Addresses oss-pr-review-agent-shin feedback on PR #28010:
supports_function_calling, supports_tool_choice, and supports_vision were
marked true but had no tests proving the payloads actually reached the
OpenAI-compatible endpoint.
Added:
- test_gemma_maas_supports_function_calling — verifies the utility returns True
when the model_cost entry carries supports_function_calling=true
- test_gemma_maas_supports_vision — same for supports_vision
- test_vertex_ai_gemma_function_calling_passthrough — verifies tools + tool_choice
appear in the JSON body POSTed to /endpoints/openapi/chat/completions
- test_vertex_ai_gemma_vision_passthrough — verifies image_url content parts
survive transformation and reach the global endpoint URL
* fix: Delete uv.lock
* test(vertex_ai): add function-calling and vision pass-through tests for Gemma MaaS
Addresses oss-pr-review-agent-shin feedback on PR #28010:
P1 (patch target): Added a comment explaining why patching
litellm.llms.custom_httpx.http_handler.AsyncHTTPHandler is correct —
get_async_httpx_client() (defined in http_handler.py) instantiates
AsyncHTTPHandler within that module's scope, so the definition-site patch
intercepts it. Without the mock the test raises AuthenticationError,
confirming it never silently passes.
P2 (partner-provider regression guard): Added
test_gemma_routes_through_openai_handler() which calls
VertexAIPartnerModels.should_use_openai_handler() directly, so if Gemma's
routing to VertexPartnerProvider.llama ever changes the URL-shape tests
below it become a real regression guard rather than an unanchored unit test.
Also added:
- test_gemma_maas_supports_function_calling / supports_vision — capability
flag checks via patch.dict(litellm.model_cost)
- test_vertex_ai_gemma_function_calling_passthrough — tools + tool_choice
forwarded in the request body
- test_vertex_ai_gemma_vision_passthrough — image_url part survives
transformation to the global endpoint
Added:
- test_gemma_maas_supports_function_calling — verifies the utility returns True
when the model_cost entry carries supports_function_calling=true
- test_gemma_maas_supports_vision — same for supports_vision
- test_vertex_ai_gemma_function_calling_passthrough — verifies tools + tool_choice
appear in the JSON body POSTed to /endpoints/openapi/chat/completions
- test_vertex_ai_gemma_vision_passthrough — verifies image_url content parts
survive transformation and reach the global endpoint URL
* fix: proper patch for unit tests
---------
Co-authored-by: Iana <iana@Shivakumars-MacBook-Pro.local>
* fix(cato): guardrail all completion choices on output
When n > 1, only choices[0] was analyzed and redacted. Iterate every
Choices entry so block and anonymize actions apply to all completions.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix review
* fix(cato_networks): harden output anonymize handling and restructure nested UI routes
Guard against empty redacted_output and empty all_redacted_messages from Cato.
Restructure nested admin UI HTML exports to index.html so extensionless routes work.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Fix mypy
* fix(cato): guard missing policy_drill_down and all_redacted_messages keys
* fix(cato): avoid KeyError bypassing block action on missing analysis_result
* fix(cato): preserve non-text message fields during anonymize
Rebuild redacted messages from the original messages, overwriting only
content, so tool_calls, tool_call_id, name and multimodal fields survive
the anonymize action.
* fix(cato): preserve trailing messages when fewer redacted messages returned
Avoid silently truncating the conversation in _anonymize_request when Cato
returns fewer redacted messages than were sent, and isolate the no-api-key
config test from a pre-existing CATO_API_KEY environment variable.
* fix(cato,model-info): preserve stream block signal on sender teardown; forward api_key in dynamic model-info lookup
Suppress ConnectionClosed (alongside CancelledError) when tearing down the
Cato streaming sender task so a backend ConnectionClosed cannot mask the
original StreamingCallbackError (e.g. a guardrail block) raised by the
receive loop.
Thread api_key through get_model_info -> _get_model_info_helper so an
explicit key reaches a provider's dynamic get_model_info for a caller-supplied
api_base. Previously only api_base was forwarded, so authenticated Ollama and
Lemonade servers at a custom base could only be queried unauthenticated.
* fix(cato): surface mid-stream forwarding errors instead of blocking on recv
If the upstream LLM stream errors mid-flight, the sender task dies before
sending the terminal done frame, so the consumer would block on websocket.recv()
until Cato closes the connection. Race recv against the sender task and raise the
stored sender exception promptly as a StreamingCallbackError.
* fix(cato): drop spoofable end_user_id from guardrail user identity
Only the key/JWT-bound user_email is a trusted identity. end_user_id is
resolved from caller-supplied request fields (OpenAI user param, headers,
metadata), so an authenticated caller with no bound user_email could set it
to another user's email and have LiteLLM forward x-cato-user-email for that
victim, poisoning Cato audit and policy attribution. Forward only user_email
and omit the header otherwise.
* fix(cato): harden output anonymize path against missing content key
* fix(cato): fall back to original message when redacted content key is missing
* refactor(model-info): drop unused api_key from cached model-info helper
_cached_get_model_info_helper is only called by the cost-tracking hot path,
which never authenticates, so the api_key parameter was never populated.
Keeping it in the lru_cache key offered no benefit and risked fragmenting
the high-RPS cache and retaining credential strings per entry.
* fix(cato): preserve None content on tool-call-only choices in output hook
* fix(ollama): respect static-model guard in OllamaConfig.get_model_info
Delegate to OllamaModelInfo.get_model_info so statically-priced Ollama
models short-circuit before the /api/show network call instead of
hitting the server unconditionally.
* fix(lemonade,ollama): treat empty api_key as unset to avoid leaking server creds
An empty-string api_key was treated as an explicit key, so it passed the
guard meant to keep server-side credentials off caller-supplied bases and
then fell back through the env/global key chain. A caller could point
api_base at a server they control and send api_key="" to receive the
configured provider key in the Authorization header. Gate the credential
fallback on the api_key being truthy instead of merely not-None.
* fix(cato): inspect and redact Responses-API input, not just messages
The guardrail only read data["messages"], so /v1/responses requests, which
carry their text in data["input"], reached Cato as an empty message list
and bypassed inspection entirely. Send build_inspection_messages(data) so
both shapes are analyzed, and write anonymized results back with
apply_redacted_messages_back when the request used input.
* perf(utils): keep api_key out of get_model_info lru_cache key
* fix(cato): propagate ssl_verify to streaming WebSocket connection
The streaming hook applied ssl_verify only to the HTTP handler; the
websockets.connect() call used default verification, so a custom Cato
instance behind TLS with a self-signed cert worked for non-streaming
calls but failed every streaming request. Resolve the ssl_verify setting
into the connect() ssl argument, mirroring the HTTP handler.
* refactor(utils): rename shadowing local in _get_model_info_helper
* fix(cato): flatten multimodal chat content before inspection
Chat Completions requests whose message content is a multimodal parts
array were posted to Cato as the raw OpenAI parts, so text inside
content: [{"type":"text", ...}] reached the model without Cato ever
inspecting the string. Flatten each message's list content to plain text
while keeping the list 1:1 with the request so the index-based redaction
write-back stays valid; Responses-API input requests still go through
build_inspection_messages.
* test(lemonade): clear get_model_info cache around api_base test
* fix(cato): inspect and redact Responses-API input even when messages present
_inspection_messages returned early once messages was non-empty, so a
/v1/responses caller could place benign text in messages and disallowed
text in input and have only messages reach Cato while the model used
input. Inspect both fields and write anonymize redactions back to input
as well as the index-aligned messages.
* test(log_db_metrics): assert table_name event_metadata contract
log_db_metrics now emits minimal event_metadata via _safe_db_event_metadata
(table_name only, function_name/function_kwargs/function_args dropped as
redundant with call_type and unsafe to stamp on a span). The success-path
test still asserted function_name membership and crashed with TypeError on
the None metadata returned when no table_name is passed. Pass a table_name
and assert the surfaced contract instead.
* fix(cato): inspect and redact completion prompt and Responses-API instructions
The Cato guardrail only inspected chat messages and the Responses-API input field, so blocked text placed in the legacy /v1/completions prompt or the /v1/responses instructions field reached the model without ever being sent to Cato. Both fields are now appended as synthetic inspection messages, and the anonymize path slices Cato's redactions back to the field they came from.
* fix(cato): serialize non-str/bytes websocket chunks before forwarding
* fix(cato): inspect tool descriptions and tool-call arguments
* fix(cato): map redacted output by assistant index; restore get_model_info.cache_info
* fix(cato): block output even when detection_message is null/empty
A block_action returned by Cato on the output hook whose detection_message
was null or empty was let through to the caller: the truthiness guard on
detection_message skipped the HTTPException and the unblocked response was
returned. Raise the HTTPException directly in _handle_block_action_on_output
so the output path blocks unconditionally, mirroring the input path.
* fix(cato): inspect and redact nested tool param and legacy function descriptions
Tool/function parameter descriptions and the legacy functions[] array are
forwarded to the model but were not seen by Cato, so blocked text hidden there
bypassed inspection and anonymization. Recursively walk every description string
in tools[].function and functions[] schemas for both the analyze payload and the
anonymize write-back.
* fix(cato): traverse schema descriptions iteratively to satisfy recursive detector
The nested walk() generator recursed over tool/function JSON schemas with no
depth bound, which the recursive_detector code-quality gate rejects. Replace it
with an explicit-stack DFS that yields the same (container, key) refs in the
same pre-order, so schema description redaction is unchanged.
* fix(cato): inspect and redact response_format JSON schema descriptions
response_format json_schema descriptions are forwarded to the model, so
blocked text hidden in nested schema descriptions could bypass Cato
inspection and redaction. Extend the schema-description walk to cover
response_format alongside tools and legacy functions.
* fix(cato): skip output rewrite when Cato returns no redaction
Return None from call_cato_guardrail_on_output on monitor/no-action so the
post-call hook only mutates the message when there is an actual redaction,
instead of redundantly re-writing the original content.
* refactor(utils): resolve explicit api_key model info without the cache
Move the model-info build into a non-cached _build_model_info helper and drop
api_key from the lru-cached _cached_get_model_info signature. Both cached
helpers now take the same (model, provider, api_base) key and never forward
api_key, while explicit per-caller keys are resolved through the builder
directly instead of reaching into the cache wrapper's __wrapped__.
* fix(cato): inspect and redact non-description schema string values
Tool, function and response_format JSON schemas forward more than just
description text to the model. enum, const, default, examples and title
values are sent verbatim, so blocked content hidden in any of them
bypassed Cato inspection and redaction. Walk those schema string values
alongside descriptions on both the inspection and anonymize paths.
* fix(model-info): surface swallowed dynamic model-info errors
The provider-specific get_model_info dispatch falls back to the static cost
map when a provider's dynamic lookup raises, which is intentional graceful
degradation. Previously the exception was discarded with a bare debug line,
so a real failure (e.g. a provider whose get_model_info signature does not
accept api_key) was invisible. Log the exception at warning level with the
model and provider context so the fallback is diagnosable.
* fix(cato): inspect and redact Responses API output in post-call hook
The post-call success hook only handled ModelResponse, so /v1/responses
(which returns a ResponsesAPIResponse) bypassed the Cato output guardrail.
Extract and inspect/redact every output_text content block and function-call
arguments string, blocking on a block action, so generated text cannot escape
inspection by using the Responses API.
* chore: reset _experimental/out folder
* chore(ui): remove orphaned prebuilt dashboard chunk files
The _experimental/out manifests are byte-identical to the base branch, so the
served dashboard already matches base. 436 unreferenced Next.js chunk files had
accumulated in the directory and are not loaded by any manifest; removing them
restores the committed UI artifacts to the base build and drops the artifact
churn from this PR's diff.
* fix(guardrails,ollama): forward ssl_verify to Cato init and raise_for_status on /api/show
---------
Co-authored-by: Alex Yaroslavsky <trexinc@gmail.com>
Co-authored-by: Graham Neubig <neubig@gmail.com>
Co-authored-by: Graham Neubig <398875+neubig@users.noreply.github.com>
Co-authored-by: openhands <openhands@all-hands.dev>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Piotr Placzko <piotr@icep-design.com>
Co-authored-by: Iana <iana@Shivakumars-MacBook-Pro.local>
Co-authored-by: mateo-berri <277851410+mateo-berri@users.noreply.github.com>
e8fcb01215