litellm/tests/logging_callback_tests
ishaan-berri 8a9faa81b2
feat(guardrails): LLM-as-a-Judge guardrail (#26360)
* feat(guardrails): add LLM_AS_A_JUDGE to SupportedGuardrailIntegrations

* feat(types): add EvalVerdict, StandardLoggingEvalInformation; wire eval_information into SpendLogsMetadata

* feat(guardrails): add self-contained llm_as_a_judge guardrail hook

* fix(a2a): filter agent-only litellm_params from acompletion kwargs; pass agent_id into body

* feat(ui): add LLMJudgeFields criteria builder component

* feat(ui): wire LLM-as-a-Judge into add guardrail form

* feat(ui): update EvalViewer — title 'LLM Judge Results', weighted score column, summary row

* fix(ui): wire EvalViewer into LogDetailContent to show LLM judge results on logs page

* fix(guardrails-ui): route llm_as_a_judge to criteria builder step; rename to LiteLLM LLM as a Judge; add litellm logo

* fix(guardrail-viewer): stack lifecycle + eval details vertically to avoid badge overflow in narrow drawer

* fix(guardrail-create): surface config validation errors on create instead of silently orphaning guardrail in DB

* fix(guardrail-registry): hardcode llm_as_a_judge in initializer registry so it loads regardless of package install path

* fix(llm-as-a-judge): fix P1 code quality issues - validate weights/on_failure, guard pre_call, handle multimodal, move imports to module level, fix spurious finally logging

* fix(guardrail_endpoints): use correct PK field in rollback delete and log rollback failure

* fix(llm_as_a_judge): support Pydantic object in _get_litellm_param fallback chain

* fix(LLMJudgeFields): replace @tremor/react Button with antd Button

* fix(llm_as_a_judge): remove dead registry dicts, fix KeyError in prompt builder, set correct status on judge failure

* test(llm_as_a_judge): add unit tests for guardrail hook

* fix(llm_as_a_judge): remove @log_guardrail_information decorator to fix duplicate guardrail_information entries

The decorator and the manual finally block both called add_standard_logging_guardrail_information_to_request_data, producing two entries per request. The decorator also misclassified HTTPException(422) blocks as guardrail_failed_to_respond (it checks for 400). The finally block correctly tracks status throughout, so removing the decorator is sufficient.

* fix(test_gcs_pub_sub): ignore metadata.eval_information in comparison

* fix(test_spend_management): ignore metadata.eval_information in payload comparison

* fix(types/guardrails): add input_type and messages to ApplyGuardrailRequest

* fix(guardrail_endpoints): pass input_type and messages through apply_guardrail endpoint

* fix(guardrail_endpoints): auto-detect post_call guardrails and use input_type=response

* fix(a2a_endpoints): merge agent litellm_params guardrails into data before post_call hooks

* fix(llm_as_a_judge): use float sum with tolerance for weight validation

* fix(guardrail_registry): split long import line for black formatting

* fix(llm_as_a_judge): guard guardrail_name Optional for mypy

* fix(llm_as_a_judge): set guardrail_status=guardrail_intervened when score fails, regardless of on_failure mode

* fix(a2a_endpoints): use try/finally so deferred spend log fires even when guardrail blocks with 422

* fix(litellm_logging): declare _defer_async_logging and _enqueue_deferred_logging on Logging class for mypy

* fix(logging_worker): restore queue.join() in flush() to wait for in-flight callbacks
2026-04-24 17:15:32 -07:00
..
gcs_pub_sub_body test fixes 2026-03-30 16:59:27 -07:00
langfuse_expected_request_body [Test Fix] fix gov pricing tests (#25022) 2026-04-02 15:55:55 -07:00
base_test.py
conftest.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
create_mock_standard_logging_payload.py
gettysburg.wav
log.txt
test_alerting.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_amazing_s3_logs.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_assemble_streaming_responses.py
test_bedrock_knowledgebase_hook.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_built_in_tools_cost_tracking.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_custom_callback_router.py test: test 2026-03-28 19:17:38 -07:00
test_datadog_llm_obs.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_datadog.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_dynamic_otel_keys.py Litellm fix langfuse otel trace (#20382) 2026-02-03 22:40:19 -08:00
test_gcs_pub_sub.py feat(guardrails): LLM-as-a-Judge guardrail (#26360) 2026-04-24 17:15:32 -07:00
test_generic_api_callback.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_humanloop_unit_tests.py
test_langfuse_e2e_test.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_langfuse_unit_tests.py add tests for fix 2026-03-15 00:58:08 +05:30
test_langsmith_unit_test.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_log_db_redis_services.py
test_logging_redaction_e2e_test.py Merge remote-tracking branch 'origin/litellm_internal_staging' into litellm_yj_apr17 2026-04-17 17:36:40 -07:00
test_moderations_api_logging.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_opentelemetry_unit_tests.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_otel_logging.py fix(logging): add litellm_call_id to StandardLoggingPayload and OTel span (#26133) 2026-04-21 15:24:32 -07:00
test_pagerduty_alerting.py
test_posthog.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_spend_logs.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_sqs_logger.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_standard_logging_payload_excluded_fields.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_standard_logging_payload.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_token_counting.py test_stream_token_counting_anthropic_with_include_usage 2025-10-31 20:24:58 -07:00
test_unit_test_litellm_logging.py
test_unit_tests_init_callbacks.py style: run black formatter on files from main merge 2026-04-17 13:02:59 -07:00
test_view_request_resp_logs.py Fix CI: Revert security scan changes and add GitGuardian ignore rules (#18358) 2025-12-22 17:03:53 -08:00