litellm/tests/logging_callback_tests
rioiart 1ac2655b17
Fix/organization max budget not enforced (#17334)
* test: add failing tests for organization budget enforcement bug

Add comprehensive tests exposing that organization-level budgets are
retrieved but never enforced during request authentication. Tests verify:

1. Basic org budget exceeded scenario (team under budget, org over)
2. Multiple teams collectively exceeding org budget
3. Organization budget fields exist but are never checked
4. Inconsistency between team budget enforcement (works) and org (doesn't)

Tests intentionally fail to document the bug. Will be fixed in next commit.

Related to organization_max_budget not being enforced in auth_checks.py

* fix: enforce organization budget in auth checks

Add organization budget enforcement to common_checks() in auth_checks.py.
Previously, organization_max_budget was retrieved from DB but never checked,
allowing teams to collectively exceed their organization's budget limit.

Changes:
- Add _organization_max_budget_check() function following team budget pattern
- Call org budget check after team budget check in common_checks()
- Add "organization_budget" to budget_alerts type literals
- Update tests to verify org budget is enforced

Budget hierarchy is now properly enforced:
  Organization Budget (hard ceiling)
    └─ Team Budget (sub-allocation)
        └─ Team Member Budget (per-user within team)
            └─ Key Budget (per-key)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>

* fix: add organization_id to budget alerts, fix enum comparison and linting of newly added code

- Add organization_id field to CallInfo class for better alert context
- Include organization_id in budget alerts (token, soft, team, org)
- Fix event_group enum comparison (was comparing enum to string)
- Add OrganizationBudgetAlert class for organization budget alerting
- Add organization_budget to test parameterizations
- Apply Black formatting to slack_alerting.py

---------

Co-authored-by: Claude <noreply@anthropic.com>
2025-12-02 22:46:03 -08:00
..
gcs_pub_sub_body Changes for CI/CD Tests 2025-11-22 10:21:52 -08:00
langfuse_expected_request_body test logging tests + mcp server QA checks 2025-11-15 08:58:46 -08:00
base_test.py
conftest.py [Perf] Improvements for Async Success Handler (Logging Callbacks) - Approx +130 RPS (#13905) 2025-08-23 13:13:23 -07:00
create_mock_standard_logging_payload.py
gettysburg.wav
log.txt
test_alerting.py Fix/organization max budget not enforced (#17334) 2025-12-02 22:46:03 -08:00
test_amazing_s3_logs.py fix tests 2025-10-25 10:19:24 -07:00
test_assemble_streaming_responses.py
test_azure_blob_storage.py
test_bedrock_knowledgebase_hook.py [Fix] - Bedrock Knowledge Bases - add support for filtering kb queries (#16543) 2025-11-12 12:38:50 -08:00
test_built_in_tools_cost_tracking.py fix: use fastuuid helper (#14903) 2025-09-25 15:47:01 -07:00
test_custom_callback_router.py fix tests 2025-10-25 10:19:24 -07:00
test_datadog_llm_obs.py [Feat] Allow redacting message / response content for specific logging integrations - DD LLM Observability (#13158) 2025-07-31 16:44:16 -07:00
test_datadog.py Fix: Datadog callback regression when ddtrace is installed (#17393) 2025-12-02 17:27:50 -08:00
test_gcs_pub_sub.py [Feat] Responses API Session Handling - Multi media support (#13347) 2025-08-07 10:59:53 -07:00
test_generic_api_callback.py make generic api OSS + support multiple generic API's (#17152) 2025-11-26 18:38:38 -08:00
test_humanloop_unit_tests.py
test_langfuse_e2e_test.py fix LF tests 2025-10-04 10:39:17 -07:00
test_langfuse_unit_tests.py
test_langsmith_unit_test.py test logging tests + mcp server QA checks 2025-11-15 08:58:46 -08:00
test_log_db_redis_services.py
test_logging_redaction_e2e_test.py test_redaction_responses_api 2025-11-08 13:43:22 -08:00
test_moderations_api_logging.py fix: use fastuuid helper (#14903) 2025-09-25 15:47:01 -07:00
test_opentelemetry_unit_tests.py test_opentelemetry_integration 2025-11-26 12:28:47 -08:00
test_otel_logging.py [Feat] OTEL - Log Cost Breakdown on OTEL Logger (#16334) 2025-11-06 13:53:53 -08:00
test_pagerduty_alerting.py
test_posthog.py fix: Add atexit handlers to flush callbacks for async completions (#16487) 2025-11-11 19:12:53 -08:00
test_spend_logs.py fix tests 2025-10-25 10:19:24 -07:00
test_sqs_logger.py Add Langfuse OTEL and SQS to health check (#16514) 2025-11-12 18:25:30 -08:00
test_standard_logging_payload.py [Feat] - Cost Tracking - show input, output, tool call cost breakdown in StandardLoggingPayload (#14921) 2025-09-25 15:48:22 -07:00
test_token_counting.py test_stream_token_counting_anthropic_with_include_usage 2025-10-31 20:24:58 -07:00
test_unit_test_litellm_logging.py
test_unit_tests_init_callbacks.py use_callback_in_llm_call 2025-10-01 18:32:37 -07:00
test_view_request_resp_logs.py fix: use fastuuid helper (#14903) 2025-09-25 15:47:01 -07:00