725c0c158f
* fix(proxy): readiness check returns 200 when database is unreachable _db_health_readiness_check() catches health_check() exceptions but never updates db_health_cache to "disconnected" and never re-raises. The caller health_readiness() always returns 200 with "db": "connected" hardcoded, regardless of actual DB state. In Kubernetes, this means pods with dead database connections stay in the Service endpoints and continue receiving traffic they cannot serve. Changes: - Set db_health_cache to "disconnected" and re-raise the exception on health_check failure so health_readiness() returns 503 - Use actual db_health_status["status"] in the response instead of hardcoding "db": "connected" - Reduce cache TTL from 2 minutes to 15 seconds. The 2-minute window is too wide for readiness probes (typically 10-15s intervals) and means a pod can report healthy for up to 2 minutes after the DB dies - Only serve cached results when status is "connected". The previous condition (status != "unknown") would also cache "disconnected" for 2 minutes, delaying recovery detection after a DB comes back * fix(proxy): add DB connection self-healing to readiness check When the Prisma query engine's internal TCP connection pool holds dead connections (caused by network blips, Cloud SQL proxy restarts, or node-level issues), health_check() fails with httpx.ConnectError. The engine never recovers on its own because nothing triggers a disconnect/connect cycle to restart the subprocess with fresh connections. This leaves pods permanently failing readiness checks until they are manually restarted, even after the underlying DB becomes reachable again. Add a reconnect attempt to _db_health_readiness_check() when health_check() fails: 1. disconnect() - kills the query engine subprocess and closes all connections (has built-in backoff retry: 3 tries, 10s max) 2. connect() - starts a new engine with fresh TCP connections (has built-in backoff retry: 3 tries, 10s max) 3. health_check() - verifies the new connection works (has built-in backoff retry: 3 tries, 10s max) If reconnect succeeds, the pod immediately returns to service (200). If it fails, the original exception is re-raised (503). Reconnect attempts are rate-limited by probe frequency (~10-15s), so a permanently unreachable DB gets one attempt per cycle with no retry loops. This uses the same disconnect/connect mechanism that PrismaWrapper.recreate_prisma_client() uses for IAM token refresh, and aligns with the community-documented pattern for Prisma connection recovery in long-running processes (prisma/prisma#24718, #27024). * Add poetry lock and modify test_health_endpoints * Address allow_requests_on_db_unavailable regression * Address comments * resolve greptile issue * Restore accidentally deleted UI HTML files These were removed in an earlier commit but still exist on main. Restoring to keep the PR diff clean. * Guard reconnect with is_database_transport_error Only attempt disconnect/connect/health_check cycle for transport-level failures (unreachable DB, dropped connection). Data-layer errors like UniqueViolationError indicate the DB is reachable, so reconnecting would be pointless churn. * Address greptile's comments * Fix module alias after rebase and add adversarial test coverage - Unify module alias to _health_endpoints_module after rebase conflict - Add test for non-transport error with flag on (exercises is_database_transport_error guard) - Add test for disconnect() failure during reconnect cycle - Split non-transport error test into flag-off (re-raises) and flag-on (skips reconnect) variants * Remove stale UI HTML files reintroduced during rebase |
||
|---|---|---|
| .. | ||
| a2a_protocol | ||
| anthropic_interface/exceptions | ||
| caching | ||
| completion_extras | ||
| containers | ||
| enterprise | ||
| expected_responses_api_request | ||
| experimental_mcp_client | ||
| google_genai | ||
| images | ||
| integrations | ||
| interactions | ||
| litellm_core_utils | ||
| llms | ||
| ocr | ||
| passthrough | ||
| proxy | ||
| responses | ||
| router_strategy | ||
| router_utils | ||
| secret_managers | ||
| test_router | ||
| types | ||
| vector_stores | ||
| __init__.py | ||
| conftest.py | ||
| log.txt | ||
| readme.md | ||
| test_a2a_registry_lookup.py | ||
| test_acompletion_session_reuse_e2e.py | ||
| test_add_deployment_no_master_key.py | ||
| test_aembedding_session_reuse_e2e.py | ||
| test_anthropic_beta_headers_filtering.py | ||
| test_azure_video_router.py | ||
| test_claude_haiku_4_5_config.py | ||
| test_claude_opus_4_6_config.py | ||
| test_constants.py | ||
| test_container_router.py | ||
| test_cost_calculation_log_level.py | ||
| test_cost_calculator.py | ||
| test_deepseek_model_metadata.py | ||
| test_eager_tiktoken_load.py | ||
| test_exception_exports.py | ||
| test_exception_header_preservation.py | ||
| test_exception_mapping_request_attribute.py | ||
| test_filter_out_litellm_params.py | ||
| test_get_blog_posts.py | ||
| test_gpt_image_cost_calculator.py | ||
| test_groq_streaming_encoding.py | ||
| test_lazy_imports.py | ||
| test_logging.py | ||
| test_lowest_latency_zero_tokens.py | ||
| test_main.py | ||
| test_model_param_helper.py | ||
| test_model_response_normalization.py | ||
| test_nested_drop_params.py | ||
| test_project_tags_pydantic.py | ||
| test_redis.py | ||
| test_register_model_custom_pricing.py | ||
| test_responses_api_bridge_non_stream.py | ||
| test_responses_id_security.py | ||
| test_router_google_genai.py | ||
| test_router_model_cost_isolation.py | ||
| test_router_per_deployment_num_retries.py | ||
| test_router_redis_init.py | ||
| test_router_silent_experiment.py | ||
| test_router.py | ||
| test_service_logger.py | ||
| test_shared_session_integration.py | ||
| test_ssl_verify_unit.py | ||
| test_streaming_connection_cleanup.py | ||
| test_system_message_format_bug.py | ||
| test_utils.py | ||
| test_uuid_helper.py | ||
| test_video_generation.py | ||
| test_xai_responses_auto_routing.py | ||
Testing for litellm/
This directory 1:1 maps the the litellm/ directory, and can only contain mocked tests.
The point of this is to:
- Increase test coverage of
litellm/ - Make it easy for contributors to add tests for the
litellm/package and easily run tests without needing LLM API keys.
File name conventions
litellm/proxy/test_caching_routes.pymaps tolitellm/proxy/caching_routes.pytest_<filename>.pymaps tolitellm/<filename>.py