diff --git a/docs/my-website/docs/benchmarks.md b/docs/my-website/docs/benchmarks.md index 5ed2263d05..6dcb8b8a1a 100644 --- a/docs/my-website/docs/benchmarks.md +++ b/docs/my-website/docs/benchmarks.md @@ -5,6 +5,55 @@ import Image from '@theme/IdealImage'; Benchmarks for LiteLLM Gateway (Proxy Server) tested against a fake OpenAI endpoint. + +LiteLLM Gateway has **8ms P95 latency** at 1k RPS (See benchmarks [here](#4-instances)) + +## Machine Spec used for testing + +Each machine deploying LiteLLM had the following specs: + +- 4 CPU +- 8GB RAM + +## Configuration + +- Database: PostgreSQL +- Redis: Not used + + +### 2 Instance LiteLLM Proxy + +In these tests the baseline latency characteristics are measured against a fake-openai-endpoint. + +#### Performance Metrics + +| **Type** | **Name** | **Median (ms)** | **95%ile (ms)** | **99%ile (ms)** | **Average (ms)** | **Current RPS** | +| --- | --- | --- | --- | --- | --- | --- | +| POST | /chat/completions | 200 | 630 | 1200 | 262.46 | 1035.7 | +| Custom | LiteLLM Overhead Duration (ms) | 12 | 29 | 43 | 14.74 | 1035.7 | +| | Aggregated | 100 | 430 | 930 | 138.6 | 2071.4 | + + + + + + +### 4 Instances + +| **Type** | **Name** | **Median (ms)** | **95%ile (ms)** | **99%ile (ms)** | **Average (ms)** | **Current RPS** | +| --- | --- | --- | --- | --- | --- | --- | +| POST | /chat/completions | 100 | 150 | 240 | 111.73 | 1170 | +| Custom | LiteLLM Overhead Duration (ms) | 2 | 8 | 13 | 3.32 | 1170 | +| | Aggregated | 77 | 130 | 180 | 57.53 | 2340 | + +#### Key Findings +- Doubling from 2 to 4 LiteLLM instances halves median latency: 200 ms → 100 ms. +- High-percentile latencies drop significantly: P95 630 ms → 150 ms, P99 1,200 ms → 240 ms. +- Setting workers equal to CPU count gives optimal performance. + + ## Setting Up Benchmarking with Network Mock The fastest way to benchmark proxy overhead is using `network_mock` mode. This intercepts outbound requests at the httpx transport layer and returns canned responses, no need for setting up a mock provider. @@ -61,38 +110,6 @@ model_list: api_key: "test" ``` -### 2 Instance LiteLLM Proxy - -In these tests the baseline latency characteristics are measured against a fake-openai-endpoint. - -#### Performance Metrics - -| **Type** | **Name** | **Median (ms)** | **95%ile (ms)** | **99%ile (ms)** | **Average (ms)** | **Current RPS** | -| --- | --- | --- | --- | --- | --- | --- | -| POST | /chat/completions | 200 | 630 | 1200 | 262.46 | 1035.7 | -| Custom | LiteLLM Overhead Duration (ms) | 12 | 29 | 43 | 14.74 | 1035.7 | -| | Aggregated | 100 | 430 | 930 | 138.6 | 2071.4 | - - - - - - -### 4 Instances - -| **Type** | **Name** | **Median (ms)** | **95%ile (ms)** | **99%ile (ms)** | **Average (ms)** | **Current RPS** | -| --- | --- | --- | --- | --- | --- | --- | -| POST | /chat/completions | 100 | 150 | 240 | 111.73 | 1170 | -| Custom | LiteLLM Overhead Duration (ms) | 2 | 8 | 13 | 3.32 | 1170 | -| | Aggregated | 77 | 130 | 180 | 57.53 | 2340 | - -#### Key Findings -- Doubling from 2 to 4 LiteLLM instances halves median latency: 200 ms → 100 ms. -- High-percentile latencies drop significantly: P95 630 ms → 150 ms, P99 1,200 ms → 240 ms. -- Setting workers equal to CPU count gives optimal performance. - ## `/realtime` API Benchmarks End-to-end latency benchmarks for the `/realtime` endpoint tested against a fake realtime endpoint. @@ -115,17 +132,6 @@ End-to-end latency benchmarks for the `/realtime` endpoint tested against a fake | **System** | 4 vCPUs, 8 GB RAM, 4 workers, 4 instances | | **Database** | PostgreSQL (Redis unused) | -## Machine Spec used for testing - -Each machine deploying LiteLLM had the following specs: - -- 4 CPU -- 8GB RAM - -## Configuration - -- Database: PostgreSQL -- Redis: Not used ## Infrastructure Recommendations diff --git a/docs/my-website/sidebars.js b/docs/my-website/sidebars.js index b1ec952544..00f6a80e86 100644 --- a/docs/my-website/sidebars.js +++ b/docs/my-website/sidebars.js @@ -1059,16 +1059,7 @@ const sidebars = { "proxy/health_check_routing" ], }, - { - type: "category", - label: "Load Testing", - items: [ - "benchmarks", - "load_test_advanced", - "load_test_sdk", - "load_test_rpm", - ] - }, + "benchmarks", { type: "category", label: "Contributing", @@ -1097,6 +1088,9 @@ const sidebars = { "data_retention", "proxy/security_encryption_faq", "migration_policy", + "load_test_advanced", + "load_test_sdk", + "load_test_rpm", { type: "category", label: "❤️ 🚅 Projects built on LiteLLM",