ai-workspace-services/litellm
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: refactor benchmarking docs to be clearer

Browse Source
This commit is contained in:
Krrish Dholakia 2026-04-11 17:30:09 -07:00
parent 2c786ca2e6
commit 12bca649fc
2 changed files with 53 additions and 53 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

92
docs/my-website/docs/benchmarks.md
View File

@ -5,6 +5,55 @@ import Image from '@theme/IdealImage';
Benchmarks for LiteLLM Gateway (Proxy Server) tested against a fake OpenAI endpoint.
LiteLLM Gateway has **8ms P95 latency** at 1k RPS (See benchmarks [here](#4-instances))
## Machine Spec used for testing
Each machine deploying LiteLLM had the following specs:
- 4 CPU
- 8GB RAM
## Configuration
- Database: PostgreSQL
- Redis: Not used
### 2 Instance LiteLLM Proxy
In these tests the baseline latency characteristics are measured against a fake-openai-endpoint.
#### Performance Metrics
| **Type** | **Name** | **Median (ms)** | **95%ile (ms)** | **99%ile (ms)** | **Average (ms)** | **Current RPS** |
| --- | --- | --- | --- | --- | --- | --- |
| POST | /chat/completions | 200 | 630 | 1200 | 262.46 | 1035.7 |
| Custom | LiteLLM Overhead Duration (ms) | 12 | 29 | 43 | 14.74 | 1035.7 |
| | Aggregated | 100 | 430 | 930 | 138.6 | 2071.4 |
<!-- <Image img={require('../img/1_instance_proxy.png')} /> -->
<!-- ## **Horizontal Scaling - 10K RPS**
<Image img={require('../img/instances_vs_rps.png')} /> -->
### 4 Instances
| **Type** | **Name** | **Median (ms)** | **95%ile (ms)** | **99%ile (ms)** | **Average (ms)** | **Current RPS** |
| --- | --- | --- | --- | --- | --- | --- |
| POST | /chat/completions | 100 | 150 | 240 | 111.73 | 1170 |
| Custom | LiteLLM Overhead Duration (ms) | 2 | 8 | 13 | 3.32 | 1170 |
| | Aggregated | 77 | 130 | 180 | 57.53 | 2340 |
#### Key Findings
- Doubling from 2 to 4 LiteLLM instances halves median latency: 200 ms → 100 ms.
- High-percentile latencies drop significantly: P95 630 ms → 150 ms, P99 1,200 ms → 240 ms.
- Setting workers equal to CPU count gives optimal performance.
## Setting Up Benchmarking with Network Mock
The fastest way to benchmark proxy overhead is using `network_mock` mode. This intercepts outbound requests at the httpx transport layer and returns canned responses, no need for setting up a mock provider.
@ -61,38 +110,6 @@ model_list:
api_key: "test"
```
### 2 Instance LiteLLM Proxy
In these tests the baseline latency characteristics are measured against a fake-openai-endpoint.
#### Performance Metrics
| **Type** | **Name** | **Median (ms)** | **95%ile (ms)** | **99%ile (ms)** | **Average (ms)** | **Current RPS** |
| --- | --- | --- | --- | --- | --- | --- |
| POST | /chat/completions | 200 | 630 | 1200 | 262.46 | 1035.7 |
| Custom | LiteLLM Overhead Duration (ms) | 12 | 29 | 43 | 14.74 | 1035.7 |
| | Aggregated | 100 | 430 | 930 | 138.6 | 2071.4 |
<!-- <Image img={require('../img/1_instance_proxy.png')} /> -->
<!-- ## **Horizontal Scaling - 10K RPS**
<Image img={require('../img/instances_vs_rps.png')} /> -->
### 4 Instances
| **Type** | **Name** | **Median (ms)** | **95%ile (ms)** | **99%ile (ms)** | **Average (ms)** | **Current RPS** |
| --- | --- | --- | --- | --- | --- | --- |
| POST | /chat/completions | 100 | 150 | 240 | 111.73 | 1170 |
| Custom | LiteLLM Overhead Duration (ms) | 2 | 8 | 13 | 3.32 | 1170 |
| | Aggregated | 77 | 130 | 180 | 57.53 | 2340 |
#### Key Findings
- Doubling from 2 to 4 LiteLLM instances halves median latency: 200 ms → 100 ms.
- High-percentile latencies drop significantly: P95 630 ms → 150 ms, P99 1,200 ms → 240 ms.
- Setting workers equal to CPU count gives optimal performance.
## `/realtime` API Benchmarks
End-to-end latency benchmarks for the `/realtime` endpoint tested against a fake realtime endpoint.
@ -115,17 +132,6 @@ End-to-end latency benchmarks for the `/realtime` endpoint tested against a fake
| **System** | 4 vCPUs, 8 GB RAM, 4 workers, 4 instances |
| **Database** | PostgreSQL (Redis unused) |
## Machine Spec used for testing
Each machine deploying LiteLLM had the following specs:
- 4 CPU
- 8GB RAM
## Configuration
- Database: PostgreSQL
- Redis: Not used
## Infrastructure Recommendations

14
docs/my-website/sidebars.js
View File

@ -1059,16 +1059,7 @@ const sidebars = {
"proxy/health_check_routing"
],
},
{
type: "category",
label: "Load Testing",
items: [
"benchmarks",
"load_test_advanced",
"load_test_sdk",
"load_test_rpm",
]
},
"benchmarks",
{
type: "category",
label: "Contributing",
@ -1097,6 +1088,9 @@ const sidebars = {
"data_retention",
"proxy/security_encryption_faq",
"migration_policy",
"load_test_advanced",
"load_test_sdk",
"load_test_rpm",
{
type: "category",
label: "❤️ 🚅 Projects built on LiteLLM",