diff --git a/AGENTS.md b/AGENTS.md index 8e7b5f2bd2..d72b00f7e1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -94,6 +94,10 @@ LiteLLM supports MCP for agent workflows: - Support for external MCP servers (Zapier, Jira, Linear, etc.) - See `litellm/experimental_mcp_client/` and `litellm/proxy/_experimental/mcp_server/` +## RUNNING SCRIPTS + +Use `poetry run python script.py` to run Python scripts in the project environment (for non-test files). + ## TESTING CONSIDERATIONS 1. **Provider Tests**: Test against real provider APIs when possible diff --git a/CLAUDE.md b/CLAUDE.md index 50bed6e43e..1598432339 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -25,6 +25,9 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co - `poetry run pytest tests/path/to/test_file.py -v` - Run specific test file - `poetry run pytest tests/path/to/test_file.py::test_function -v` - Run specific test +### Running Scripts +- `poetry run python script.py` - Run Python scripts (use for non-test files) + ## Architecture Overview LiteLLM is a unified interface for 100+ LLM providers with two main components: diff --git a/VERTEX_ENV_SETUP.md b/VERTEX_ENV_SETUP.md deleted file mode 100644 index 93a631c82f..0000000000 --- a/VERTEX_ENV_SETUP.md +++ /dev/null @@ -1,261 +0,0 @@ -# Vertex AI Environment Variables Setup Guide - -## Overview - -LiteLLM can load Vertex AI credentials from environment variables instead of storing them in config files. This is more secure and easier to manage for local development. - -## Environment Variables - -LiteLLM looks for these environment variables (in order of precedence): - -### 1. **DEFAULT_VERTEXAI_PROJECT** (Required) -Your GCP project ID that has Vertex AI enabled. - -```bash -export DEFAULT_VERTEXAI_PROJECT="my-gcp-project-id" -``` - -### 2. **DEFAULT_VERTEXAI_LOCATION** (Required) -The region/location for Vertex AI services. - -```bash -export DEFAULT_VERTEXAI_LOCATION="global" -# or -export DEFAULT_VERTEXAI_LOCATION="us-central1" -``` - -Common locations: -- `global` - For Discovery Engine and global services -- `us-central1` - US Central region -- `us-east1` - US East region -- `europe-west1` - Europe West region -- `asia-southeast1` - Asia Southeast region - -### 3. **DEFAULT_GOOGLE_APPLICATION_CREDENTIALS** (Required) -Path to your service account JSON key file. - -```bash -export DEFAULT_GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account-key.json" -``` - -### 4. **GOOGLE_APPLICATION_CREDENTIALS** (Fallback) -Standard Google Cloud environment variable (used as fallback). - -```bash -export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account-key.json" -``` - -## Quick Setup - -### Option 1: Interactive Script - -```bash -chmod +x setup_vertex_env.sh -source setup_vertex_env.sh -``` - -### Option 2: Manual Setup - -1. **Set environment variables** (for current session): - -```bash -export DEFAULT_VERTEXAI_PROJECT="your-project-id" -export DEFAULT_VERTEXAI_LOCATION="global" -export DEFAULT_GOOGLE_APPLICATION_CREDENTIALS="$HOME/.gcp/service-account.json" -export GOOGLE_APPLICATION_CREDENTIALS="$HOME/.gcp/service-account.json" -``` - -2. **Make them persistent** (add to `~/.zshrc` or `~/.bashrc`): - -```bash -echo 'export DEFAULT_VERTEXAI_PROJECT="your-project-id"' >> ~/.zshrc -echo 'export DEFAULT_VERTEXAI_LOCATION="global"' >> ~/.zshrc -echo 'export DEFAULT_GOOGLE_APPLICATION_CREDENTIALS="$HOME/.gcp/service-account.json"' >> ~/.zshrc -echo 'export GOOGLE_APPLICATION_CREDENTIALS="$HOME/.gcp/service-account.json"' >> ~/.zshrc -``` - -3. **Reload your shell**: - -```bash -source ~/.zshrc -``` - -## Service Account Setup - -### 1. Create a Service Account - -```bash -gcloud iam service-accounts create litellm-vertex-sa \ - --display-name="LiteLLM Vertex AI Service Account" -``` - -### 2. Grant Necessary Permissions - -For Discovery Engine (vector stores): -```bash -gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \ - --member="serviceAccount:litellm-vertex-sa@YOUR_PROJECT_ID.iam.gserviceaccount.com" \ - --role="roles/discoveryengine.viewer" - -gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \ - --member="serviceAccount:litellm-vertex-sa@YOUR_PROJECT_ID.iam.gserviceaccount.com" \ - --role="roles/discoveryengine.dataStoreEditor" -``` - -For general Vertex AI: -```bash -gcloud projects add-iam-policy-binding YOUR_PROJECT_ID \ - --member="serviceAccount:litellm-vertex-sa@YOUR_PROJECT_ID.iam.gserviceaccount.com" \ - --role="roles/aiplatform.user" -``` - -### 3. Create and Download Key - -```bash -gcloud iam service-accounts keys create ~/service-account-key.json \ - --iam-account=litellm-vertex-sa@YOUR_PROJECT_ID.iam.gserviceaccount.com -``` - -## Verify Setup - -### Check Environment Variables - -```bash -python3 << 'EOF' -import os -print("✓ Environment Variables:") -print(f" DEFAULT_VERTEXAI_PROJECT: {os.getenv('DEFAULT_VERTEXAI_PROJECT')}") -print(f" DEFAULT_VERTEXAI_LOCATION: {os.getenv('DEFAULT_VERTEXAI_LOCATION')}") -print(f" DEFAULT_GOOGLE_APPLICATION_CREDENTIALS: {os.getenv('DEFAULT_GOOGLE_APPLICATION_CREDENTIALS')}") -print(f" GOOGLE_APPLICATION_CREDENTIALS: {os.getenv('GOOGLE_APPLICATION_CREDENTIALS')}") - -# Check if credentials file exists -creds_path = os.getenv('DEFAULT_GOOGLE_APPLICATION_CREDENTIALS') -if creds_path and os.path.exists(creds_path): - print(f"\n✅ Credentials file found at: {creds_path}") -else: - print(f"\n❌ Credentials file NOT found at: {creds_path}") -EOF -``` - -### Test Authentication - -```bash -python3 << 'EOF' -import os -import json -from google.oauth2 import service_account -from google.auth.transport.requests import Request - -creds_path = os.getenv('DEFAULT_GOOGLE_APPLICATION_CREDENTIALS') -project = os.getenv('DEFAULT_VERTEXAI_PROJECT') - -try: - # Load credentials - credentials = service_account.Credentials.from_service_account_file( - creds_path, - scopes=['https://www.googleapis.com/auth/cloud-platform'] - ) - - # Get access token - credentials.refresh(Request()) - - print("✅ Authentication successful!") - print(f" Project: {project}") - print(f" Service Account: {credentials.service_account_email}") - print(f" Token expiry: {credentials.expiry}") - -except Exception as e: - print(f"❌ Authentication failed: {e}") -EOF -``` - -## Using with Vector Store Passthrough - -Once your environment is set up, the vector store passthrough will work in two ways: - -### 1. **With Vector Store Config** (Priority 1) -If you have a vector store configured with its own credentials in `litellm_params`, those will be used first: - -```yaml -vector_stores: - - vector_store_id: test-store-123 - custom_llm_provider: vertex_ai - litellm_params: - vertex_project: "specific-project" - vertex_location: "us-central1" - vertex_credentials: "{...}" # Inline credentials -``` - -### 2. **Environment Variables Fallback** (Priority 2) -If the vector store doesn't have explicit credentials, it falls back to your environment variables: - -```yaml -vector_stores: - - vector_store_id: test-store-123 - custom_llm_provider: vertex_ai - # No litellm_params - will use DEFAULT_VERTEXAI_PROJECT, DEFAULT_VERTEXAI_LOCATION, etc. -``` - -### 3. **Model Config Fallback** (Priority 3) -If neither above work, it looks for credentials in your model configuration. - -## Troubleshooting - -### "No credentials found" - -Check that all environment variables are set: -```bash -env | grep -E "(DEFAULT_VERTEXAI|GOOGLE_APPLICATION_CREDENTIALS)" -``` - -### "Authentication failed" - -Verify your service account key is valid: -```bash -cat $DEFAULT_GOOGLE_APPLICATION_CREDENTIALS | python3 -m json.tool -``` - -### "Permission denied" - -Ensure your service account has the necessary roles: -```bash -gcloud projects get-iam-policy YOUR_PROJECT_ID \ - --flatten="bindings[].members" \ - --filter="bindings.members:serviceAccount:litellm-vertex-sa@*" -``` - -### Different Credentials for Different Projects - -If you need to use different credentials for different vector stores, configure them explicitly in the vector store config rather than relying on environment variables. - -## Start LiteLLM Proxy - -Once your environment is configured: - -```bash -# Start the proxy (it will automatically load env vars) -litellm --config proxy_server_config.yaml - -# Or with debug logging -export LITELLM_LOG=DEBUG -litellm --config proxy_server_config.yaml -``` - -You should see logs like: -``` -Vertex: Loading vertex credentials from /path/to/service-account.json -Found credentials for vertex_ai_default -``` - -## Test the Endpoint - -```bash -curl -X POST http://0.0.0.0:4000/vertex_ai/discovery/v1/projects/fake-project/locations/global/dataStores/test-store-123/servingConfigs/default_config:search \ - -H 'Authorization: Bearer YOUR_LITELLM_API_KEY' \ - -H 'Content-Type: application/json' \ - -d '{"query": "test query"}' -``` - -The proxy will use your environment credentials to make the request to Vertex AI! - diff --git a/setup_vertex_env.sh b/setup_vertex_env.sh deleted file mode 100644 index e66a0e1d85..0000000000 --- a/setup_vertex_env.sh +++ /dev/null @@ -1,63 +0,0 @@ -#!/bin/bash -# Setup script for Vertex AI credentials using environment variables -# -# This script helps configure your environment for Vertex AI Discovery passthrough -# without needing to put credentials in the config.yaml file - -echo "🔧 Setting up Vertex AI credentials from environment variables" -echo "================================================================" -echo "" - -# Prompt for GCP Project ID -read -p "Enter your GCP Project ID: " PROJECT_ID -export DEFAULT_VERTEXAI_PROJECT="$PROJECT_ID" -echo "✓ Set DEFAULT_VERTEXAI_PROJECT=$PROJECT_ID" - -# Prompt for Vertex Location -echo "" -echo "Common locations: us-central1, global, us-east1, europe-west1" -read -p "Enter your Vertex AI location (default: global): " LOCATION -LOCATION=${LOCATION:-global} -export DEFAULT_VERTEXAI_LOCATION="$LOCATION" -echo "✓ Set DEFAULT_VERTEXAI_LOCATION=$LOCATION" - -# Prompt for credentials file path -echo "" -read -p "Enter the path to your service account JSON file: " CREDS_PATH - -# Expand ~ to home directory if needed -CREDS_PATH="${CREDS_PATH/#\~/$HOME}" - -# Check if file exists -if [ -f "$CREDS_PATH" ]; then - export DEFAULT_GOOGLE_APPLICATION_CREDENTIALS="$CREDS_PATH" - echo "✓ Set DEFAULT_GOOGLE_APPLICATION_CREDENTIALS=$CREDS_PATH" - - # Also set standard GOOGLE_APPLICATION_CREDENTIALS for fallback - export GOOGLE_APPLICATION_CREDENTIALS="$CREDS_PATH" - echo "✓ Set GOOGLE_APPLICATION_CREDENTIALS=$CREDS_PATH" -else - echo "⚠️ Warning: File not found at $CREDS_PATH" - echo " Please make sure the path is correct" -fi - -echo "" -echo "================================================================" -echo "✅ Environment variables configured!" -echo "" -echo "Current settings:" -echo " DEFAULT_VERTEXAI_PROJECT=$DEFAULT_VERTEXAI_PROJECT" -echo " DEFAULT_VERTEXAI_LOCATION=$DEFAULT_VERTEXAI_LOCATION" -echo " DEFAULT_GOOGLE_APPLICATION_CREDENTIALS=$DEFAULT_GOOGLE_APPLICATION_CREDENTIALS" -echo "" -echo "To make these persistent, add the following to your ~/.zshrc or ~/.bashrc:" -echo "" -echo " export DEFAULT_VERTEXAI_PROJECT=\"$PROJECT_ID\"" -echo " export DEFAULT_VERTEXAI_LOCATION=\"$LOCATION\"" -echo " export DEFAULT_GOOGLE_APPLICATION_CREDENTIALS=\"$CREDS_PATH\"" -echo " export GOOGLE_APPLICATION_CREDENTIALS=\"$CREDS_PATH\"" -echo "" -echo "You can now start LiteLLM proxy with:" -echo " litellm --config proxy_server_config.yaml" -echo "" -