LiteLLM

Overview

Universal LLM API: Provides a proxy server as a single, standardized way to call over 100 Large Language Models (LLMs) from providers like OpenAI, Anthropic, Google, Cohere, and more.
OpenAI-Compatible Format: It unifies different LLM APIs by using the familiar OpenAI chat.completions input/output format.
Easy Model Switching: Allows developers to switch between different models (e.g., from GPT-4 to Claude 3) by changing just a single line of code (the model string).
Centralized Management: Enables centralized API key management, cost tracking, user-based spending limits, and request logging across all models.
Enterprise-Grade Reliability: Supports features like automatic retries, fallbacks (if one model fails, it tries another), and load balancing across multiple API keys or model deployments.
Consistent Outputs: Guarantees that the output object from any model call will have a consistent structure, making it reliable to parse results in your application.

Up and Running

Setup Database

podman exec -ti -u postgres postgres psql
  CREATE USER litellm PASSWORD 'PUT_LITELLM_DB_PASSWORD_HERE';
  CREATE DATABASE litellm OWNER litellm;
  \q

Get Google's AIStudio API Key (https://ai.google.dev/gemini-api/docs/rate-limits#current-rate-limits)

Login to https://aistudio.google.com
Click Get API Key
Click Create API Key
NOTE: You may need to create a Google project
Click Create API key
Create environment variable with GOOGLE API Key provided
```
export GOOGLE_API_KEY="PUT_GOOGLE_API_KEY_HERE"
```

Create configuration file name $LLM_STACK_DIR/litellm-config.yml with the following content:

model_list:
- model_name: my-llama3.2:1b
  litellm_params:
    model: ollama/llama3.2:1b
    api_base: http://ollama:11434
    input_cost_per_token: 0
    output_cost_per_token: 0
    temperature: 0.2
- model_name: my-bge-m3:567m
  litellm_params:
    model: ollama/bge-m3:567m
    api_base: http://ollama:11434
    input_cost_per_token: 0
    output_cost_per_token: 0
- model_name: my-gemini-2.5-pro
  litellm_params:
    model: gemini/gemini-2.5-pro
    api_key: "os.environ/GOOGLE_API_KEY"
    input_cost_per_token: 0.00000125
    output_cost_per_token: 0.00001
- model_name: my-gemini-2.5-flash
  litellm_params:
    model: gemini/gemini-2.5-flash
    api_key: "os.environ/GOOGLE_API_KEY"
    input_cost_per_token: 0.0000003
    output_cost_per_token: 0.0000025

general_settings:
  enforce_user_param: False

Launch LiteLLM:

podman run -d \
  --name litellm \
  --network llm-stack \
  -e STORE_MODEL_IN_DB=True \
  -e LITELLM_MASTER_KEY="PUT_LITELLM_KEY_HERE" \
  -e DATABASE_URL="postgresql://litellm:PUT_LITELLM_DB_PASSWORD_HERE@postgres:5432/litellm" \
  -e GOOGLE_API_KEY="$GOOGLE_API_KEY" \
  -v "$LLM_STACK_DIR/litellm-config.yml":/etc/litellm/litellm-config.yml \
  -e PORT="4000" \
  -p 4000:4000 \
  ghcr.io/berriai/litellm:v1.74.0-stable --config /etc/litellm/litellm-config.yml

podman logs -f litellm

# Open the firewall so port 4000 for external access
sudo firewall-cmd --permanent --add-port=4000/tcp
sudo firewall-cmd --reload

For HA, you can have multiple instances of LiteLLM, but you'll need to configure it with a Redis instance to control RPM/TPM

Usage

# Inspect database
podman exec -ti -u postgres postgres psql litellm
  \d
  \q

# Get a list of available models
curl http://localhost:4000/v1/models -H "Authorization: Bearer PUT_LITELLM_KEY_HERE" | jq

# Call local model
curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer PUT_LITELLM_KEY_HERE" \
  -H "Content-Type: application/json" \
  -d '{
      "model": "my-llama3.2:1b",
      "messages": [
          {
              "role": "system",
              "content": "You are a helpful assistant."
          },
          {
              "role": "user",
              "content": "Hello!"
          }
      ]
  }' | jq

curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer PUT_LITELLM_KEY_HERE" \
  -H "Content-Type: application/json" \
  -d '{
      "model": "my-gemini-2.5-flash",
      "messages": [
          {
              "role": "system",
              "content": "You are a helpful assistant."
          },
          {
              "role": "user",
              "content": "Hello!"
          }
      ]
  }' | jq