Ship a Claude backup in 20 minutes with Ollama and LiteLLM

You will set up a local Claude backup that your app can call the same way it calls a cloud model. First, install Ollama (a local LLM runner). Then pull an open model and verify it can answer a basic prompt.

Install Ollama on Linux:

bash
curl -fsSL https://ollama.com/install.sh | sh

Pull and run an open model. I use Llama 3.1 8B for a fast CPU baseline. On GPU it is snappy.

bash
ollama pull llama3.1:8b
ollama run llama3.1:8b -p "Write one sentence on why backups matter."

You should see a short sentence back. If you prefer a raw HTTP check, hit the local API:

bash
curl http://localhost:11434/api/generate \
  -H 'Content-Type: application/json' \
  -d '{"model":"llama3.1:8b","prompt":"Say hi in 3 words"}'

That confirms the local model is alive. Next you will add LiteLLM (an open-source multi-LLM proxy) so your app talks to one endpoint and falls back to the local model when a cloud model goes missing. Stop here if you only wanted a quick local safety net. The full guide adds a proxy, cross-platform setup, and real fallbacks.

You will set up a local Claude backup that your app can call the same way it calls a cloud model. First, install Ollama (a local LLM runner). Then pull an open model and verify it can answer a basic prompt.

Install Ollama on Linux:

bash
curl -fsSL https://ollama.com/install.sh | sh

Pull and run an open model. I use Llama 3.1 8B for a fast CPU baseline. On GPU it is snappy.

bash
ollama pull llama3.1:8b
ollama run llama3.1:8b -p "Write one sentence on why backups matter."

You should see a short sentence back. If you prefer a raw HTTP check, hit the local API:

bash
curl http://localhost:11434/api/generate \
  -H 'Content-Type: application/json' \
  -d '{"model":"llama3.1:8b","prompt":"Say hi in 3 words"}'

That confirms the local model is alive. Next you will add LiteLLM (an open-source multi-LLM proxy) so your app talks to one endpoint and falls back to the local model when a cloud model goes missing. Stop here if you only wanted a quick local safety net. The full guide adds a proxy, cross-platform setup, and real fallbacks.

Setup

We will set up two free pieces:

• Ollama (ollama/ollama) to run open models locally.
• LiteLLM (BerriAI/litellm), an open-source multi-LLM proxy with an OpenAI-style API.

macOS

bash
# Ollama
brew install ollama
ollama serve &

# Pull models
ollama pull llama3.1:8b
ollama pull mistral:7b

# LiteLLM proxy
python3 -m venv ~/.venvs/litellm
~/.venvs/litellm/bin/pip install "litellm[proxy]"

Linux

bash
# Ollama
curl -fsSL https://ollama.com/install.sh | sh
sudo systemctl enable ollama
sudo systemctl start ollama

# Pull models
ollama pull llama3.1:8b
ollama pull mistral:7b

# LiteLLM proxy
python3 -m venv ~/.venvs/litellm
~/.venvs/litellm/bin/pip install "litellm[proxy]"

Windows 11

bash
# Ollama
winget install Ollama.Ollama
# Start Ollama from Start Menu, or run:
& "$Env:ProgramFiles\Ollama\ollama.exe" serve

# Pull models (PowerShell)
ollama pull llama3.1:8b
ollama pull mistral:7b

# LiteLLM proxy
py -m venv %USERPROFILE%\litellm-venv
%USERPROFILE%\litellm-venv\Scripts\pip install "litellm[proxy]"

LiteLLM proxy config with local fallback

Create config.yaml next to your virtualenv. This routes a single alias called safe-chat to a primary local model and falls back to a second local model.

yaml
model_list:
  - model_name: local-primary
    litellm_params:
      model: ollama/llama3.1:8b
      api_base: "http://localhost:11434"
      api_key: "ollama"
  - model_name: local-fallback
    litellm_params:
      model: ollama/mistral:7b
      api_base: "http://localhost:11434"
      api_key: "ollama"
router_settings:
  enable_fallback: true
  fallbacks:
    - "local-primary->local-fallback"

model_alias_map:
  safe-chat: local-primary

Start the proxy:

bash
~/.venvs/litellm/bin/litellm --port 4000 --config config.yaml

Your app now talks to http://localhost:4000 with the OpenAI Chat Completions API and gets automatic fallback.

Verify

Verify Ollama responds:

bash
curl http://localhost:11434/api/generate \
  -H 'Content-Type: application/json' \
  -d '{"model":"llama3.1:8b","prompt":"Name 2 colors"}'

Expected snippet:

json
{
  "model": "llama3.1:8b",
  "created_at": "2026-06-13T12:00:00Z",
  "response": "Blue and green.",
  "done": true
}

Verify LiteLLM proxy with OpenAI-style call:

bash
curl http://localhost:4000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer test-key" \
  -d '{
    "model": "safe-chat",
    "messages": [{"role": "user", "content": "Reply with the word BACKUP only"}],
    "temperature": 0
  }'

Expected snippet:

json
{
  "object": "chat.completion",
  "model": "safe-chat",
  "choices": [
    {"index": 0, "message": {"role": "assistant", "content": "BACKUP"}}
  ]
}

To test fallback, stop the primary model temporarily by renaming it in the config or using a non-existent tag, then call again. You should still get a response from the fallback model.

Configuration tips

• Control context and output size. In each request to Ollama, set options to match your app:

bash
curl http://localhost:11434/api/chat \
  -H 'Content-Type: application/json' \
  -d '{
    "model":"llama3.1:8b",
    "messages":[{"role":"user","content":"Summarize this in 2 bullets"}],
    "options": {"num_ctx": 8192, "num_predict": 256, "temperature": 0.2}
  }'

• Keep models warm. Add keep_alive to avoid cold starts:

bash
curl http://localhost:11434/api/generate \
  -H 'Content-Type: application/json' \
  -d '{"model":"llama3.1:8b","prompt":"ping","keep_alive":"2h"}'

• Expose the proxy only to your app. Bind to localhost and use a long random token:

bash
~/.venvs/litellm/bin/litellm --host 127.0.0.1 --port 4000 --config config.yaml --api_key "sk-local-9c77a2c1"

• Add a cloud primary later without breaking clients. Add a new model to config.yaml, update fallbacks to new-primary->local-primary. Clients still hit model alias safe-chat.

• GPU tuning. On NVIDIA boxes, Ollama detects CUDA. If RAM is tight, use smaller tags like mistral:7b or set num_ctx to 4096.

Troubleshooting

• Port 11434 already in use

- Symptom: Ollama fails to start or curl to 11434 hits another service. - Fix: stop the other service or change Ollama bind address: export OLLAMA_HOST=127.0.0.1:11500 && ollama serve and update api_base in config.yaml.

• Model pull is slow or fails

- Symptom: ollama pull llama3.1:8b hangs or errors on disk. - Fix: check space with df -h. Remove old models ollama list then ollama rm <model>. Use a closer mirror network or retry later.

• GPU not used, performance is low

- Symptom: CPU at 100 percent, GPU idle. - Fix: install recent NVIDIA driver and CUDA runtime per your distro, then restart Ollama. If still CPU only, it will run, just slower. Drop to a smaller model like mistral:7b.

When it beats Claude

• You need answers during a provider freeze or policy change. The local model keeps your app online with no code change.
• You want to red team risky prompts privately. Run them locally without sending data to a cloud.
• You need predictable costs in QA. Local inference cost is fixed after setup, so you can hammer test suites.
• You ship to air gapped or high control environments. Local models satisfy that constraint with zero external calls.

Sources

-