Providers
No providers yet.
Categories
No categories yet.
| Provider | Endpoint | Key | Status | |
|---|---|---|---|---|
adapt |
Local providers found
Reachable on this machine. Review and add the ones you want — nothing is wired automatically.
No providers connected
Detect a local provider (Ollama, LocalAI, LM Studio, …), or add Copilot / an OpenAI-compatible provider.
Order = failover order. Same-provider members cascade on throttle; cross-provider members fail over.
No categories
Create one, then pin real models (from any provider) into an ordered failover chain.
Tip: open a provider's Models first so they appear in the picker.
No catalog yet — click Refresh (or Test on Providers).
Categories — virtual models that fail over
No models yet
Add a provider and click Refresh / Test, or create a category.
Applies to the next key you mint (top-right). Blank/0 = unrestricted. Editable per key after minting.
| Project | Name | Token | Limits | Status | |
|---|---|---|---|---|---|
|
No keys yet
Mint a key for a project (top-right). Usage is attributed to project + key. Use the token as the Bearer / API key in your CLI.
By project & key
| Project | Key | Requests | Tokens (in/out) | Cost |
|---|---|---|---|---|
No usage recorded yet.
Throttling & fallback
| When | Requested | Chain (attempts → served) |
|---|---|---|
|
·429 |
No failover events yet. They appear when a category retries past a failed/throttled target.
Auth & storage
~/.llmgw/ (keys.json & secrets.json, 0600).What llm-gateway is
A small, self-hosted proxy that sits between your coding CLIs and every model provider you use. Point Claude Code, Codex, or any OpenAI/Anthropic-compatible tool at this one endpoint, and it routes each request to a real upstream — automatically failing over to the next when a provider throttles or errors. That's how you spread work across subscriptions instead of burning through one.
The mental model
Claude Code, Codex, Copilot CLI, scripts
One base URL + one key. Resolves the model, does failover, records usage.
Copilot, Bedrock, OpenAI-compatible, Ollama…
Three things to know
Providers are your upstreams — an id + type + key. Keys never touch the config file; they live in a 0600 secrets store. Manage them on Providers.
Categories are the model names your CLIs actually request. Each is an ordered failover chain of pinned provider/model targets — e.g. a smart category that tries Claude first, then a local model. Build them on Categories.
Keys are per-project bearer tokens you mint here. Every request is attributed to a project + key so Usage can show cost and failover behaviour per project.
Quickstart
Get a CLI talking to the gateway in three steps. Your gateway base URL is .
Mint a project key
On Keys, enter a project and key name, then Mint key. Copy the llmgw_… token — it's shown once.
Point your CLI at the gateway
Set the base URL + token as environment variables, then use a category or provider/model as the model. Full per-CLI recipes are under Guides → Connect a CLI.
Connect a coding CLI
The gateway speaks both the OpenAI and Anthropic wire formats, so most tools just need two environment variables: the base URL and your minted key. Use a category (e.g. smart) or a provider/model as the model name.
OpenAI-style — Codex, OpenAI SDKs, most tools
Anthropic-style — Claude Code / claude CLI
Claude Code reads ANTHROPIC_BASE_URL + ANTHROPIC_AUTH_TOKEN; requests hit /v1/messages natively.
Verify the connection
A JSON list of models means you're wired up. Getting 401? See Troubleshooting.
github_copilot here as a provider (Settings → device login) to lend its models to your other CLIs — the reason this gateway exists.Add a provider
A provider is one upstream you can route to. Open Providers → + Add provider, pick a preset (which pre-fills the base URL) or a type, then save. Hit Test afterwards to confirm the gateway can reach it and list models.
/v1; a preset is a starting point, not a guarantee. Confirm it against the provider's current docs and your plan/region before relying on it.Provider types
openai_compatible — base URL + API key. Works with any OpenAI drop-in (OpenRouter, Groq, Together, Fireworks, and the rest).
anthropic — native Anthropic Messages upstream (https://api.anthropic.com) + key.
bedrock — region + a Bedrock API key; see Amazon Bedrock.
github_copilot — no key; sign in with a device code on Settings, then it lends its models to your CLIs.
ollama — local & keyless; see Ollama.
~/.llmgw/secrets.json (0600), keyed by provider id — never written to the committed config file.Well-known base URLs
| Provider | Base URL |
|---|---|
|
Amazon Bedrock
Bedrock now exposes an OpenAI-compatible endpoint that accepts a plain bearer token, so this gateway reaches every Bedrock model over ordinary HTTP — no boto3, no SigV4 signing, no AWS SDK in the image.
Create a Bedrock API key
In the AWS console: Amazon Bedrock → API keys → Generate. You get a long-lived AWS_BEARER_TOKEN_BEDROCK. Make sure the models you want are enabled in that region under Model access.
Add the provider
On Providers, add one of type bedrock, set the region (e.g. us-east-1), and paste the API key. The endpoint is templated to https://bedrock-runtime.<region>.amazonaws.com/v1.
Request Bedrock models
Click Test / Models to enumerate, then use ids like us.anthropic.claude-sonnet-4-6 or openai.gpt-oss-120b — directly, or pinned into a category.
bedrock-mantle host? Set the drawer's Base URL (advanced override) and it wins over the region template.Ollama (local models)
Ollama runs models on your own machine — free and keyless. The gateway auto-detects it and speaks its native /api/chat, including tool-calling.
Auto-detect
Click Detect local on Providers. It probes the usual addresses — Ollama and OpenAI-compatible servers (LocalAI, LM Studio, vLLM, …) — and, importantly when the gateway runs in Docker, host.docker.internal, surfacing whatever answers so you can add it with one click.
Or add it manually
Add a provider of type ollama and set a custom base URL if your daemon lives elsewhere. No key required.
GitHub Copilot as a provider
If you have a Copilot subscription, the gateway can proxy its models to your other CLIs — no API key, just a one-time device login.
Sign in
On Settings, click Sign in (device code), open the shown URL and enter the code. The OAuth token is cached under ~/.llmgw/cache/ and refreshed automatically.
Use its models
Once signed in, copilot/… models appear in Models and can be pinned into categories like any other provider.
allow_copilot_proxy / LLMGW_EXPERIMENTAL_COPILOT_PROVIDER=1. Respect GitHub's terms for your subscription.gpt-5.5, gpt-5.4-mini, gpt-5.3-codex) and reject /chat/completions. The gateway can translate for them — but only when you opt in. See API adaptation; off by default your CLI gets the provider's real error.API adaptation experimental
Some models speak only one OpenAI-family API. GPT-5.x on Copilot (gpt-5.5, gpt-5.4-mini, gpt-5.3-codex) is reachable only via the Responses API and rejects /chat/completions. When you opt in, the gateway translates your Chat Completions request to the model's native endpoint and translates the reply back — so your CLI keeps speaking one API.
Enable per provider
Set force_api_support: true on a provider in config.yaml. Every model that provider serves becomes adaptation-eligible, but only models whose catalog says they need it are actually translated — gpt-4o, Claude, etc. are untouched.
Or per request
Send "force_api_support": true in the request body. A request value overrides the provider config (send false to force it off). The gateway strips the field before calling upstream.
How the target is chosen
Deterministically from the model's published supported_endpoints in the catalog — never trial-and-error. A Responses-only model gets Chat→Responses; a reasoning model that requires max_completion_tokens gets max_tokens renamed.
X-LLMGW-Adapted header (e.g. chat->responses). If you set the flag on the request, the body also echoes a no-op forced_support object; config-triggered adaptation stays header-only so strict clients aren't surprised.max_tokens is consumed by hidden reasoning tokens and can return empty with finish_reason: length. Give them ≥ 200.Categories & failover
A category is a named, ordered chain of pinned provider/model targets. Your CLIs request the category name; the gateway tries each target in order and advances to the next on a throttle or error — before the first streamed byte, so failover is invisible to the client. This is the mechanism for juggling subscription limits.
Example
smart:
1. copilot / claude-sonnet-4.5 ← try premium first
2. bedrock / us.anthropic.claude-sonnet-4-6
3. ollama / llama3.1 ← free local fallbackHow resolution works
• A category name (smart) runs its whole chain.
• A provider/model (copilot/gpt-4o-mini) targets exactly that one model.
• Anything else returns 404 with the list of valid names.
Projects & API keys
Mint bearer tokens scoped to a project and a key name. Each request carries one, so the gateway can attribute cost and failover behaviour back to the project that made it.
• Mint on Keys — the llmgw_… token is shown once; copy it then.
• Use it as the Bearer token / API key in any CLI (see Connect a CLI).
• Attribution shows up on Usage → By project & key.
• Revoke anytime from the Keys table — the token stops working immediately.
• Governance (optional, per key): an expiry, model/provider allowlists, a requests/minute rate limit and a daily budget, plus an instant disable toggle — set on mint or edit later.
~/.llmgw/keys.json (0600). The admin key (or open-local mode) is separate and always works for this panel.Audio & vision
The gateway proxies audio and routes image requests to capable models. Audio needs an OpenAI-compatible provider that serves it — e.g. a local LocalAI with whisper (speech-to-text) and piper (text-to-speech).
Transcribe (STT)
POST /v1/audio/transcriptions (multipart: file + model). Address the model as provider/model, e.g. localai/whisper-base.
Speak (TTS)
POST /v1/audio/speech with {model, input, voice} — returns audio bytes, e.g. localai/vits-piper-en_US-amy-sherpa.
Vision
Send image parts in a chat message as usual. A multimodal request is routed to a vision-capable target (from the catalog); if the route has none it fails clearly. Copilot vision is unlocked automatically.
API endpoints
Standard surfaces your CLIs already know. All except /health require a bearer key.
| Method | Path | Purpose |
|---|---|---|
| GET | /v1/models | List categories + provider/model names |
| POST | /v1/chat/completions | OpenAI chat (streaming supported) |
| POST | /v1/responses | OpenAI Responses API |
| POST | /v1/messages | Anthropic Messages (native) |
| GET | /health | Liveness (no auth) |
Model naming
The model field in a request is either a category or a fully-qualified target. Models lists every valid value.
Category — a name you defined, e.g. smart or cheap. Runs its failover chain. Best for day-to-day use.
provider/model — a specific pinned model, e.g. copilot/gpt-4o-mini or bedrock/us.anthropic.claude-sonnet-4-6. No failover; goes straight to that upstream.
Real names only — models are listed exactly as the upstream names them; the gateway does not invent aliases.
404 with the list of valid names, so a mistyped model fails fast and legibly.Storage & file layout
All state lives under ~/.llmgw/ (override with LLMGW_STATE_DIR). Secrets are written 0600.
| Path | Contents |
|---|---|
config.yaml | Providers, categories, policies (no keys) — safe to read/commit |
secrets.json | Provider API keys, 0600 |
keys.json | Minted project keys, 0600 |
usage.db | Cost/usage ledger (project + key) |
telemetry.db | Throttle / failover events |
cache/ | Copilot OAuth token cache |
Troubleshooting
The usual suspects and how to clear them.
401 Unauthorized
The bearer token is missing or wrong. Mint one on Keys and set it as OPENAI_API_KEY / ANTHROPIC_AUTH_TOKEN. In open-local mode auth is off entirely.
404 — model not a category or provider/model
The requested model name isn't known. Check exact spelling on Models; use a category name or provider/model.
Provider Test shows 0 models
Usually a wrong base URL, a missing/invalid key, or the upstream not exposing a catalog. Re-check the URL (trailing /v1?) and the key; for Copilot, sign in on Settings.
Requests keep failing over / 429
A provider is throttling. That's the gateway working — check Usage → Throttling & fallback to see which target throttled and add a healthier fallback to the category.
Detect local finds nothing
Ensure the local server is running (e.g. ollama serve, or LocalAI/LM Studio listening). In Docker, start the container with --add-host host.docker.internal:host-gateway so it can reach host services.