LLM Gateway /docs

This gateway sits between your apps and many LLM providers. Clients speak the OpenAI API; the gateway resolves the model through routes (v1-style ordered lists), aliases, an auto-router, a fallback chain and a per-provider cooldown, then adapts the request to each upstream (OpenAI, Anthropic, Gemini, Azure).

Quick start

# 1. Generate a downstream key in the dashboard (/dashboard -> API Keys)
# 2. Call the gateway
curl https://YOUR_WORKER.dev/v1/chat/completions \
  -H "Authorization: Bearer AI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o-mini",
    "messages": [{"role":"user","content":"Hello!"}]
  }'

Authentication

Every /v1/* request must carry a downstream key issued by the dashboard:

If the secret AI_API_KEY is set on the Worker it is also accepted as a master key (no expiry, no rate limit, no model allowlist).

Key lifecycle

Endpoints

POST /v1/chat/completions

curl https://YOUR_WORKER.dev/v1/chat/completions \
  -H "Authorization: Bearer AI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"smart","messages":[{"role":"user","content":"Hi"}],"temperature":0.4}'

Response (non-streaming):

{
  "id": "chatcmpl-9f2c...",
  "object": "chat.completion",
  "created": 1712000000,
  "model": "gpt-4o-mini",
  "choices": [{"index":0,"message":{"role":"assistant","content":"Hello!"},
               "finish_reason":"stop"}],
  "usage": {"prompt_tokens":11,"completion_tokens":9,"total_tokens":20}
}

Streaming ("stream": true):

data: {"id":"chatcmpl-1","object":"chat.completion.chunk","model":"gpt-4o-mini",
       "choices":[{"index":0,"delta":{"content":"Hel"},"finish_reason":null}]}

data: {"id":"chatcmpl-1","object":"chat.completion.chunk",
       "choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Response headers: x-gateway-request-id, x-gateway-provider, x-gateway-model, x-gateway-latency-ms, x-gateway-attempts (how many candidates were tried), and x-gateway-degraded: cooldown when every healthy target was cooling down.

GET /v1/models

{
  "object": "list",
  "data": [
    {"id":"gpt-4o-mini","object":"model","created":1712000000,"owned_by":"gateway",
     "aliases":["mini","fast"],"targets":["openai/gpt-4o-mini"]}
  ]
}

Model routing

{
  "id": "smart",
  "aliases": ["gpt-4o", "best"],
  "targets": [
    {"provider": "openai",  "model": "gpt-4o"},
    {"provider": "azure",   "model": "gpt4o-deploy"}
  ],
  "fallbacks": ["cheap"],
  "params": {"temperature": 0.7}
}

Retry policy (v1 fast retry): up to maxAttempts attempts per request (default 4), every candidate tried in turn, including on 400 / 409 / 413 / 422 — providers disagree about geo blocks, unsupported parameters and model names — and the first client error is relayed once every candidate failed, with an x-gateway-attempts header. 401 / 403 put the provider into cooldown and rotate to the next target. 404, 408, 429, 5xx and network errors rotate to the next target after an exponential backoff (250 ms × 2^n, capped at 2 s). Upstream response headers slower than headerTimeoutMs (default 15000 ms) abort the attempt as a timeout — the timer is cleared as soon as the headers arrive, so long streams are never cut off. The last failure is returned as 502 upstream_error (or the original status for client errors).

Errors

All errors use the OpenAI envelope:

{"error":{"message":"...","type":"rate_limit_error","param":null,"code":"rate_limit_exceeded"}}

Dashboard & admin API

/dashboard manages providers, models, keys, settings and usage. It talks to /api/admin/*, protected by a session cookie (POST /api/admin/login with the admin token, or a first-run setup token).

Configuration reference

Open Dashboard