LUMEN

Lightweight Unified Model ENdpoint

A universal, self-hostable LLM gateway written in Rust. One OpenAI-compatible endpoint in front of many providers - for chat, embeddings and reranking alike. It is designed to be light, fast and sovereign: a single static binary, zero telemetry, and prompts that are never logged by default.

What’s here

• Getting started - installation, quickstart and configuration basics.
• Chat - completions, streaming, vision and tool calling.
• Embeddings - embeddings, batching and multimodal embeddings.
• Reranking - reranking.
• Operations - token accounting & cost, metrics & dashboards, logging, usage log & multi-tenant metadata, keys, quotas & budgets, resilience tuning, deployment and upgrades.
• Examples - ready-made scenario configs.
• Reference - providers, error codes and the performance baseline.
• Architecture decisions - the ADRs that pin the design.
• Project - backlog and contributing.

The five pillars

Every trade-off is decided in this order:

1. Performance - < 1 ms added latency p99, zero-copy streaming, ~15 MB RAM idle.
2. Sovereignty - zero telemetry, prompts never logged by default, single binary.
3. Robustness - propagated cancellation, backpressure, DB off the request path.
4. Multi-capability - chat + embeddings + rerank are first-class citizens.
5. Token observability - every request of every capability produces a token count: upstream usage when reported, otherwise a local estimate flagged estimated. Never a silent zero. See token accounting & cost.

Want to contribute? Start with the contribution guide.

Installation

LUMEN ships as a single static binary. Pick one of three ways to get it: Docker, a prebuilt binary from a GitHub release, or a build from source.

Docker

docker run -p 8080:8080 \
  -v ./config.toml:/config.toml \
  -e OPENAI_API_KEY=sk-... \
  ghcr.io/qdequele/lumen:latest

The image sets LUMEN_SERVER__HOST=0.0.0.0 for you, so the server binds to all interfaces inside the container. The image is multi-arch: linux/amd64 and linux/arm64.

Prebuilt binary

Static musl binaries for x86_64-unknown-linux-musl and aarch64-unknown-linux-musl are attached to every GitHub release cut from a v* tag, each with a .sha256 checksum file alongside it. Verify a download before unpacking:

shasum -a 256 -c lumen-x86_64-unknown-linux-musl.tar.gz.sha256

From source

Needs a recent stable Rust toolchain (MSRV 1.94, per Cargo.toml and checked in CI against the committed Cargo.lock):

cargo build --release -p server --bin lumen

The binary lands at target/release/lumen. Run it with:

lumen --config config.toml

Validate a config without booting

lumen --check-config [--config <PATH>] validates a config file the same way the server does at boot (parsing, semantic validation and provider registry construction) and exits: 0 if valid, non-zero otherwise. It binds no listener, opens no database, and contacts no provider, so it is safe to run in a CI or deploy pipeline ahead of a real boot:

lumen --check-config --config config.toml

Next

Continue to the Quickstart, or browse the fully commented config.example.toml on GitHub.

Quickstart

Zero to a successful chat + embed + rerank request.

1. Minimal config

The ids below (gpt-4o, text-embedding-3-small, rerank-english) are the same ones used throughout config.example.toml. This minimal file needs an OpenAI key (chat + embeddings) and a Cohere key (rerank).

# config.toml - minimal quickstart config
[[providers]]
name = "openai"
kind = "openai"
api_key_env = "OPENAI_API_KEY"

[[providers.models]]
id = "gpt-4o"
upstream_id = "gpt-4o-2024-08-06"
capabilities = ["chat"]

[[providers.models]]
id = "text-embedding-3-small"
capabilities = ["embed"]

[[providers]]
name = "cohere"
kind = "cohere"
api_key_env = "COHERE_API_KEY"

[[providers.models]]
id = "rerank-english"
upstream_id = "rerank-v3.5"
capabilities = ["rerank"]

2. Run

Docker (the released image; sets LUMEN_SERVER__HOST=0.0.0.0 for you):

docker run -p 8080:8080 \
  -v ./config.toml:/config.toml \
  -e OPENAI_API_KEY=sk-... \
  -e COHERE_API_KEY=... \
  ghcr.io/qdequele/lumen:latest

From source (needs a recent stable Rust toolchain):

export OPENAI_API_KEY=sk-...
export COHERE_API_KEY=...
cargo run -p server -- --config config.toml

By default auth is off, so these requests need no Authorization header. Providers whose API-key env var is unset are only rejected when a request actually routes to them, so a partial set of keys is fine.

If you turn auth on ([auth] enabled = true), bootstrap your first virtual key with lumen keys create --name bootstrap - it runs offline against the auth database, before the server is ever started. See Keys, quotas & budgets.

3. Chat

curl -s http://localhost:8080/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Say hello in one word."}]
  }'

The response is the OpenAI chat completion envelope, including a usage object with the token count.

4. Embeddings

curl -s http://localhost:8080/v1/embeddings \
  -H 'content-type: application/json' \
  -d '{
    "model": "text-embedding-3-small",
    "input": ["the quick brown fox", "a lazy dog"]
  }'

The response carries data[].embedding for each input, plus a usage object.

5. Rerank

curl -s http://localhost:8080/v1/rerank \
  -H 'content-type: application/json' \
  -d '{
    "model": "rerank-english",
    "query": "What is the capital of France?",
    "documents": ["Paris is the capital of France.", "Berlin is in Germany."],
    "top_n": 2
  }'

Results come back sorted by descending relevance_score. documents must be non-empty: an empty list is rejected with LM-2010.

Next steps

• Chat completions
• Embeddings
• Reranking
• Token accounting

Configuration basics

Everything is one TOML file, plus LUMEN_* environment variable overrides that use __ for nesting, e.g. LUMEN_SERVER__PORT=9090. In the TOML file itself, top-level keys must appear before any [table] header. The exhaustively commented reference is config.example.toml on GitHub.

Section tour

log_format - "pretty" (human-readable, default) or "json" (production).

[server] - the HTTP server: host (bind address; use "0.0.0.0" in a container) and port (must not be 0), body_limit (max request body in bytes), first_token_timeout_ms (how long to wait for the upstream’s first sign of life before failing with LM-3011; for streaming this is time to the first SSE frame, for non-streaming it is the whole upstream call), and sse_heartbeat_ms (idle interval after which a : ping SSE comment keeps proxies from reaping a silent stream).

[auth] - virtual keys, hard budgets, quotas and the usage log. Disabled by default, in which case the gateway is an open proxy with no database at all. See Keys & budgets.

[telemetry] - which x-lumen-metadata keys become Prometheus labels on the token counters. See Usage log.

[resilience] - retries, fallbacks, circuit breaker, timeouts and health checks. Every value is the built-in default and the whole section is optional; see config.example.toml for the full set. Details in Resilience.

[[providers]] / [[providers.models]] - one [[providers]] block per upstream (id, upstream_id, capabilities, modalities, costs, per-model fallbacks). See Providers for the full provider matrix and per-provider notes.

[image_fetch] - server-side fetching of remote image URLs for multimodal input. See Multimodal input.

API keys

API keys are never written in the config. A provider references the name of the environment variable that holds its key, via api_key_env.

Hot reload

A SIGHUP, a file watch, or an admin provider-key rotation triggers a reload: the new config is validated, then the provider registry, price table, resilience policy and the runtime-safe [auth] knobs are atomically swapped (the bind address and a few other knobs still need a restart). Details in Deployment.

Viewing the live config over the admin API

GET /admin/config returns the config file the gateway booted from, verbatim. It requires the master key AND auth.enabled = true: the whole /admin/* router is only mounted when auth is on, so on a default deployment (where [auth] is disabled) this route does not exist at all and answers 404 rather than 401.

{ "config": "<raw toml, byte for byte>", "hash": "<64 hex chars, BLAKE3>" }

config is the file’s exact bytes, never a re-serialisation of the merged in-memory config: Config::load overlays LUMEN_* environment variables on top of the file, and showing that merged view would make environment overrides look like file content, and a later write would bake them in permanently. hash is a BLAKE3 content hash of those same bytes, meant to be echoed as If-Match on the PUT /admin/config that applies a new one (ADR 010), so two operators editing at once cannot silently clobber each other.

Applying a new config over the admin API

PUT /admin/config is the highest-privilege route in the gateway: it can repoint any provider’s base_url (or add a new provider entirely) and thereby redirect customer traffic to a different upstream. Master key required, same as every other /admin/* route, and likewise only mounted when auth.enabled = true: with auth off the route is absent and returns 404, not 401.

The If-Match contract:

• Send the submitted document as the raw request body (Content-Type is irrelevant; the body is treated as the TOML file’s new contents).
• Send the hash from a prior GET /admin/config as the If-Match header. A missing If-Match header is rejected with 400 (LM-1001) - the request is malformed like any other missing-required-input case.
• If If-Match does not equal the config file’s current hash (someone else applied a change since you last read it), the request is rejected with 412 (LM-1004) and the file is left untouched. GET /admin/config again to see what changed, then re-apply against the fresh hash.

What happens on a successful apply:

1. The submitted bytes are staged in a temporary file next to the real one (same directory, so the final rename is atomic).
2. The staged file is validated exactly like a hot reload would: parsed, merged with LUMEN_* env vars, and used to build a candidate provider registry. A parse failure or a registry-build failure (e.g. a provider missing a required base_url) is rejected with 400 (LM-1001); the staging file is removed and the real config file is never touched. The error message names the offending field or setting (e.g. "server.first_token_timeout_ms must not be 0") but never the staging file’s own filesystem path - that path is an implementation detail of this route, not something the operator wrote or needs to see, and it is logged server-side instead.
3. Only once validation succeeds is the current file copied to a .bak sibling (e.g. lumen.toml.bak) - one generation of history, enough to revert a bad apply by hand - and the staged file renamed into place.
4. The hot-reload trigger fires, so the new config takes effect without a restart (see Hot reload).

A rejected apply (400 or 412) is guaranteed to leave the config file byte-for-byte unchanged and to leave no temporary file behind. Concurrent PUTs are serialised gateway-side, so two operators racing the same pre-apply hash can never both land: exactly one wins, and the other sees a 412 for a hash that moved out from under it, never a silently corrupted mix of the two documents.

Not every field actually takes effect without a restart. A 204 means the write and the reload trigger both succeeded, not that every field you changed is now live: the restart-only settings named in Hot reload - the bind address, auth.enabled, auth.db_path, and the bounded usage-log channel knobs (usage_channel_capacity, usage_batch_max, usage_flush_ms) - are silently unaffected by a PUT just as they are by SIGHUP, with no separate signal in the response. Check reload.rs’s module documentation (or this page’s Hot reload section) for the authoritative restart-only list before relying on a config change through this route.

Security note: the master key is equivalent to host filesystem access. api_key_env accepts ANY environment variable name, not just ones a provider convention would suggest. Because PUT /admin/config lets a master-key holder add a provider with an attacker-controlled base_url and api_key_env pointed at any variable present in the gateway’s own process environment - LUMEN_MASTER_KEY itself, or any cloud credential the process happens to carry - a single crafted config plus one request to that provider’s model returns the named secret back to the caller as a Bearer token. Before remote config apply existed, reaching this required filesystem write access on the gateway host; with this route, the master key alone is enough. In practice: holding the master key is now equivalent to filesystem write access on the gateway host, plus read access to its entire process environment. Mitigate with a separate master key per gateway, exposing the admin surface on a private network only, and using the read-only config mount opt-out (see Deployment) on any gateway that does not need remote apply.

Validate before you boot

Run lumen --check-config --config config.toml to validate a config file without starting the server. See Installation.

Chat completions

POST /v1/chat/completions speaks the OpenAI request and response format. The model field is one of your configured model ids (the id in a [[providers.models]] block, not necessarily the upstream’s own model name - see Providers for aliasing with upstream_id).

Request

curl -s http://localhost:8080/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Say hello in one word."}]
  }'

Response

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1731000000,
  "model": "gpt-4o",
  "choices": [
    {
      "index": 0,
      "message": { "role": "assistant", "content": "Hello!" },
      "finish_reason": "stop"
    }
  ],
  "usage": { "prompt_tokens": 12, "completion_tokens": 3, "total_tokens": 15 }
}

Unknown fields pass through

Request fields LUMEN does not model as a typed struct field (tools, response_format, provider-specific extensions, …) are preserved verbatim and forwarded to the upstream untouched, rather than stripped. Provider-specific parameters keep working without waiting on a LUMEN release to add them by name.

Verbatim passthrough applies to OpenAI-compatible providers. On the translated kinds (anthropic, google, vertex_ai, bedrock, cohere), response_format, seed, logprobs, top_logprobs, logit_bias and parallel_tool_calls are mapped natively where the upstream supports them and otherwise dropped with a debug log - or rejected up front with LM-1001 when the provider sets strict = true. See the chat-extras matrix in Providers.

Routing and request errors

Full taxonomy in Error codes.

Fallbacks

If the model has a fallbacks list and the primary provider fails, the request fails over automatically. The model that actually served the request (primary or a fallback) is reported in the x-lumen-model-used response header. See Resilience.

Providers

Which provider kinds serve chat and their setup is in Providers.

Streaming

Add "stream": true to a /v1/chat/completions request and the response becomes text/event-stream: a series of data: {...} frames (each a chat.completion.chunk), terminated by a literal data: [DONE].

curl -s http://localhost:8080/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Count to 3."}],
    "stream": true
  }'

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1731000000,"model":"gpt-4o","choices":[{"index":0,"delta":{"role":"assistant","content":"1"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1731000000,"model":"gpt-4o","choices":[{"index":0,"delta":{"content":", 2, 3"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1731000000,"model":"gpt-4o","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Passthrough design

When the upstream already speaks OpenAI-shaped SSE (the OpenAI-family kinds, vllm), the gateway forwards the upstream bytes to the client verbatim - no per-chunk deserialize/re-serialize round trip. Providers that translate a foreign event schema (anthropic, google) build typed chunks instead, which is not zero-copy but is required because they must translate the schema anyway. See ADR 004.

Heartbeats

If the stream goes idle for longer than sse_heartbeat_ms ([server] in config.example.toml), the gateway injects a : ping SSE comment to keep intermediate proxies from reaping the connection as silent.

Commitment: the first content frame, not the open

Retries and fallback stay live past the upstream opening the stream (2xx + headers): after opening, the gateway peeks the first frame before committing to the client. An upstream that opens 200 then errors, or closes, before delivering any content frame is a pre-commit failure - the gateway retries and falls over per the resilience policy (and penalises that provider’s circuit breaker) exactly like an open failure, instead of surfacing a terminal SSE error frame. A silent open (no bytes at all) still fails over via the first_token_timeout_ms bound on the peek. Only once the first content frame is forwarded does the request commit; from that point the guards below own the rest and nothing retries. See ADR 005 (first-frame-peek amendment).

Guards (post-commit)

• No first content frame within first_token_timeout_ms ([server]), across every retry/fallback attempt -> the request fails with LM-3011 (504).
• The upstream connection dies mid-stream (after commit) without a [DONE] terminator, or every link in the fallback chain fails to deliver a content frame at all -> the client receives a terminal SSE error frame carrying LM-3010 (502).

Both are documented in Error codes.

Client disconnect

If the client disconnects mid-stream, the upstream call is aborted and the request’s accounting settles at HTTP 499 (LM-6001, client_cancelled) - never counted as a 5xx internal error. See ADR 006 and Error codes.

Usage in streams

The final chunk carries usage when the upstream reports it there; otherwise the gateway falls back to a local estimate flagged "estimated": true. Every request produces a token count, never a silent zero - see Token accounting.

Vision (image input)

POST /v1/chat/completions accepts OpenAI’s content-parts message shape, so a user message can carry text and image parts in one array. It is opt-in per model: a model only accepts image parts once its config declares the image modality (modalities = ["text", "image"]; the default is ["text"]). GET /v1/models reflects the opt-in back as "modalities": ["text","image"] per model.

{
  "model": "gpt-4o",
  "messages": [{
    "role": "user",
    "content": [
      { "type": "text", "text": "What is this?" },
      { "type": "image_url", "image_url": { "url": "data:image/png;base64,iVBORw0KG..." } },
      { "type": "image_url", "image_url": { "url": "https://example.com/cat.png" } }
    ]
  }]
}

image_url.url is either a data:<media-type>;base64,<payload> inline URI or a remote http(s) URL.

Pre-flight

Sending an image part to a model whose modalities lack "image" is rejected with LM-2003 (400) before any upstream call. The check inspects the whole fallback chain, not just the primary, so a fallback missing the image modality is caught up front too.

Per-provider handling

OpenAI-family kinds, vllm and azure forward image parts verbatim - both data: URIs and remote URLs. anthropic and cohere translate both forms into their own schema (both upstreams fetch a remote URL themselves). google, vertex_ai and bedrock translate only inline data: URIs into their own schema; a remote http(s) URL routed to any of the three is rejected pre-flight with LM-2004 (400), since none of them fetches a URL itself and the gateway never fetches a chat image URL on the caller’s behalf (an SSRF vector it deliberately avoids). Full per-kind table in Providers - Vision.

Provider-native sources

Two provider-native reference forms are recognised in image_url.url, for callers whose images are already uploaded to the provider: an Anthropic Files API reference (anthropic-file:<file_id>) and a Gemini-native reference (gs://bucket/object, or a Gemini Files API URI under https://generativelanguage.googleapis.com/). A reference routed to a model whose primary provider does not match the reference’s own provider is rejected pre-flight with LM-2008 (400) instead of surfacing as a confusing upstream failure. Details, including the gs:// / Developer API caveat, are in Providers - Vision.

Token accounting

Upstream-reported usage is authoritative and already folds in image tokens. When an upstream reports no usage at all, the local estimation fallback counts each image content part with a flat per-image heuristic (85 tokens at "detail": "low", 765 tokens otherwise) rather than counting it as zero, and the response is still flagged "estimated": true. See Token accounting.

Tool calling

/v1/chat/completions accepts OpenAI’s tools request field and returns tool_calls on the response message, for every chat provider: passed through untouched for the OpenAI-family kinds, vllm and azure (via the same unknown-field passthrough that carries tools and tool_choice, see Chat completions), translated to and from the provider’s own schema for anthropic (tool_use content blocks), google/vertex_ai (tools[].functionDeclarations, toolConfig.functionCallingConfig), bedrock (toolConfig/toolUse/toolResult on the Converse API) and cohere (mostly a field rename - OpenAI-shaped tool_calls pass through largely unchanged - with tool_choice collapsing to Cohere’s REQUIRED/NONE strings; forcing one named tool has no v2 equivalent and falls back to auto).

Two-leg flow

1. Request with tools:

{
  "model": "gpt-4o",
  "messages": [{ "role": "user", "content": "What is the weather in Paris?" }],
  "tools": [{
    "type": "function",
    "function": {
      "name": "get_weather",
      "parameters": { "type": "object", "properties": { "city": { "type": "string" } } }
    }
  }]
}

2. Response with a tool call (finish_reason: "tool_calls"):

{
  "choices": [{
    "index": 0,
    "message": {
      "role": "assistant",
      "content": null,
      "tool_calls": [{
        "id": "call_1",
        "type": "function",
        "function": { "name": "get_weather", "arguments": "{\"city\":\"Paris\"}" }
      }]
    },
    "finish_reason": "tool_calls"
  }]
}

3. Follow-up request, appending the assistant’s tool call and a tool role message with the result:

{
  "model": "gpt-4o",
  "messages": [
    { "role": "user", "content": "What is the weather in Paris?" },
    { "role": "assistant", "content": null, "tool_calls": [ /* as above */ ] },
    { "role": "tool", "tool_call_id": "call_1", "content": "15C, cloudy" }
  ]
}

The model grounds its final answer in the tool result and returns a normal finish_reason: "stop" message.

Streaming

With "stream": true, tool_calls arrive as incremental deltas in OpenAI format (id and name first, then argument fragments), same as OpenAI’s own streaming tool-call shape. See Streaming.

Coverage

Translation is implemented for anthropic, google, vertex_ai, bedrock and cohere; the OpenAI-family kinds, vllm and azure pass tools/tool_choice through untouched since they already speak (or near-passthrough, for azure) the OpenAI shape. Provider setup and the full capability matrix are in Providers.

Embeddings

POST /v1/embeddings speaks the OpenAI request and response format. The model field is one of your configured model ids (the id in a [[providers.models]] block - see Providers).

Request

curl -s http://localhost:8080/v1/embeddings \
  -H 'content-type: application/json' \
  -d '{
    "model": "text-embedding-3-small",
    "input": ["the quick brown fox", "a lazy dog"]
  }'

Response

{
  "object": "list",
  "data": [
    { "object": "embedding", "index": 0, "embedding": [0.0023, -0.009, ...] },
    { "object": "embedding", "index": 1, "embedding": [0.0071, 0.014, ...] }
  ],
  "model": "text-embedding-3-small",
  "usage": { "prompt_tokens": 8, "total_tokens": 8 }
}

encoding_format

"encoding_format": "float" (the default) returns each embedding as a JSON float array. "base64" returns it as an OpenAI-style base64 string of little-endian f32 bytes instead. This is purely an output concern: LUMEN always holds vectors as floats internally, so it works uniformly even for providers with no native encoding_format, such as Ollama and TEI.

Accepted input shapes

Pre-tokenized shapes are rejected on providers that cannot consume them (see below). Content-part shapes require a model that opts into the image modality (see Multimodal embeddings).

Pre-tokenized input on text-only providers

Token-id array input ([1,2,3] or [[1,2],[3,4]]) passes through natively on OpenAI-compatible providers. Providers whose upstream API only accepts text - cohere, tei, ollama, jina, voyage, mistral - reject it before any upstream call with LM-1001 (400), naming the provider and the rejected shape. See Error codes.

Unknown fields and input_type (Cohere)

Unlike /v1/chat/completions, unknown request fields on /v1/embeddings are captured but never re-serialized into the outgoing provider body - they stop at the gateway rather than being forwarded, since a strict OpenAI-compatible upstream may reject fields it does not recognize. The one field the gateway itself reads is input_type, consumed only by the Cohere translation to override Cohere’s query-vs-document intent (search_query, search_document, classification, clustering; defaults to search_document). An unrecognized input_type is rejected with LM-1001 before any upstream call. See Providers - cohere.

Strict mode

By default a provider silently drops a request field it cannot honor (for example dimensions sent to Ollama, which has no such parameter). Setting strict = true on that provider’s [[providers]] block makes it reject such a request instead, with LM-1001 naming the field. encoding_format is always handled at the response edge and is never affected by strict.

Providers

Which provider kinds serve embed, and their batch limits, are in Providers.

Batching

An embed request with more inputs than the target provider’s batch limit is split into sub-batches, run with bounded concurrency, and reassembled in the original input order before the response is returned. This is invisible to the client: one request in, one response out, data[].index numbered against the original input regardless of how it was split upstream.

Where limits come from

Every provider kind has a built-in embed batch limit. The native kinds (openai, mistral, cohere, jina, voyage, tei, ollama, azure) each have their own limit; the OpenAI-compatible hosts all share a 2048-input limit. The exact numbers are in the Providers matrix, in the native-kinds table and in the OpenAI-compatible hosts section.

Usage

A batched request still produces exactly one usage object: prompt_tokens and total_tokens are summed across every sub-batch response before the client sees them.

Multimodal embeddings

An embed model opts into image input by declaring the image modality:

[[providers.models]]
id = "embed-multilingual"
capabilities = ["embed"]
modalities = ["text", "image"]

Once opted in, input items may be a bare string or an array of content parts ({"type":"text",...} / {"type":"image_url",...}), and both kinds of item can be mixed within one batch. Image input sent to a model without the image modality is rejected with LM-2003 (400) before any upstream call. See Embeddings for the other accepted input shapes.

Image sources

image_url.url is a data:<media-type>;base64,<payload> inline URI or a remote http(s) URL.

• data: URIs always work - the bytes travel in the request body, no fetch required.
• Remote http(s) URLs require [image_fetch] enabled = true. With fetching disabled (the default), a remote URL is rejected with LM-2005 (400) - pass a data: URI instead, or enable fetching.

The guarded fetch

When [image_fetch] is enabled, a remote URL is fetched server-side, base64-encoded, and inlined before being sent to the provider. The fetch is guarded:

• The private/loopback/link-local IP block is non-configurable (always on).
• The connection is pinned to the resolved address that was vetted against that block, so a DNS answer cannot rebind to a different address after the check (DNS-rebinding safe).
• Scheme, host, and URL-prefix allowlists (allowed_schemes, allowed_hosts, allowed_url_prefixes).
• A streamed size cap (max_bytes) and a fetch timeout (timeout_ms).
• Only image/* response content types are accepted.
• Redirects are re-validated against the same guards, not just the original URL.
• A per-request cap on the number of images fetched.

A request rejected by any of these guards fails with LM-2006 (400); the specific reason is logged server-side only, never returned to the client. A URL that passes the guards but fails at the remote host (network error, timeout, or error status) fails with LM-2007 (502). See Error codes.

Config reference

# ---------------------------------------------------------------------------
# Multimodal image fetching (M9) - OFF by default.
#
# When enabled, a remote http(s) image URL in a /v1/embeddings content-parts
# request is fetched server-side, base64-encoded, and inlined as a data: URI
# before being sent to the provider. Disabled → such a URL is rejected with
# LM-2005 (clients must pass a data: URI themselves).
#
# Guards (always on when enabled): the private/loopback/link-local IP block is
# non-configurable; the connection is pinned to the vetted resolved address
# (DNS-rebinding safe); downloads are capped and time-bounded; only image/*
# content types are accepted. Restrict `allowed_hosts` / `allowed_url_prefixes`
# to your own asset origins in production - leaving both empty with fetching on
# logs a startup warning.
# ---------------------------------------------------------------------------
[image_fetch]
enabled = false
max_bytes = 10485760          # 10 MiB per image
timeout_ms = 5000             # per-fetch timeout
allowed_schemes = ["https"]   # add "http" to allow plaintext
allowed_hosts = []            # e.g. ["cdn.example.com", ".mycompany.com"]; empty = any public host
allowed_url_prefixes = []     # e.g. ["https://cdn.example.com/images/"]; empty = no prefix restriction

Per-provider semantics

Not every multimodal provider embeds mixed input the same way: Cohere (embed-v4) and Voyage embed one combined text+image vector per item, while Jina embeds one modality per item - a mixed item is sent as its image, and its caption text is not combined into the vector. See Providers for the full per-kind notes.

Accounting

Every fetched or inlined image is counted: lumen_media_total and lumen_media_bytes_total in Prometheus, and matching columns in the usage_log table. See Metrics.

Reranking

POST /v1/rerank speaks the Cohere request and response format: query, documents, top_n. The model field is one of your configured model ids (the id in a [[providers.models]] block - see Providers).

Request

curl -s http://localhost:8080/v1/rerank \
  -H 'content-type: application/json' \
  -d '{
    "model": "rerank-english",
    "query": "What is the capital of France?",
    "documents": ["Paris is the capital of France.", "Berlin is in Germany."],
    "top_n": 2
  }'

Response

{
  "results": [
    { "index": 0, "relevance_score": 0.98 },
    { "index": 1, "relevance_score": 0.02 }
  ],
  "usage": { "search_units": 1, "total_tokens": 42, "tokens_estimated": true }
}

Results come back sorted by descending relevance_score. Each result’s index points back to the position of that document in the request’s documents array, not the sorted position.

Empty documents

documents must be non-empty. An empty list is rejected before any upstream call with LM-2010 (400). See Error codes.

Billing: search units and tokens

Rerank billing units vary by upstream: Cohere and Pinecone bill in search units (one unit is approximately one query over up to 100 documents), while Jina and Voyage bill in tokens. Set cost_per_1k_searches on a model’s [[providers.models]] block to price the search-unit case:

[[providers.models]]
id = "rerank-english"
upstream_id = "rerank-v3.5"
capabilities = ["rerank"]
cost_per_1k_searches = 2.0

Search units are counted on the lumen_rerank_search_units_total{model, provider} Prometheus counter for every request, upstream-reported when available, otherwise a gateway estimate (usage.estimated: true).

Independently of the billing unit, every /v1/rerank response also carries a usage.total_tokens count (ADR 003), for uniform token observability across capabilities: Jina and Voyage surface their upstream-reported token count unflagged; every other provider (Cohere, TEI, Pinecone, …) gets a gateway-derived query + documents heuristic flagged "tokens_estimated": true. This token count also feeds lumen_tokens_total{capability="rerank",...} and usage_log, alongside the search-unit accounting - see Token accounting & cost.

One model, two capabilities

A single model id can serve both embed and rerank if the underlying upstream model supports both, for example Cohere’s embed-v4.0:

[[providers.models]]
id = "embed-multilingual"
upstream_id = "embed-v4.0"
capabilities = ["embed", "rerank"]

Cross-vendor fallback

Like any capability, a rerank model can list fallbacks across different provider kinds. A three-hop chain across Cohere, Jina and Voyage survives any single vendor outage:

[[providers.models]]
id = "rerank-english"
upstream_id = "rerank-v3.5"
capabilities = ["rerank"]
fallbacks = ["jina-rerank", "voyage-rerank"]

The model that actually served the request (primary or a fallback) is reported in the x-lumen-model-used response header. See Resilience.

Providers

Which provider kinds serve rerank and their setup is in Providers.

Token accounting & cost

EVERY request of every capability (chat, embed, rerank) produces a token count. Never a silent zero: upstream usage is used when a provider reports it, otherwise the gateway falls back to a local estimate flagged "estimated": true. This is a central reason LUMEN exists, not an afterthought - see ADR 003.

Why it has to be guaranteed

Upstream usage reporting is inconsistent across providers: TEI’s /embed returns a bare vector array with no usage at all; streaming chat only carries usage when the client asks for it (and some providers omit it even then); rerank is billed in search units by Cohere and Pinecone but in tokens by Jina/Voyage, and TEI reports neither. A gateway that only passes through whatever the upstream says would show 0 tokens for a large slice of traffic. LUMEN closes that gap: a count is guaranteed for every call, and the count is always honest about whether it was measured or estimated.

Where it surfaces

1. Response body - the OpenAI-compatible usage object on chat and embeddings responses, and usage.search_units plus usage.total_tokens on rerank responses (rerank carries both a search-unit and a token count, each independently flagged - estimated for search_units, tokens_estimated for total_tokens - see Reranking - Billing).
2. Prometheus - lumen_tokens_total (every capability, including rerank’s token count) and lumen_rerank_search_units_total (rerank’s search-unit count), both carrying an estimated (or upstream-reported) signal. See Metrics & dashboards.
3. usage_log (when [auth] is enabled) - per-request token counts, cost and the estimated flag, alongside status and metadata for later slicing. See Usage log & multi-tenant metadata.

Cached and reasoning token breakdown

When an upstream reports a token breakdown, the gateway surfaces it instead of discarding it (issue #99), on the same three sinks and under the same “never zero, never invented” rule:

• Response body - an OpenAI-compatible usage.prompt_tokens_details (cached_tokens) and usage.completion_tokens_details (reasoning_tokens). For Anthropic prompt caching, cache_read_input_tokens maps to cached_tokens (a cache read/hit, same as OpenAI) and cache_creation_input_tokens rides a distinct cache_creation_tokens field in prompt_tokens_details (a cache write, which has no OpenAI equivalent).
• Prometheus - lumen_token_breakdown_total{capability, model, provider, kind} with kind one of cached, reasoning, cache_write. It is a subset of lumen_tokens_total, split out so cache and reasoning usage can be summed on their own.
• usage_log - the nullable cached_tokens, reasoning_tokens and cache_write_tokens columns.

A breakdown is only ever an upstream fact: when the upstream reports none, all three surfaces omit it (None/NULL/no series), and a locally estimated count carries no breakdown at all.

How estimation works

Source priority, in order:

1. Upstream-reported usage. Authoritative and free - already present in the response body or the final SSE chunk. estimated = false.
2. Local estimation fallback, when the upstream omits usage. estimated = true. The default is a cheap, allocation-light byte/char heuristic that is safe to run anywhere, including the request path; an accurate tokenizer is an opt-in, off-hot-path option run via spawn_blocking.

The hot-path rule holds regardless: the request path never runs a heavy tokenizer. Upstream usage is passed through as-is; when it is missing, the cheap heuristic fills in the response usage field, flagged estimated, and never blocks or slows the request. Streaming chat estimates the same way when the upstream sends no usage at all in its final chunk.

Images. Upstream-reported usage already folds in image cost (OpenAI, Anthropic and Gemini all report image tokens as part of prompt_tokens), so a vision request with upstream usage is exactly as accurate as a text-only one. When the upstream reports nothing, the two capabilities diverge:

• Chat counts each image content part with a flat per-image estimate (85 tokens at "detail": "low", 765 tokens otherwise) instead of counting it as zero. See Vision (image input).
• Embeddings still estimates image parts as zero tokens when the upstream reports no usage; this undercounts image-heavy requests on a no-usage upstream, and a per-image heuristic for embeddings is a backlog item. Media volume itself (count and decoded bytes) is tracked separately via lumen_media_total / lumen_media_bytes_total and the usage_log media_count/media_bytes columns, not through the token counters. See Multimodal embeddings.

Either way, a locally-estimated count is always flagged "estimated": true

• the client is never told a number is measured when it is not.

Cost

Per-model prices feed cost accounting and hard budgets:

[[providers.models]]
id = "gpt-4o"
capabilities = ["chat"]
cost_per_1m_input = 2.5
cost_per_1m_output = 10.0

Rerank models price by search unit instead:

[[providers.models]]
id = "rerank-english"
capabilities = ["rerank"]
cost_per_1k_searches = 2.0

A model without prices set costs 0, so hard budgets never bite on it. See Keys, quotas & budgets for how cost feeds budget enforcement.

Metrics & dashboards

GET /metrics exposes every gateway metric in Prometheus text exposition format. It is unauthenticated by design - restrict it at the network layer (firewall, reverse proxy, service mesh) rather than expecting the gateway to gate it. See SECURITY.md.

Metrics reference

lumen_tokens_total, lumen_rerank_search_units_total, lumen_media_total and lumen_media_bytes_total also gain one extra label per key listed in telemetry.metadata_labels - see Usage log & multi-tenant metadata for the allowlist mechanics and a multi-tenant query example.

Per-request cost is not a Prometheus series: it is a usage_log-only figure, computed from the cost_per_1m_* / cost_per_1k_searches prices in config. See Token accounting & cost.

What to alert on

Several failure modes are deliberately silent on the request path: the gateway sheds work instead of failing requests, and the only trace is a counter. “The drop is visible in your dashboards” is only true if something watches these:

The monitoring rig ships these as ready-made Prometheus rules in monitoring/prometheus/alerts.yml - copy them into your own Prometheus and tune the windows. Routing them to a pager needs an Alertmanager, which the rig intentionally leaves out.

See it live: the monitoring rig

monitoring/ is a one-command Docker Compose stack that runs the gateway against real providers with Prometheus scraping it and a pre-provisioned Grafana dashboard covering every metric above: token rates by provider/model/capability/direction, rerank search units, media accounting, circuit-breaker state and the internal counters. ./smoke.py exercises chat, streaming, embeddings and rerank per provider and asserts a non-zero token count on each; ./traffic.py generates sustained randomized traffic, tagged with multi-tenant metadata, so the dashboard has something to show. It is the fastest way to see all of this live - see monitoring/README.md.

Logging

Logs are the third observability leg next to metrics and the usage log. The gateway writes structured logs to stdout only (no file appender, no rotation of its own - that is the supervisor’s or the log pipeline’s job).

What is never logged

Prompts, responses and provider secrets are treated as radioactive:

• Request and response content is never logged, at any level, including debug and trace. This is the sovereignty pillar, not a default you can toggle.
• Provider API keys and the master key never appear in logs or errors; the redacting Debug implementations keep them out of debug output, and tests enforce it.

What is logged is metadata: request ids, models, providers, HTTP status, latency, operational events (boot, config reloads, circuit-breaker transitions, flush failures), and the client-supplied x-lumen-metadata header when present (a bounded JSON object of up to 16 keys, whole header capped at 4 KiB). Operators must not include secrets or prompt content in the metadata header, as it is logged in full and stored in the usage_log table.

Format

log_format is a top-level config key:

log_format = "json"   # "pretty" (default) or "json"

• pretty: human-readable, colored, for local development.
• json: one JSON object per line with event fields flattened, for production pipelines (Loki, CloudWatch, journald + a parser, …).

The format is boot-time only (logging initializes before anything else runs, right after config parses).

Level and filtering: RUST_LOG

Filtering uses the standard tracing env-filter syntax via the RUST_LOG environment variable. When RUST_LOG is unset, the gateway runs at info.

RUST_LOG=debug lumen --config config.toml            # everything, verbose
RUST_LOG=warn lumen --config config.toml             # quiet: warnings and errors
RUST_LOG="info,lumen=debug,lumen_server=debug" lumen ...   # gateway binary + HTTP layer
RUST_LOG="info,lumen_providers=debug" lumen ...      # debug for provider translation

Targets follow Rust module paths: the binary crate is lumen (its per-request events use the lumen::http and lumen::usage targets); the libraries are lumen_server, lumen_providers, lumen_router, lumen_auth, lumen_telemetry, lumen_core (ADR 001 naming).

Where debug is specifically useful: dropped OpenAI extras on translated providers. In lenient mode, a request field with no equivalent on the target provider (see chat completions) is dropped with a debug line naming the field. If a client swears it sent logprobs and nothing happened, this is where the evidence is.

Rejected metadata needs no debug: malformed x-lumen-metadata never fails a request; it is dropped with a warn line (visible at the default level) and a lumen_metadata_rejected_total increment - see the usage log.

Docker and systemd

Both capture stdout natively:

docker logs <container>                  # or a logging driver
journalctl -u lumen.service -f           # systemd unit from the deployment page

For production pipelines set log_format = "json" (the monitoring rig runs this way) and let the collector parse one object per line.

Usage log & multi-tenant metadata

When [auth] is enabled, every request writes a row to the usage_log table: token counts, cost, status, the model that actually served the request, and metadata - never message content. Prompts and responses are never logged, by default and in the usage log alike.

Never on the request path

Usage-log writes never block a request. Each accounted call pushes an entry onto a bounded async channel (usage_channel_capacity); a separate batched writer task drains it and writes to the database in batches (usage_batch_max entries, or at least every usage_flush_ms). If the channel is ever full, an entry is dropped rather than backpressuring the request path, and lumen_usage_log_dropped_total increments so the drop is visible on /metrics.

Rows age out on their own: retention_days purges usage_log rows older than that many days.

Querying it over HTTP

GET /admin/usage (master-key gated, like every /admin/* route) returns aggregates over these rows - filtered by key, budget group (ADR 009), model, provider, capability and time window, grouped by the dimension you choose. Every row carries the key’s group_id at admission (refusal rows included), so per-pool reporting covers refused traffic too. Because rows arrive through the bounded channel above, requests from the last flush interval may not be visible yet. See Keys, quotas & budgets.

Exporting raw rows: GET /admin/usage/export

GET /admin/usage aggregates over one dimension at a time. A control plane building its own multi-dimensional view (per-tenant AND per-model AND per-day, say) needs the underlying rows instead of a fixed aggregate shape (ADR 010). GET /admin/usage/export returns them directly, cursor-paginated, master-key gated like every other /admin/* route.

Query parameters (all optional, unknown parameters are rejected with 400 LM-1001):

The response shape:

{
  "since": 1785600000,
  "until": 1785686400,
  "rows": [ { "id": 1042, "key_id": "...", "model": "...", "...": "..." } ],
  "next_cursor": 1043
}

since and until are the EFFECTIVE window for this call: either what the caller passed, or the resolved default (24 hours before until, until itself defaulting to “now”). Each row has the same columns as the usage_log table (id, key_id, group_id, model, model_used, provider, capability, token/media/cache counters, cost, latency, status, metadata, ts). As with every other usage surface, rows never carry prompt or response content, only accounting fields.

Pagination is by primary key, not offset, so an export cannot skip or repeat a row when new requests land mid-export: keep requesting with cursor set to the previous page’s next_cursor until next_cursor comes back null, which signals the window is exhausted. A full page is not by itself proof that more data exists; the exhausted signal is always a null cursor, even if that means one extra call returning zero rows at the very end.

Pin the window explicitly when paginating. With no explicit since/ until, both default relative to “now” and are recomputed independently on every call - so a multi-page export that never passes since/until is filtering each page against a window that keeps sliding forward while it pages through. Read since and until off the FIRST response and pass those same two values back on every subsequent page, instead of relying on the defaults again; this is exactly why the response echoes them.

limit is capped at 10000 per page; a request above the cap is rejected (400) rather than silently clamped, so a caller always knows the page it got back was the size it asked for.

curl -s -H "Authorization: Bearer $LUMEN_MASTER_KEY" \
  "http://localhost:8080/admin/usage/export?since=2026-08-01T00:00:00Z&limit=500"

x-lumen-metadata

Clients may attach a per-request metadata header, canonically x-lumen-metadata (alias cf-aig-metadata, for drop-in compatibility with Cloudflare AI Gateway clients). The value is a flat JSON object of string/number/bool values, bounded so log records and memory stay bounded: at most 16 keys, each key at most 64 bytes, each value at most 256 bytes, and the whole header at most 4 KiB.

The full (bounded) object is attached to structured logs and stored in the usage_log metadata column for later filtering. It is opaque - LUMEN never parses it for meaning or PII - and it is logged, so it must never carry secrets or prompt content.

Missing, malformed, oversized or wrong-typed metadata never fails the request: it is dropped with a log line and a lumen_metadata_rejected_total increment, and the call proceeds normally. See ADR 002.

Prometheus label allowlist

Only keys listed in telemetry.metadata_labels become Prometheus labels on the token/media counters; every other key stays logs-only:

[telemetry]
metadata_labels = []              # e.g. ["team", "env"]

The default is empty, which means client-supplied metadata can never mint a single new Prometheus time series - metric cardinality is a deliberate, operator-bounded decision, never a client-driven one. An allowlisted key absent from a given request gets the label value "". Keep the value sets you allowlist bounded: every distinct combination of allowlisted values is its own time series.

Multi-tenant recipe

Send org/team/project identifiers in the metadata header, allowlist the keys you want to slice by, and query per tenant on /metrics. With [auth] enabled, requests also need a virtual key:

curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $LUMEN_KEY" \
  -H 'content-type: application/json' \
  -H 'x-lumen-metadata: {"org_id":"acme","team_id":"rag","project_id":"docs-chat"}' \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi"}]}'

[telemetry]
metadata_labels = ["org_id", "team_id", "project_id"]

Tokens per organization over the last 24 hours:

sum by (org_id) (increase(lumen_tokens_total{org_id!=""}[24h]))

The monitoring/ rig’s traffic.py script simulates exactly this pattern across several tenants and its dashboard has a dedicated multi-tenant panel row - see monitoring/README.md.

Disconnect accounting

A client that disconnects mid-stream is not a gateway failure and must not be recorded as a fake success. The request’s accounting settles at 499 (LM-6001) instead: usage_log.status and the lumen_request_duration_seconds{status="499"} sample both reflect the disconnect, kept out of both the internal-error class and the 5xx status class so a client hanging up never inflates internal-error alerts. See Error codes.

Keys, quotas & budgets

Auth is off by default: with [auth].enabled = false the gateway is an open proxy, with no database at all. Turn it on to get virtual keys, hard budgets and RPM/TPM quotas.

Enabling

[auth]
enabled = true
db_path = "lumen.db"   # SQLite file, created if missing

When enabled, the LUMEN_MASTER_KEY environment variable is required: 64 hex characters (32 bytes). It serves two roles - the bearer token of the /admin/* API, and the AES-256-GCM key that seals provider keys stored at rest. It is never logged and never stored.

What you get

With auth on, every /v1/* request is checked against a virtual key:

• Virtual keys are stored only as BLAKE3 hashes; the plaintext is returned exactly once, at creation, and never again.
• Hard budgets (budget_max, in USD) and RPM/TPM quotas (rpm_limit, tpm_limit) are enforced in memory, before any upstream call - a rejected request never spends and never reaches a provider.
• The database is never on the request path. Budget spend is flushed from memory to SQLite on flush_interval_ms (default 10000). While the process is running, enforcement itself lives in memory ahead of the flush, so budget overruns cannot occur. A crash loses at most that much accounting; after a restart, budgets reload from the last persisted state, so unflushed usage lost in a crash can permit spend beyond the intended budget until the gap closes.
• That database is the only copy of the key hashes, stored provider keys, budget state and usage ledger: see Backups for how not to lose it, and Scaling and high availability for why in-memory enforcement means one instance in v1.

Refusals

See Error codes for the full taxonomy.

Bootstrapping the first key (CLI)

On a fresh deployment there is no usable client key yet, and the admin API requires a running server. lumen keys closes that loop: it runs offline, straight against the SQLite file at auth.db_path - no server needed.

export LUMEN_MASTER_KEY=<64 hex chars>   # same gate as the /admin API
lumen keys create --config config.toml --name team-search \
  --budget-max 50 --rpm-limit 60 --tpm-limit 100000
lumen keys list --config config.toml

keys create prints the record plus the one-time plaintext key as a JSON object on stdout (the same shape as POST /admin/keys, shown below) and never logs it. --budget-max, --rpm-limit, --tpm-limit and --expires-at are optional, exactly like their JSON counterparts; keys list prints the records only (no hashes, no plaintext).

If the server is already running, keys list is safe, but a key created by the CLI only joins the live in-memory key table at the next restart or config reload - prefer POST /admin/keys against the running server in that case.

The admin API

Every route under /admin/* is mounted only when [auth].enabled = true, and every route is gated by the master key (Authorization: Bearer <LUMEN_MASTER_KEY>). Changes apply to the database and the in-memory state together, so they take effect immediately with no restart.

Budget groups

A budget group is a shared pool that several keys draw from (ADR 009). The pattern it exists for: prepaid credits per customer, one key per project of that customer. Without groups you would chunk the customer’s credit across the project keys and rebalance from outside - racy, and credit strands on idle keys. With groups: one group per customer, its keys join it, and a top-up is a single grant.

Create the pool first:

curl -s http://localhost:8080/admin/groups \
  -H "Authorization: Bearer $LUMEN_MASTER_KEY" \
  -H 'content-type: application/json' \
  -d '{"name": "acme-corp", "budget_max": 500.0}'

name is required; budget_max (USD) is optional - omit it for an unlimited pool that only attributes usage. A group has no plaintext and no hash, so unlike key creation there is no one-time secret: the 201 response is just the record, and GET /admin/groups lists the same shape.

{
  "id": "...",
  "name": "acme-corp",
  "budget_max": 500.0,
  "budget_spent": 0.0,
  "created_at": 1752537600,
  "deleted_at": null
}

A group carries budget only: no group-level RPM/TPM, no disabled flag, no expiry (each member key’s own limits still apply; group-level limits are backlog).

Membership. A key belongs to at most one group, via group_id on POST /admin/keys or PATCH /admin/keys/{id}:

curl -s http://localhost:8080/admin/keys \
  -H "Authorization: Bearer $LUMEN_MASTER_KEY" \
  -H 'content-type: application/json' \
  -d '{"name": "acme-project-search", "group_id": "<group id>", "budget_max": 100.0}'

On the patch, group_id is tri-state: leave the field out and membership is unchanged, send JSON null and the key leaves its group, send a string and it joins that group. This is the one deliberate exception to “patch fields cannot clear to NULL” - leaving a group must not require re-minting the key. A group_id naming an unknown or deleted group is refused with 400 LM-1001 before anything is written. Key records expose group_id wherever records appear. lumen keys create takes --group-id <ID> too, but the group must already exist: there is no offline lumen groups subcommand - create groups through the admin API, which needs only the master key.

Enforcement works exactly like a key budget, one level up:

• Admission checks both. The key’s own budget_max (when set) AND the group pool, in memory, before any upstream call. Either refusal is a 402 LM-4001; the message says which scope refused (“budget exceeded for this key” vs “budget exceeded for this key’s group”).
• Refunds mirror per-key budgets (ADR 007): a request refused at admission consumes nothing from either pool, and actual cost settles against both on completion.
• Flush and crash semantics are identical to key budgets: pool spend flushes to SQLite on flush_interval_ms, enforcement lives in memory ahead of the flush, and a crash loses at most one flush interval of pool accounting.

Pool spend is the group’s own accumulator, never recomputed from its members: spend a key accrued before joining (or after leaving) is not moved retroactively.

To top up a customer, raise the cap - pool spend is preserved, and the new cap binds every member key on its next request, no restart (name can be patched the same way; an unknown or deleted id returns 400 LM-1001):

curl -s -X PATCH http://localhost:8080/admin/groups/<id> \
  -H "Authorization: Bearer $LUMEN_MASTER_KEY" \
  -H 'content-type: application/json' \
  -d '{"budget_max": 1000.0}'

A PATCH sends an absolute cap, so top-ups that can race (an automated billing flow) should use grant instead - it adds atomically and never loses a concurrent update.

DELETE /admin/groups/{id} is a soft delete, like keys, and it is refused (400) while the group still has active member keys: move them out (PATCH with "group_id": null or another group) or delete them first. Silently dropping members out of pool enforcement would be worse than the error. The tombstone keeps usage_log attribution, and its final pool spend is flushed on removal.

Attribution. Every usage_log row - successes and 402/429 refusal rows alike - carries the key’s group id (when it has one), so per-customer reporting includes the traffic the pool refused. GET /admin/usage (below) takes a group_id filter and group_by=group_id; under that grouping, rows from ungrouped keys aggregate under an empty group name.

Granting budget (top-ups)

POST /admin/groups/{id}/grant adds to a pool’s cap atomically. It exists for the prepaid-credits flow above: a customer buys credits, and the billing control plane grants their pool.

curl -s -X POST http://localhost:8080/admin/groups/<id>/grant \
  -H "Authorization: Bearer $LUMEN_MASTER_KEY" \
  -H 'content-type: application/json' \
  -d '{"amount": 25.0}'

The body is {"amount": <USD>}, the amount to add to budget_max; spend and quota windows are untouched. The 200 response is the updated group record, the same shape as GET /admin/groups. The key half is POST /admin/keys/{id}/grant: same body, same rules, and the 200 response is the updated key record (the record only - no plaintext, nothing was minted).

Why not a PATCH? A patch sends an absolute cap, so two racing top-ups can lose one: both workers read 500, both add 25, both write 525, and one customer payment vanishes. A grant is an atomic increment on both sides - budget_max = budget_max + ? inside SQLite, a fetch_add on the live in-memory entry - so concurrent grants all land. Like every admin change it takes effect on the very next request, no restart.

Three refusals, all 400 LM-1001:

• A non-positive, non-finite or oversized amount. Zero and negatives are rejected; overflowing literals like 1e999 never even parse; and a single grant is capped at 1e12 USD so repeated grants can never sum the stored cap toward infinity (which would read back as unlimited).
• Unknown or deleted id, like every other admin write.
• A capless target (budget_max null): there is no cap to raise, and silently doing nothing would be worse than the error. Set a cap first (PATCH {"budget_max": ...}), then grant.

One retry caveat for billing automation: a grant is not idempotent. A call that timed out or disconnected may still have landed in the database, and blindly retrying would credit the cap twice - verify with GET /admin/groups (or /admin/keys) before retrying.

Create a key

curl -s http://localhost:8080/admin/keys \
  -H "Authorization: Bearer $LUMEN_MASTER_KEY" \
  -H 'content-type: application/json' \
  -d '{
    "name": "team-search",
    "budget_max": 50.0,
    "rpm_limit": 60,
    "tpm_limit": 100000
  }'

name is required; budget_max, rpm_limit, tpm_limit and expires_at (unix seconds) are all optional - omit any of them for “unlimited”. The response is the only place the plaintext key ever appears:

{
  "key": "sk-lumen-...",
  "id": "...",
  "name": "team-search",
  "group_id": null,
  "budget_max": 50.0,
  "budget_spent": 0.0,
  "rpm_limit": 60,
  "tpm_limit": 100000,
  "expires_at": null,
  "disabled": false,
  "created_at": 1752537600,
  "deleted_at": null
}

Store key now - it is never shown again. PATCH /admin/keys/{id} takes the same budget/quota fields (plus disabled) to adjust an existing key; fields left out of the patch are unchanged, and an unknown id returns 400 LM-1001.

Rotate a key (POST /admin/keys/{id}/rotate)

A lost or leaked key does not have to be replaced by a new one: rotation mints a fresh secret for the same key record.

curl -s -X POST http://localhost:8080/admin/keys/<id>/rotate \
  -H "Authorization: Bearer $LUMEN_MASTER_KEY"

The response has the exact same shape as creation - the new plaintext in key, shown exactly once, plus the record. Everything else is preserved: the id (so usage_log attribution is unbroken), the name, budgets, accrued spend and quotas. The swap is applied to the in-memory table too, so the old plaintext stops authenticating on the very next request and the new one works without a restart. An unknown or deleted id returns 400 LM-1001.

Delete a key (DELETE /admin/keys/{id})

curl -s -X DELETE http://localhost:8080/admin/keys/<id> \
  -H "Authorization: Bearer $LUMEN_MASTER_KEY"

Returns 204 No Content. Deletion is a soft delete by design: usage_log rows reference the key id, so removing the row would orphan the key’s usage history. Instead the row is tombstoned (deleted_at is set) and kept for attribution and audit. A deleted key:

• stops authenticating immediately (the in-memory table is updated, no restart needed), and never loads again at boot;
• disappears from GET /admin/keys; pass ?include_deleted=true to list tombstones (the audit view);
• rejects any further PATCH, DELETE or rotate with 400 LM-1001, like an unknown id - it cannot be resurrected by accident.

Retention of the tombstoned rows follows your usage-log retention policy: they are plain rows in the same database, with no plaintext and no hash that still authenticates. If you only want to pause a key, use PATCH {"disabled": true} instead - that one is reversible.

Store a provider key (PUT /admin/provider-keys/{name})

curl -s -X PUT http://localhost:8080/admin/provider-keys/openai \
  -H "Authorization: Bearer $LUMEN_MASTER_KEY" \
  -H 'content-type: application/json' \
  -d '{"key": "sk-..."}'

{name} is the provider’s configured name (as in [[providers]]). The body is {"key": "<provider api key>"}; a successful call returns 204 No Content. The key is sealed with AES-256-GCM under LUMEN_MASTER_KEY before it touches disk, for providers whose api_key_env is unset or empty. The call pings the hot-reload trigger after sealing the key, so the reloader re-reads provider keys from the encrypted store (off the request path) and rebuilds the provider registry - a rotated key takes effect without a restart. Environment-sourced keys keep precedence over a stored key. See Deployment - Hot reload.

Usage & spend reporting (GET /admin/usage)

Aggregates the usage log per key, budget group, model, provider or capability - the HTTP query surface over the same rows the batched writer persists:

curl -s "http://localhost:8080/admin/usage?group_by=provider&since=2026-07-15T00:00:00Z" \
  -H "Authorization: Bearer $LUMEN_MASTER_KEY"

Query parameters (all optional):

The response echoes the effective window and grouping, then one aggregate per group - request counts split by status class, token totals, the estimated-vs-upstream split (ADR 003), rerank search units, media counts and cost:

{
  "since": 1784073600,
  "until": 1784160000,
  "group_by": "provider",
  "truncated": false,
  "groups": [
    {
      "group": "openai",
      "requests": 1204,
      "requests_ok": 1180,
      "requests_client_error": 20,
      "requests_server_error": 4,
      "tokens_in": 803211,
      "tokens_out": 121408,
      "tokens_total": 924619,
      "estimated_requests": 17,
      "upstream_requests": 1187,
      "search_units": 0,
      "media_count": 3,
      "media_bytes": 402133,
      "cost": 12.41
    }
  ]
}

Groups are ordered by cost (highest first) and capped at limit; when more groups matched, truncated is true and the returned groups are the most expensive ones. A window that matches nothing is a normal 200 with an empty groups array. Invalid filters, timestamps, group_by values or limits are 400 LM-1001.

Two accounting notes:

• Recent requests may lag. Usage rows travel through the bounded channel and its batched writer (see Usage log), so requests from the last flush interval (usage_flush_ms, default 2 s) may not appear yet - and entries dropped under a jammed channel (lumen_usage_log_dropped_total) never will.
• upstream_requests counts rows whose numbers are exact, which includes admission refusals (402/429): they consumed zero tokens, and zero is exact. estimated_requests counts rows whose token counts were locally estimated per ADR 003.
• Provider attribution on refusals. Rows served by a provider carry the provider that actually served them (under a fallback this may differ from the primary). Admission-refusal rows (402/429) never reached a provider; they carry the requested model’s primary provider, so per-provider reports still see the traffic that was headed there.

One encoding note: an RFC3339 +HH:MM offset contains a +, which in a query string means a space - percent-encode it as %2B (until=2026-07-15T03:00:00%2B02:00), or use Z/unix seconds.

Outbound webhooks for budget events

Everything above is pull: a control plane asks LUMEN what happened. Webhooks are the push half (ADR 011), and they exist for one problem in particular. A hard budget refuses with 402 LM-4001 the instant the pool empties, so a prepaid-credits backend that only polls will always learn about the exhaustion after the customer has been refused. A budget.threshold event at 80% is the trigger for an auto-recharge that lands as a POST /admin/keys/{id}/grant before that ever happens.

Absent by default. With no webhook configured, LUMEN makes no outbound call to anything but its providers, and does not even export a lumen_webhook_* metric. Enabling it is a deliberate choice, and it requires auth.enabled = true (every event describes a virtual key or a budget group).

There are two ways to configure one, and they compose: the admin API for a control plane, and the config file for a GitOps deployment.

Creating a webhook through the API

This is the path a billing backend wants: nothing to restart, nothing to edit on the host. Three calls, all gated by LUMEN_MASTER_KEY.

1. Store a signing secret. Generate one, keep your copy - LUMEN seals it and will never hand it back.

export WEBHOOK_SECRET="whsec_$(openssl rand -hex 24)"

# The secret reaches curl on stdin, never in an argument list (any local
# process can read those), and `jq` builds the JSON so a secret containing a
# quote or a backslash cannot corrupt the body. `printf` is a shell builtin,
# so it does not put the value in the process table either.
printf '%s' "$WEBHOOK_SECRET" | jq -Rs '{secret: .}' |
  curl -s -X PUT http://localhost:8080/admin/webhooks/signing-key \
    -H "Authorization: Bearer $LUMEN_MASTER_KEY" \
    -H 'content-type: application/json' \
    --data-binary @-

204. The secret is encrypted with AES-256-GCM under the master key before it touches the disk, exactly like a stored provider key.

2. Create the webhook.

curl -s -X PUT http://localhost:8080/admin/webhooks \
  -H "Authorization: Bearer $LUMEN_MASTER_KEY" \
  -H 'content-type: application/json' \
  -d '{
    "url": "https://backend.example.com/lumen/events",
    "events": ["budget.threshold", "budget.exhausted", "key.disabled"],
    "thresholds": [50, 80, 95],
    "channel_capacity": 1024,
    "timeout_ms": 5000,
    "max_attempts": 5,
    "retry_base_ms": 500
  }'

url is the only required field; every other one has the default shown above. The 200 response is the new state:

{
  "enabled": true,
  "source": "database",
  "settings": { "url": "https://backend.example.com/lumen/events", "...": "..." },
  "signed": true,
  "signing_key_stored": true,
  "updated_at": 1787691194
}

The new policy is in force from that moment: the very next request that crosses a threshold enqueues an event. Delivery itself stays asynchronous and best-effort, so a full queue or a receiver that stays down past the retry budget still drops it (see the guarantees below). The settings are stored, so a restart comes up identically.

3. Check it.

curl -s http://localhost:8080/admin/webhooks \
  -H "Authorization: Bearer $LUMEN_MASTER_KEY"

signed: true is the field to watch. If it is false, deliveries carry no x-lumen-signature and your receiver cannot tell them from anyone else’s traffic.

Editing is the same PUT: it replaces every setting, so send the whole document rather than a fragment (there is no PATCH - a partial update of a delivery policy is how you end up retrying against a URL you meant to change). Every field is editable at runtime, channel_capacity included: LUMEN rebuilds the queue and lets the previous sender drain what it already had.

Rotating the secret is another PUT /admin/webhooks/signing-key. It applies to the next delivery attempt, with nothing restarted. Retries of an event already in flight keep the signature they were created with, so a rotation never makes a pending retry unverifiable.

Deleting:

curl -s -X DELETE http://localhost:8080/admin/webhooks \
  -H "Authorization: Bearer $LUMEN_MASTER_KEY"

Emission stops immediately, and the decision is stored: a [webhooks] block in the config file will not quietly re-enable it on the next reload. DELETE /admin/webhooks/signing-key forgets the secret separately.

Declaring a webhook in the config file

For a deployment where the config file is the source of truth and no control plane calls the API:

[webhooks]
url = "https://backend.example.com/lumen/events"
signing_key_env = "LUMEN_WEBHOOK_SECRET"
events = ["budget.threshold", "budget.exhausted", "key.disabled"]
thresholds = [50, 80, 95]
channel_capacity = 1024
timeout_ms = 5000
max_attempts = 5
retry_base_ms = 500

signing_key_env names the environment variable holding the secret; the secret itself never appears in the file. A named-but-unset variable with no stored secret is a boot error: a billing integration silently downgraded to unsigned deliveries is worse than a refused start.

Which one wins. Settings written through PUT /admin/webhooks are stored in the database and take precedence over this block, so a runtime change is not undone by the next reload. GET /admin/webhooks reports source as "database" or "config" so the two are never ambiguous. If you manage the file, avoid the PUT (or expect the file to become decorative); if you manage the API, the block is just the boot-time default.

The events

Both budget events fire for keys and for budget groups; the payload’s scope says which, and subject_id is the id you would pass to the matching grant route.

Edge-triggering

A threshold fires once per budget epoch, not once per request past it. Cross 80% on Tuesday and every request for the rest of the week is silent. A grant that buys headroom re-arms the thresholds it drops below: top a $100 key that has spent $85 up to $300, and 85/300 = 28% re-arms 50% and 80% for the new epoch. A cap reduction does not un-fire what already fired.

budget.exhausted works the same way: the first refusal signals, the next thousand do not, and a grant re-arms it.

One consequence worth planning for: a restart re-arms from the last flushed spend, so an event can legitimately fire twice if the crash window (auth.flush_interval_ms, 10 s by default) swallowed the settle that first crossed it.

The payload

{
  "id": "evt_9f2c1b7ad04e4a1c8f3b6e2d5a90c714",
  "event": "budget.threshold",
  "scope": "key",
  "subject_id": "3b14b24efc6dc198000cdf5506ddea6d",
  "subject_name": "team-search",
  "budget_max": 100.0,
  "budget_spent": 82.0,
  "threshold": 80,
  "ts": 1787691194
}

Accounting facts only. Never a plaintext key, never client metadata, never prompt or response content - the same no-content construction as usage_log. budget_max is omitted for an uncapped subject, and threshold only appears on budget.threshold.

Verifying a delivery

Every POST carries four headers:

Verify against the raw bytes you received, before any JSON round-trip:

import hashlib, hmac

def verify(raw_body: bytes, header: str, secret: str) -> bool:
    expected = hmac.new(secret.encode(), raw_body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, header)

The secret comes from whichever source is configured: the environment variable named by signing_key_env wins when it is set and non-empty, and the secret stored through PUT /admin/webhooks/signing-key fills in otherwise. No route ever returns it, in any form; GET /admin/webhooks reports the booleans signed and signing_key_stored and the variable name, exactly as the provider surface reports key presence without key material.

Delivery guarantees, and what they are not

At-least-once while the process lives. A retryable failure (5xx, 429, 408, or a network error) backs off exponentially with jitter up to max_attempts, reusing the same x-lumen-event-id every time. Any other non-2xx is treated as permanent and not retried - the receiver has said this event will never be accepted. Nothing is persisted, so a restart forgets undelivered events.

Receivers must be idempotent on x-lumen-event-id. Retries, and the post-restart re-fire described above, both re-send the same id.

Not an accounting system. A full queue drops events rather than slow a request down, and a receiver that stays down past the retry budget loses them. GET /admin/usage/export remains the source of truth: reconcile against it on a schedule and treat webhooks purely as a latency optimisation over polling.

Never on the request path. Detection is a compare on the atomic budget settle that already happens per request; delivery is a non-blocking try_send into a bounded queue drained by a background task. A dead receiver cannot add a millisecond to a customer’s request.

Watching it work

These series appear the first time a webhook is enabled, not at boot, so a gateway that never uses them stays quiet on /metrics too.

Sustained dropped or dead means your billing loop is running blind: fall back to the export route until the receiver is healthy again.

What a reload can change

A config reload re-resolves the same precedence: a stored row still wins, so a reload never reverts a PUT. When the file block is what is in force, every field is re-applied, and removing the block stops emission.

How a field is applied depends on the field. url, events, thresholds, timeout_ms, max_attempts and retry_base_ms are retuned in place, on the queue and sender task already running - so a retarget cannot lose what is already queued. channel_capacity cannot be resized in place, so it replaces both: new events go to the new queue while the previous sender finishes delivering the events it had already accepted, then exits. Either way nothing already accepted is discarded.

The one thing a reload cannot do is see a new environment variable: a running process cannot observe a change to its own environment. Pointing signing_key_env at a variable that was not set when the gateway started needs a restart, or the sealed-secret route instead.

Operator notes

Per SECURITY.md, protect LUMEN_MASTER_KEY and the SQLite database file together: either one alone is not enough to read a stored provider key, but both together decrypt it. Treat them as a single secret.

Resilience tuning

LUMEN survives flaky upstreams without becoming flaky itself. A request goes through, in order: retries, then fallback to the next model in the chain, then the circuit breaker deciding whether to even try a given provider. None of this touches the database on the request path. Retries and the circuit breaker are always active with sane defaults; only per-model fallbacks and background health checks are opt-in. All of it lives under [resilience] - see ADR 005 for the design.

Retries

Applied only to retryable upstream failures: 5xx, connect/read timeouts, and 429. A client 4xx is never retried - a fallback provider would reject it too. Backoff is exponential with equal jitter, and honors an upstream Retry-After header as a floor (a Retry-After: 3 guarantees at least a 3-second wait). While streaming, commitment happens at the first content frame, not at the open: after the upstream opens (2xx + headers), the gateway peeks the first frame before committing. An upstream that opens 200 then errors or closes before delivering any content still retries and falls over, charging the circuit breaker like an open failure, instead of surfacing a terminal SSE error frame - so an immediately-dead stream is still retried even though it opened. Once the first content frame reaches the client the request is committed and a later mid-stream error becomes a clean SSE error frame instead of a retry. See ADR 005 (first-frame-peek amendment) and Streaming.

[resilience]
retry_max_attempts = 3   # total attempts per provider incl. the first (>= 1)
retry_base_ms = 200      # base backoff wait after the first failure
retry_max_ms = 5000      # ceiling on the exponential backoff term

Set retry_max_attempts = 1 to disable retries entirely.

Fallback chains

Each model can declare fallbacks, a list of other model ids to try in order if the primary fails. Fallback chains are validated at boot: every fallback id must exist and serve the same capability as the model it backs, so a runtime resolution miss never happens in practice. Whichever model actually served a request is reported in the x-lumen-model-used response header, so a caller (and the usage log) can see when a fallback fired.

Circuit breaker

Tracked per (provider, model) pair:

circuit_failure_threshold = 5     # consecutive failures that trip it open
circuit_cooldown_ms = 30000       # time spent open before a half-open probe

After circuit_failure_threshold consecutive provider-fault failures, the circuit opens. While open, that link is skipped instantly - no upstream call - straight to the next fallback, or answered with 503 LM-3020 (plus Retry-After set to the cooldown remainder) if none remains. After circuit_cooldown_ms, a single half-open probe decides whether to close again. State is exported as lumen_circuit_state{provider,model} (0 closed, 1 open, 2 half-open).

The three timeouts

A provider’s own connect_timeout_ms override is picked up on hot reload like the other resilience knobs, since the registry rebuilds its clients from the new config.

[resilience]
connect_timeout_ms = 5000
total_timeout_ms = 600000   # 10 minutes

Health checks

Off by default. When health_check_enabled = true, a background task probes, on health_check_interval_ms, every provider that has an explicit base_url (self-hosted TEI/Ollama, or any explicit override) - providers on a built-in vendor URL are never probed and report unknown, since the gateway hardcodes no vendor endpoints. Results are published at GET /health/providers and the lumen_provider_up{provider} gauge. This is independent of the gateway’s own liveness: GET /health never depends on provider state and does no I/O.

health_check_enabled = false
health_check_interval_ms = 30000

How this shapes the error codes you see

Retries, fallback and the circuit breaker all happen before a 3xxx error ever reaches a client - by the time one surfaces, the resilience machinery has already given up. See Error codes: “How resilience shapes these codes” for the full mapping from a given upstream failure to the code you’ll see.

Deployment

Docker

docker run -p 8080:8080 \
  -v ./config.toml:/config.toml \
  -e OPENAI_API_KEY=sk-... \
  ghcr.io/qdequele/lumen:latest

The image is built from a multi-stage Dockerfile: a static musl binary copied onto a distroless/static non-root base - no shell, no libc in the final image, just the gateway. It is multi-arch (linux/amd64 and linux/arm64). The image sets LUMEN_SERVER__HOST=0.0.0.0 for you, so the server binds to all interfaces inside the container; mount your config at /config.toml (the image’s default CMD).

Bare binary

Static musl binaries for x86_64-unknown-linux-musl and aarch64-unknown-linux-musl are attached to every GitHub release cut from a v* tag - a single self-contained file, no runtime dependencies, which makes it systemd-friendly as a single process:

lumen --config /etc/lumen/config.toml

Bind the host/port via [server] in the config file, or the LUMEN_SERVER__HOST / LUMEN_SERVER__PORT environment variables.

systemd unit

A minimal hardened unit. The two numbers that matter: TimeoutStopSec must exceed the gateway’s 30 s drain window (see Shutdown and restarts), and ReadWritePaths must cover the auth database directory when auth is enabled (set an absolute auth.db_path; the default lumen.db is relative to the working directory).

[Unit]
Description=LUMEN gateway
After=network-online.target
Wants=network-online.target

[Service]
User=lumen
Group=lumen
WorkingDirectory=/var/lib/lumen
ExecStart=/usr/local/bin/lumen --config /etc/lumen/config.toml
# Provider API keys and LUMEN_MASTER_KEY, mode 0600, never in the config.
EnvironmentFile=/etc/lumen/env
# SIGHUP = config hot reload (see below). systemd's default stop signal is
# SIGTERM, which is the graceful-drain path.
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
# The gateway drains in-flight requests for up to 30 s on SIGTERM, then
# runs a final accounting flush with a bounded wait of up to 5 s; give the
# whole clean path headroom before systemd escalates to SIGKILL.
TimeoutStopSec=40
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=/var/lib/lumen

[Install]
WantedBy=multi-user.target

TLS and the reverse proxy

LUMEN intentionally does not terminate TLS. Put a reverse proxy (nginx, Caddy, your load balancer) in front of it, and leave HSTS to that proxy. The gateway speaks plain HTTP and should not be exposed directly to the internet without one.

Caddy needs two lines (automatic HTTPS, streams flush correctly by default):

gateway.example.com {
    reverse_proxy 127.0.0.1:8080
}

nginx needs response buffering off, or SSE streams arrive in bursts instead of token by token, and a read timeout longer than your slowest stream:

location / {
    proxy_pass http://127.0.0.1:8080;
    proxy_http_version 1.1;
    proxy_buffering off;
    proxy_read_timeout 300s;
}

Every response does carry a conservative set of default security headers:

• X-Content-Type-Options: nosniff
• X-Frame-Options: DENY
• Referrer-Policy: no-referrer
• Content-Security-Policy: default-src 'none'

Surface control

• Restrict /admin/* and /metrics at the network layer (firewall, reverse proxy, service mesh) as appropriate for your deployment. /admin/* requires the master key, but /metrics is unauthenticated by design - see SECURITY.md.
• GET /health is safe to point a liveness probe at: it never depends on provider state and does no I/O.

Scaling and high availability

The honest answer to “can I run two replicas behind the load balancer?” is: it depends on whether auth is enabled, and v1 does not paper over that.

With [auth].enabled = false (the default), the gateway is a stateless proxy: no database, no keys, no budgets. Run as many replicas as you like; nothing breaks. Two per-instance caveats remain: circuit breakers and health probes are per-process (each replica discovers a bad upstream on its own), and /metrics is per-instance (scrape every replica and aggregate in PromQL - the counters sum correctly).

With [auth].enabled = true, v1 is single-instance by design. Hard budgets and RPM/TPM quotas are enforced in per-process memory (that is what keeps the database off the request path), and spend is flushed to a per-node SQLite file. Budget-group pools (ADR 009) are enforced the same way, in the same per-instance memory, so the single-instance constraint applies to a shared pool identically. Behind a load balancer, N replicas each enforce the full budget and quota independently: a $100 hard budget becomes an effective $100 x N, an rpm_limit of 60 becomes 60 x N, and the usage ledger splits into N disjoint database files. Nothing crashes - the guarantees silently stop meaning what they say, which is worse.

Until then, the supported shapes with auth enabled are:

• One active instance. A supervisor (systemd, a single-replica Kubernetes Deployment with strategy: Recreate) restarts it; the drain semantics below bound the restart blip to seconds.
• Active/passive: a standby instance behind a failover VIP or LB health check, sharing nothing. On failover the standby starts from its own (empty or restored) database; budgets re-enforce from the last flushed state of whatever database it opens.

A shared Postgres backend for the auth/usage store and distributed rate limiting are the v2 items that lift this constraint - see the backlog.

Hot reload

A SIGHUP, a file-watch event, or an admin provider-key rotation (PUT /admin/provider-keys/{name}) triggers a config reload: the new config is validated first, and only then are the provider registry, price table, resilience policy and the runtime-safe [auth] knobs (flush_interval_ms, retention_days) atomically swapped in. Every reload also re-reads DB-stored provider keys, so a key rotated via the admin API takes effect without a restart even without an explicit trigger call; a DB read error keeps the previous snapshot rather than stripping a working key. In-flight requests are unaffected. If the new config is invalid, it is rejected - the old config keeps serving, and lumen_config_reload_failures_total increments so the failed reload is visible in your dashboards.

Some settings stay boot-time only and need a real restart: the bind address, auth.enabled, auth.db_path, and the bounded usage-log channel knobs (usage_channel_capacity, usage_batch_max, usage_flush_ms) - rebinding a live listener or resizing a running channel is out of scope for a live swap.

PUT /admin/config (ADR 010) applies a new config document remotely instead of an operator editing the file by hand: it stages the submitted bytes next to the real config file, validates the staged copy, then backs up the current file to .bak and renames the staged file into place before triggering the same reload path as above. This means the gateway process needs write permission on the config file’s directory, not just the file itself (the staged file and the .bak sibling are both new files created next to it, and the final apply is a rename within that directory). A read-only config mount (a common hardening choice, e.g. a Kubernetes ConfigMap volume or an immutable container layer) disables this route: the staging write fails and the request is rejected with an internal error (LM-5001), which is the correct refusal - the alternative would be a silent apply that never actually took effect. GET/reading the config still works read-only; only the PUT needs the extra permission. Relatedly, if the config path is a symlink (a pattern some ConfigMap-mount setups and manual atomic-deploy scripts use), an apply’s rename REPLACES the symlink itself with a regular file - the same rename that lands the new document in place cannot also preserve “the path is a symlink pointing elsewhere”; a setup that depends on the config path staying a symlink across reloads is not compatible with applying through this route. See Applying a new config over the admin API for the full request contract, and its security note on what holding the master key implies once this route exists.

Shutdown and restarts

What each signal does:

The drain window is a built-in constant, not configurable, and a clean stop can spend up to 5 more seconds on the final accounting flush below. Tune your supervisor against the whole path: systemd TimeoutStopSec=40 (the unit above), Kubernetes terminationGracePeriodSeconds: 40. A supervisor that kills sooner turns graceful restarts into the crash case below; if draining ever exceeds 30 s, the gateway logs a warning and exits anyway rather than hanging.

Accounting across a stop, when auth is enabled:

• Clean shutdown attempts a final accounting flush with a bounded wait. After the listener drains, the gateway performs a final budget flush and waits up to 5 seconds for the usage-log writer to drain its channel. Usage rows still in the channel when the wait expires are lost, as are rows from database write failures (both logged as warnings rather than blocking exit). In the common case, a clean shutdown loses no accounting.
• A crash loses at most flush_interval_ms (default 10 s) of budget accounting and whatever usage-log rows were still in the bounded channel. Budget enforcement itself lives in memory ahead of the flush, so a running process never allows overruns. After a restart, however, budgets reload from the last persisted state: unflushed usage lost in a crash can permit spend beyond the intended budget until the gap closes.

Rolling restarts through the reverse proxy work as expected: mark the instance down (or just send SIGTERM), let the 30 s drain finish the in-flight streams, start the new binary. With auth enabled, avoid running old and new concurrently for long - see Scaling and high availability.

Backups

Everything durable lives in one SQLite file: auth.db_path (default lumen.db). It is the only copy of the virtual-key hashes, the encrypted provider keys stored via the admin API, the budget state, and the entire usage_log ledger. With auth disabled there is no database and nothing to back up.

• Live backup (server running): the database runs in WAL mode, so use SQLite’s online backup rather than copying the file:

  sqlite3 /var/lib/lumen/lumen.db ".backup '/backups/lumen-$(date +%F).db'"
  

  A live backup can trail reality by up to flush_interval_ms (default 10 s) of budget accounting - the in-memory spend not yet flushed.

• Cold backup (consistent snapshot): stop the gateway first (a clean SIGTERM attempts a final flush, see above), then copy lumen.db together with its -wal and -shm sidecar files if present. The backup reflects the state at shutdown, though in-flight or unflushed usage rows from the bounded channel may not be included.

• Restore needs the matching LUMEN_MASTER_KEY. Stored provider keys are encrypted under it; a restored database without the same master key serves virtual keys and history fine, but every stored provider key is undecryptable (re-enter them via the admin API). Per SECURITY.md, the key and the database are a pair: back up and protect them together, but never in the same place.

• Virtual keys are stored as BLAKE3 hashes and the plaintext is shown only once at creation: a lost database is unrecoverable key-wise. Clients keep their plaintext keys, but the gateway no longer knows them; they must be re-created. Back up on a schedule that matches how much usage_log history you are willing to lose.

Validate configs in the pipeline

Run lumen --check-config in CI or your deploy pipeline before a real boot. It performs the same parsing, semantic validation and provider registry construction the server does at startup, then exits 0 if the config is valid and non-zero otherwise - without binding a listener, opening a database, or contacting a provider. See Installation.

Upgrades

An upgrade is: replace the binary (or pull the new image), restart. This page covers the two things that make that boring in practice: the database migration story and what version numbers promise.

Before upgrading

1. Read the target release’s section in CHANGELOG.md. Release-specific upgrade notes live there (for example, 0.2.0 carries a one-line repair for databases created by 0.1.0, after a migration file’s checksum changed).
2. Back up the database if auth is enabled - see Backups. Upgrades run schema migrations automatically, and the backup is your downgrade path.
3. Validate your config against the new binary before booting it for real: lumen-new --check-config --config /etc/lumen/config.toml. A removed or renamed config key fails here, in the pipeline, instead of at restart time.

Schema migrations run themselves

When auth is enabled, the gateway applies its embedded, numbered SQLite migrations automatically at boot (six of them as of 0.2.0). There is no separate migrate command and nothing to run by hand.

• Forward-only. There are no down-migrations. Rolling back to an older binary against a database a newer binary already migrated is not supported: the older binary refuses to start when it finds applied migrations it does not know. The downgrade path is the pre-upgrade backup.
• Integrity-checked. Each applied migration’s checksum is verified at boot; a mismatch is a hard, named error rather than a silent divergence.
• With [auth].enabled = false there is no database and this whole section is moot.

The restart itself

SIGTERM the old process (or let your supervisor do it), start the new one. In-flight requests get up to 30 seconds to finish and a clean shutdown attempts a final accounting flush with a bounded wait (up to 5 seconds) - the mechanics and the supervisor timeouts to pair with them are in Shutdown and restarts. With auth enabled, prefer stop-then-start over running old and new side by side; two live instances double-enforce budgets and quotas for as long as they overlap (see Scaling and high availability).

What version numbers promise

LUMEN is pre-1.0 and follows SemVer’s 0.x rules: a minor bump (0.2 -> 0.3) may contain breaking changes; a patch bump does not. Every breaking change is called out explicitly in the CHANGELOG. Per surface:

• HTTP API: the OpenAI-compatible (/v1/chat/completions, /v1/embeddings) and Cohere-compatible (/v1/rerank) surfaces track those upstream formats; gateway-specific behavior changes are CHANGELOG items.
• LM-xxxx error codes are stable identifiers. Codes are never renumbered or reused; new ones are added and documented in the error reference.
• Config format: keys can be added in any release; removals or renames are breaking changes (CHANGELOG + caught by --check-config, which rejects unknown keys).
• Metrics: renaming or re-labeling an exported series is a breaking change for your dashboards and is called out in the CHANGELOG.

Examples

Runnable scenario configs live in examples/ at the root of the repository. Each directory is self-contained: a config.toml (the gateway config), a README.md (what it demonstrates and any prerequisites), and a run.sh (the requests to fire once the gateway is up).

Every scenario follows the same two-terminal recipe: start the gateway with the scenario’s config in one terminal, then fire its run.sh in another. Provider keys are never written into a config file; each config reads them from the environment variable named by its api_key_env field.

Every config.toml in examples/ passes lumen --check-config in CI.

minimal-chat

The smallest possible LUMEN config: one provider (OpenAI), one model (gpt-4o), chat only.

Demonstrates: non-streaming chat via POST /v1/chat/completions, and the same endpoint with "stream": true.

Env vars: OPENAI_API_KEY.

# terminal 1
export OPENAI_API_KEY=sk-...
cargo run -p server -- --config examples/minimal-chat/config.toml

# terminal 2
./examples/minimal-chat/run.sh

examples/minimal-chat on GitHub

self-hosted

A fully keyless config: no cloud provider, no API key anywhere. Chat and embeddings come from Ollama, reranking from TEI. Everything runs offline once the models are pulled.

Demonstrates: chat against Ollama’s OpenAI-compatible endpoint, embeddings against Ollama’s native endpoint, and reranking against a local TEI server.

Env vars: none. Requires Ollama running locally with llama3.2 and nomic-embed-text pulled, and (optionally) TEI serving BAAI/bge-reranker-large on port 8081.

# terminal 1
cargo run -p server -- --config examples/self-hosted/config.toml

# terminal 2
./examples/self-hosted/run.sh

examples/self-hosted on GitHub

multi-provider-fallback

Cross-vendor chat fallback: gpt-4o (OpenAI) is primary, with claude-sonnet-4-5 (Anthropic) declared as its fallbacks.

Demonstrates: the x-lumen-model-used response header reporting whether the primary or the fallback served a request, and how the circuit breaker trips after repeated failures on the primary. See Resilience tuning.

Env vars: OPENAI_API_KEY, ANTHROPIC_API_KEY (the Anthropic key is only used if the fallback actually fires).

# terminal 1
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
cargo run -p server -- --config examples/multi-provider-fallback/config.toml

# terminal 2
./examples/multi-provider-fallback/run.sh

examples/multi-provider-fallback on GitHub

rag-pipeline

The two calls behind a typical RAG pipeline, wired to two different providers: embeddings via OpenAI (text-embedding-3-small) at index time, reranking via Cohere (rerank-english) at query time.

Demonstrates: POST /v1/embeddings embedding a small document corpus, then POST /v1/rerank re-scoring the same documents against a query with "top_n": 2. See Embeddings and Reranking.

Env vars: OPENAI_API_KEY, COHERE_API_KEY.

# terminal 1
export OPENAI_API_KEY=sk-...
export COHERE_API_KEY=...
cargo run -p server -- --config examples/rag-pipeline/config.toml

# terminal 2
./examples/rag-pipeline/run.sh

examples/rag-pipeline on GitHub

multi-tenant-analytics

Per-tenant cost and usage attribution: [auth] enabled with a virtual key per tenant carrying a hard budget, and [telemetry].metadata_labels turning the x-lumen-metadata header into Prometheus labels.

Demonstrates: creating a virtual key through the admin API (POST /admin/keys, master-key bearer), tagging requests with x-lumen-metadata, and slicing lumen_tokens_total on /metrics by org_id. See Usage log & multi-tenant metadata and Keys, quotas & budgets.

Env vars: OPENAI_API_KEY, LUMEN_MASTER_KEY (64 hex characters, e.g. openssl rand -hex 32).

# terminal 1
export OPENAI_API_KEY=sk-...
export LUMEN_MASTER_KEY=$(openssl rand -hex 32)
cargo run -p server -- --config examples/multi-tenant-analytics/config.toml

# terminal 2
export LUMEN_MASTER_KEY=...   # same value as terminal 1
./examples/multi-tenant-analytics/run.sh

--check-config on this scenario does not need LUMEN_MASTER_KEY set: the master key is a secret read from the environment at actual server startup, never part of the config file (and the config loader explicitly ignores it).

examples/multi-tenant-analytics on GitHub

Providers

LUMEN ships twenty-six built-in provider kinds - fifteen native integrations (their own request/response translation, including deployment-routed azure and SigV4-signed bedrock) plus eleven OpenAI-compatible hosts that reuse the OpenAI path with a per-kind base URL. Each [[providers]] block in your config selects one with a kind string and gives it a unique name (your own label). Each [[providers.models]] block under it exposes a model to clients:

[[providers]]
name = "my-openai"        # your label; must be unique
kind = "openai"           # selects the built-in implementation
api_key_env = "OPENAI_API_KEY"   # NAME of the env var holding the key
# base_url = "https://…"  # optional override (required for self-hosted kinds)

[[providers.models]]
id = "gpt-4o"             # the id clients send (owned entirely by you)
upstream_id = "gpt-4o-2024-08-06"   # what LUMEN sends upstream (defaults to `id`)
capabilities = ["chat"]   # any of "chat", "embed", "rerank"

Rules that apply to every provider:

• API keys are never in the config. api_key_env names an environment variable; LUMEN reads it only when a request actually routes to that provider. A hosted provider whose env var is unset fails only at use, not at boot - a partial set of keys is fine.
• Model ids are globally unique across all providers. A collision aborts startup and names both offending providers. Several ids may map to one upstream_id (versioned aliasing).
• capabilities must match the kind. A model can only declare capabilities its provider kind implements (table below); this is validated at boot.
• base_url is an optional override for hosted kinds, and required for the self-hosted kinds (tei, ollama), which are keyless.
• Batching: an embed request with more inputs than the provider’s batch limit is split into sub-batches, run with bounded concurrency, and reassembled in the original order. The limits below are built in.
• Multimodal embeddings (M9): declare modalities = ["text", "image"] on a model to accept image content parts on /v1/embeddings. input items may be strings or arrays of parts ({"type":"text",...} / {"type":"image_url",...}). Images are passed as data: URIs, or - with [image_fetch] enabled - as remote http(s) URLs the gateway fetches under SSRF/resource guards and inlines. Image input to a model without "image" is rejected with LM-2003; a remote URL with fetching disabled is LM-2005. Cohere (embed-v4) and Voyage embed a combined text+image vector per item; Jina embeds one modality per item (a mixed item is sent as its image, its caption text is not combined). See the multimodal-embeddings design spec for the full guard list.

The together kind (in the OpenAI-compatible table below) additionally serves rerank (LlamaRank) natively; see its section for the model config.

OpenAI-compatible hosts (chat + embed via the OpenAI path). The Embed column reflects what each host actually serves upstream: groq, deepseek, openrouter, perplexity and xai expose no /embeddings endpoint, so a model declaring embed on those kinds is rejected at config load (it could only ever 404 at request time) - unless the provider sets a custom base_url, which is taken to mean an operator-run proxy that may serve embeddings:

Self-hosted or catalog-dependent kinds (vllm, huggingface, cloudflare) stay permissive: the operator controls what their endpoint serves. If one of the embed-less hosts above later ships an embeddings API, point a kind = "openai" provider at it with a base_url override, or file an issue to update the capability table.

All embed-serving OpenAI-compatible kinds use a 2048-input embed batch limit. Anything that speaks the OpenAI wire format but isn’t listed can still be used via kind = "openai" with a base_url override.

cloudflare additionally serves rerank (not shown in the table above, which covers only the chat/embed OpenAI-compatible path): its BAAI bge-reranker-* models are served through Workers AI’s native /ai/run/{model} endpoint rather than an OpenAI-compatible one. See ### cloudflare below.

openai

• kind: openai · capabilities: chat, embed
• Auth: api_key_env (e.g. OPENAI_API_KEY), sent as a bearer token.
• base_url: optional; defaults to OpenAI’s public API. Set it to point at any OpenAI-compatible endpoint.
• Embed batch limit: 2048 inputs per upstream call.

[[providers]]
name = "openai"
kind = "openai"
api_key_env = "OPENAI_API_KEY"

[[providers.models]]
id = "gpt-4o"
upstream_id = "gpt-4o-2024-08-06"
capabilities = ["chat"]

[[providers.models]]
id = "text-embedding-3-small"
capabilities = ["embed"]

mistral

• kind: mistral · capabilities: chat, embed (OpenAI-compatible).
• Auth: api_key_env (e.g. MISTRAL_API_KEY), bearer token.
• base_url: optional override.
• Embed batch limit: 512.

[[providers]]
name = "mistral"
kind = "mistral"
api_key_env = "MISTRAL_API_KEY"

[[providers.models]]
id = "mistral-small"
upstream_id = "mistral-small-latest"
capabilities = ["chat"]

anthropic

• kind: anthropic · capabilities: chat only.
• Auth: api_key_env (e.g. ANTHROPIC_API_KEY). LUMEN authenticates with the x-api-key / anthropic-version headers, not a bearer token.
• Translation: OpenAI ⇄ Anthropic is bidirectional, including tools and streaming events, so clients keep using the OpenAI wire format. parallel_tool_calls: false maps to tool_choice.disable_parallel_tool_use; response_format, seed, logprobs, frequency_penalty and presence_penalty have no Messages API equivalent - see chat extras and the strict flag.
• base_url: optional override.

[[providers]]
name = "anthropic"
kind = "anthropic"
api_key_env = "ANTHROPIC_API_KEY"

[[providers.models]]
id = "claude-sonnet-4-5"
upstream_id = "claude-sonnet-4-5-20250929"
capabilities = ["chat"]

google

• kind: google · capabilities: chat, embed (Gemini).
• Auth: api_key_env (e.g. GEMINI_API_KEY). The key rides the x-goog-api-key header, never the URL.
• Translation: OpenAI ⇄ Gemini, including streaming (streamGenerateContent). response_format JSON mode maps to generationConfig.responseMimeType / responseSchema, seed to generationConfig.seed, and frequency_penalty / presence_penalty to generationConfig.frequencyPenalty / presencePenalty; logprobs and parallel_tool_calls have no mapping - see chat extras.
• Embeddings: served through models/{model}:batchEmbedContents (gemini-embedding-001, text-embedding-004, …). One inner request per input; the OpenAI dimensions field maps to outputDimensionality. The API is text-only: pre-tokenized token-id arrays and image content parts are rejected with LM-1001 before any upstream call. Usage follows ADR 003: usageMetadata.promptTokenCount is reported when the upstream returns it, otherwise the gateway derives a local estimate marked estimated.
• Embed batch limit: 100 inputs per upstream call (Gemini’s documented batchEmbedContents ceiling); the gateway splits larger requests.
• base_url: optional override.

[[providers]]
name = "google"
kind = "google"
api_key_env = "GEMINI_API_KEY"

[[providers.models]]
id = "gemini-2.0-flash"
upstream_id = "gemini-2.0-flash"
capabilities = ["chat"]

[[providers.models]]
id = "gemini-embedding-001"
capabilities = ["embed"]

vertex_ai

• kind: vertex_ai · capabilities: chat, embed (Gemini and text-embedding-* models on Google Cloud Vertex AI). Distinct from google, which is the public Gemini Developer API: Vertex uses regional endpoints and GCP OAuth instead of a static API key.
• Auth: api_key_env names an env var holding the full service-account key JSON (the contents of the key file downloaded from GCP, not a path and not an API key). LUMEN signs an RS256 JWT assertion with the account’s private key, exchanges it at the account’s token_uri for a short-lived OAuth2 access token (scope cloud-platform), and sends it as a Bearer header. Tokens are cached in memory and refreshed 60 s before expiry, so the exchange stays off the per-request hot path. The private key is redacted from all Debug output and never appears in logs or errors.
• base_url: required - it carries the GCP region (e.g. us-central1), not a URL. The endpoint is derived from it: https://{region}-aiplatform.googleapis.com/v1/projects/{project}/locations/{region}/publishers/google/models/{model}:generateContent (and :streamGenerateContent?alt=sse when streaming).
• Project id: taken from the service-account JSON’s project_id.
• Translation: chat is identical to google (same GenerateContent wire schema), including streaming. Like Gemini, only inline base64 image data is accepted; remote image URLs are rejected with LM-2004.
• Embeddings: Vertex does NOT expose batchEmbedContents; embeddings go through the prediction API on the same regional, project-scoped path: .../publishers/google/models/{model}:predict with instances[].content. The OpenAI dimensions field maps to parameters.outputDimensionality. Text-only: token-id arrays and image parts are rejected with LM-1001. Usage follows ADR 003: per-input statistics.token_count values are summed and reported as upstream usage; when absent the gateway derives a local estimate marked estimated.
• Embed batch limit: 1 input per upstream call. gemini-embedding-001 accepts a single instance per :predict request (other text-embedding-* models take more, but the limit is per-model, so the universally safe value is used); the gateway fans larger requests out over concurrent calls.

[[providers]]
name = "vertex"
kind = "vertex_ai"
# The env var holds the service-account key file's JSON contents:
#   export VERTEX_SA_JSON="$(cat service-account.json)"
api_key_env = "VERTEX_SA_JSON"
base_url = "us-central1"   # GCP region

[[providers.models]]
id = "gemini-flash-vertex"
upstream_id = "gemini-2.0-flash"
capabilities = ["chat"]

[[providers.models]]
id = "gemini-embedding-vertex"
upstream_id = "gemini-embedding-001"
capabilities = ["embed"]

bedrock

• kind: bedrock · capabilities: chat, embed.
• Auth: AWS Signature Version 4 (SigV4), not a bearer key. Credentials are read from the standard AWS environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and optionally AWS_SESSION_TOKEN for temporary credentials) on every request, so values updated in the process environment (or a config hot reload) take effect without a restart. api_key_env is optional and, if set, overrides only the secret access key. The secret and session token are never logged or shown in Debug.
• Credential scope (v1): only static keys and pre-issued STS session tokens. There is no AWS credential-provider chain (no IMDS/instance roles, SSO, profiles or credential_process); an expired session token keeps failing with 403 until the environment supplies a fresh one.
• API (chat): the Bedrock Converse API (POST /model/{modelId}/converse and /converse-stream), which gives one uniform schema across the Anthropic, Meta Llama, Amazon Titan/Nova, Mistral and Cohere model families. The legacy per-model InvokeModel chat schemas are intentionally not implemented (Converse covers the same chat models).
• API (embeddings): Bedrock has no Converse equivalent for embeddings, so the embed path uses per-model InvokeModel (POST /model/{modelId}/invoke), routed by model id (issue #95):
  • Amazon Titan (amazon.titan-embed-text-v2:0 and predecessors): embeds ONE text per call ({ "inputText": ... }); the gateway loops one signed request per input and reassembles the batch in order. Titan v2 honors dimensions (mapped to Titan’s dimensions/normalize); older Titan models do not, so dimensions is dropped (or rejected under strict).
  • Cohere Embed on Bedrock (cohere.embed-english-v3, cohere.embed-multilingual-v3): embeds a batch in one call ({ "texts": [...], "input_type": ... }); input_type defaults to search_document and honors an override. Cohere does not accept dimensions.
  • Batch limit: reported as 1 (the conservative floor across the two families, whose real per-call limits are 1 for Titan and 96 for Cohere), so the router splits every batch into single-input sub-batches it runs concurrently. Pre-tokenized (token-id) and image inputs are rejected with an honest 400 before any upstream call (both families are text-only).
  • Usage (ADR 003): Titan’s inputTextTokenCount and Cohere’s x-amzn-bedrock-input-token-count response header are reported as upstream usage; when absent the request edge derives the local estimate (never a silent zero).
• Region / base_url: set base_url to the runtime endpoint for your region, https://bedrock-runtime.{region}.amazonaws.com; the region is parsed back out of it for the SigV4 signing scope. VPC/PrivateLink endpoint hosts (bedrock-runtime.{region}.vpce.amazonaws.com, including a vpce-…-prefixed DNS name) are recognised too. For any other custom endpoint the region comes from AWS_REGION / AWS_DEFAULT_REGION; if no source yields a region, startup fails with a clear error rather than silently signing for a wrong region.
• Translation: OpenAI ⇄ Converse is bidirectional, including system prompts, inferenceConfig (max tokens, temperature, top-p, stop sequences), tools, and streaming. Streaming arrives as AWS event-stream binary frames, decoded and translated to OpenAI chunks. Usage (inputTokens / outputTokens) is mapped per ADR 003. Converse has no equivalent for response_format, seed, logprobs, parallel_tool_calls, frequency_penalty or presence_penalty
  • see chat extras and the strict flag.
• Images: only inline data: URIs are supported (Converse takes raw image bytes); a remote image URL is rejected (LM-2004) since Bedrock cannot fetch one.

[[providers]]
name = "bedrock"
kind = "bedrock"
base_url = "https://bedrock-runtime.us-east-1.amazonaws.com"
# api_key_env = "AWS_SECRET_ACCESS_KEY"   # optional secret override

[[providers.models]]
id = "bedrock-claude-sonnet-4-5"
upstream_id = "us.anthropic.claude-sonnet-4-5-20250929-v1:0"
capabilities = ["chat"]
modalities = ["text", "image"]

[[providers.models]]
id = "bedrock-titan-embed"
upstream_id = "amazon.titan-embed-text-v2:0"
capabilities = ["embed"]

[[providers.models]]
id = "bedrock-cohere-embed"
upstream_id = "cohere.embed-english-v3"
capabilities = ["embed"]

cohere

• kind: cohere · capabilities: chat, embed, rerank. A single model can serve any combination.
• Auth: api_key_env (e.g. COHERE_API_KEY), bearer token.
• Chat: Command R / R+ via POST /v2/chat, including streaming. The wire shape is OpenAI-adjacent (roles live directly in messages, no top-level system hoist like Anthropic; tool_calls are already OpenAI-shaped), so translation is closer to identity than Anthropic’s. tool_choice collapses to Cohere’s REQUIRED/NONE (forcing one specific named tool has no v2 equivalent and falls back to auto). response_format, seed, frequency_penalty and presence_penalty map onto Cohere’s native fields; logprobs and parallel_tool_calls do not - see chat extras. Usage prefers usage.tokens (actual counts) over usage.billed_units (what’s charged); a response reporting neither leaves the gateway’s local estimator to fill in an honestly-flagged count (ADR 003).
• Vision (issue #73): a user message carrying image parts is translated to Cohere v2 content blocks (text / image_url, OpenAI-shaped); a text-only message keeps the plain-string form, and non-user roles always flatten to text (Cohere only admits image content on user messages). Declare modalities = ["text", "image"] on a vision model (Command-A-Vision) to opt in. Both inline data: URIs and remote http(s) URLs are forwarded (Cohere fetches remote URLs itself, so LM-2004 does not apply); the optional detail hint (low/high/auto) passes through untouched. Provider-native references (anthropic-file:, gs://, Gemini Files API URIs) are rejected pre-flight with LM-2008.
• Embed batch limit: 96.
• Cost: rerank is billed in search units (cost_per_1k_searches).
• input_type override: Cohere’s embed v2 API requires an input_type and the gateway cannot know query-vs-document intent, so it defaults to search_document (the indexing case). Set input_type as an extra field on the /v1/embeddings request body to override it per request, e.g. {"model": "embed-multilingual", "input": "...", "input_type": "search_query"}. Allowed values: search_document, search_query, classification, clustering. An unrecognized value is rejected with LM-1001 before any upstream call. The field is consumed at the gateway: only the Cohere translation reads it, and it is never forwarded in the outgoing body of any other provider (a strict OpenAI-compatible upstream such as vLLM could reject unknown fields).

[[providers]]
name = "cohere"
kind = "cohere"
api_key_env = "COHERE_API_KEY"

[[providers.models]]
id = "command-r-plus"
upstream_id = "command-r-plus-08-2024"
capabilities = ["chat"]

[[providers.models]]
id = "rerank-english"
upstream_id = "rerank-v3.5"
capabilities = ["rerank"]
cost_per_1k_searches = 2.0

[[providers.models]]
id = "embed-multilingual"
upstream_id = "embed-v4.0"
capabilities = ["embed", "rerank"]

jina

• kind: jina · capabilities: embed, rerank (hosted).
• Auth: api_key_env (e.g. JINA_API_KEY), bearer token.
• Embed batch limit: 2048.

[[providers]]
name = "jina"
kind = "jina"
api_key_env = "JINA_API_KEY"

[[providers.models]]
id = "jina-rerank"
upstream_id = "jina-reranker-v2-base-multilingual"
capabilities = ["rerank"]

voyage

• kind: voyage · capabilities: embed, rerank (hosted).
• Auth: api_key_env (e.g. VOYAGE_API_KEY), bearer token.
• Embed batch limit: 128.

[[providers]]
name = "voyage"
kind = "voyage"
api_key_env = "VOYAGE_API_KEY"

[[providers.models]]
id = "voyage-rerank"
upstream_id = "rerank-2"
capabilities = ["rerank"]

mixedbread

• kind: mixedbread · capabilities: rerank (hosted, mxbai-rerank-*).
• Auth: api_key_env (e.g. MXBAI_API_KEY), bearer token.
• base_url: optional; defaults to https://api.mixedbread.com/v1.
• Schema note: Mixedbread’s endpoint is POST /v1/reranking (note the path: reranking, not rerank) and renames the request fields (input instead of documents, top_k instead of top_n) with results nested under data; the gateway translates transparently.
• Usage: billed in tokens, so the gateway reports an estimated token count (ADR 003) rather than upstream search units.

[[providers]]
name = "mixedbread"
kind = "mixedbread"
api_key_env = "MXBAI_API_KEY"

[[providers.models]]
id = "mxbai-rerank"
upstream_id = "mixedbread-ai/mxbai-rerank-large-v1"
capabilities = ["rerank"]

pinecone

• kind: pinecone · capabilities: rerank (hosted inference).
• Auth: api_key_env (e.g. PINECONE_API_KEY), sent as the Api-Key header (not a bearer token), alongside a pinned X-Pinecone-API-Version header the inference API requires.
• base_url: optional; defaults to https://api.pinecone.io.
• Schema note: documents are sent as { "text": ... } objects; only the default text rank field is used (rank_fields selection is out of scope for v1).
• Usage: Pinecone reports usage.rerank_units, carried through verbatim as the response’s search_units (not estimated).

[[providers]]
name = "pinecone"
kind = "pinecone"
api_key_env = "PINECONE_API_KEY"

[[providers.models]]
id = "pinecone-rerank"
upstream_id = "pinecone-rerank-v0"
capabilities = ["rerank"]

nvidia (NIM)

• kind: nvidia · capabilities: rerank (NVIDIA NIM ranking).
• Auth: keyless by default (self-hosted NIMs run without a key); supply api_key_env (e.g. NVIDIA_API_KEY) for the hosted API, sent as a bearer token.
• base_url: required - the NIM root (e.g. http://localhost:8000 or the NVIDIA-hosted ranking endpoint root). The gateway posts to {base}/v1/ranking.
• Schema note: the request nests query: { text } and passages: [{ text }]; there is no top_n on the wire, so the gateway requests the full ranking and truncates to top_n afterwards (as for TEI).
• Score semantics: NIM returns a raw logit, passed through unchanged as relevance_score. Scores are unbounded (can be negative) and are only comparable within a single response; higher is more relevant. No sigmoid is applied.
• Usage: NIM reports no token usage, so the gateway reports an estimated token count (ADR 003).

[[providers]]
name = "nvidia-nim"
kind = "nvidia"
base_url = "http://localhost:8000"
# api_key_env = "NVIDIA_API_KEY"   # only for the hosted API

[[providers.models]]
id = "nvidia-rerank"
upstream_id = "nvidia/llama-3.2-nv-rerankqa-1b-v2"
capabilities = ["rerank"]

together (rerank)

The together kind (see the OpenAI-compatible section for chat/embed) also serves rerank (LlamaRank) natively through Together’s Cohere-shaped /rerank endpoint. One [[providers]] entry with kind = "together" serves all three capabilities against the same base_url and bearer key. Rerank is billed in tokens, so the gateway reports an estimated token count (ADR 003).

[[providers]]
name = "together"
kind = "together"
api_key_env = "TOGETHER_API_KEY"

[[providers.models]]
id = "llama-rank"
upstream_id = "Salesforce/Llama-Rank-V1"
capabilities = ["rerank"]

tei (self-hosted)

• kind: tei · capabilities: embed, rerank.
• Auth: keyless.
• base_url: required - points at your Text Embeddings Inference server. TEI serves one model per process, so upstream_id is ignored by the upstream but kept for your own clarity.
• Embed batch limit: 32.

[[providers]]
name = "tei-local"
kind = "tei"
base_url = "http://localhost:8081"

[[providers.models]]
id = "bge-reranker"
upstream_id = "BAAI/bge-reranker-large"
capabilities = ["rerank"]

ollama (self-hosted)

• kind: ollama · capabilities: chat, embed.
• Auth: keyless.
• base_url: required - points at your Ollama server root (no /v1). Embeddings use Ollama’s native POST /api/embed; chat goes through Ollama’s OpenAI-compatible endpoint, which lives under /v1 on the same root - the gateway appends the /v1 itself, so keep base_url as the bare server root either way.
• Chat: served by the shared OpenAI-compatible path - streaming (SSE passthrough), cancellation, and token accounting (upstream usage when Ollama reports it, otherwise a local count marked estimated, ADR 003) all work exactly as for the openai kind.
• api_key asymmetry: if you set an api_key_env on this kind (e.g. an Ollama server behind an authenticated reverse proxy), the chat path sends it as a bearer token but the native embed path currently sends no Authorization header at all - keep the embed route unauthenticated at the proxy, or front only /v1 with auth.
• Embed batch limit: 512.
• Tip: a local model may take a while to load into VRAM on its first call - relax first_token_timeout_ms / total_timeout_ms on the provider block (see config.example.toml). A self-hosted box on a slow link can also override connect_timeout_ms; note that doing so gives this provider its own (unpooled) HTTP client (ADR 005, 2026-07-15 amendment), whereas the first-token and total overrides do not. All three fall back to their global defaults when unset.

[[providers]]
name = "ollama-local"
kind = "ollama"
base_url = "http://localhost:11434"
first_token_timeout_ms = 60000
total_timeout_ms = 120000
connect_timeout_ms = 10000  # optional: own client, relaxed connect deadline

[[providers.models]]
id = "local-llama"
upstream_id = "llama3.2"
capabilities = ["chat"]

[[providers.models]]
id = "nomic-embed"
upstream_id = "nomic-embed-text"
capabilities = ["embed"]

azure

• kind: azure · capabilities: chat, embed. Reuses the OpenAI JSON schema verbatim; only the URL, auth, and routing differ from openai.
• Auth: api_key_env, sent as the api-key header (never a bearer token).
• base_url: required - your Azure resource endpoint, e.g. https://<resource>.openai.azure.com (no shared public default, every resource is operator-specific).
• api_version: optional - pins the Azure API version sent as the api-version query parameter on every request (issue #65). For back-compat the older form still works: append ?api-version=YYYY-MM-DD to base_url. Precedence: the explicit api_version field wins over a base_url query string, which wins over LUMEN’s pinned built-in default (see the azure module doc comment for the exact value). Any query parameters on base_url other than api-version are ignored when building request URLs. api_version is azure-only: setting it on any other kind is rejected at boot.
• Deployment routing: Azure routes by URL path (/openai/deployments/{deployment}/...), not by the model field in the body. Set each model’s upstream_id to the Azure deployment name - the same upstream_id mechanism every other kind uses for aliasing already carries it through.
• Embed batch limit: 2048 (same array-size ceiling as the OpenAI embedding models Azure hosts).

[[providers]]
name = "azure-openai"
kind = "azure"
api_key_env = "AZURE_OPENAI_API_KEY"
base_url = "https://my-resource.openai.azure.com"
api_version = "2024-10-21"

[[providers.models]]
id = "gpt-4o"
upstream_id = "my-gpt4o-deployment"   # the Azure deployment name
capabilities = ["chat"]

[[providers.models]]
id = "azure-embed"
upstream_id = "my-embedding-deployment"
capabilities = ["embed"]

OpenAI-compatible hosts

groq, together, fireworks, deepseek, openrouter, perplexity, xai, deepinfra and huggingface all work the same way: set the kind, point api_key_env at the host’s token, and (optionally) override base_url. The built-in default base URL is used otherwise.

[[providers]]
name = "groq"
kind = "groq"
api_key_env = "GROQ_API_KEY"
[[providers.models]]
id = "fast"
upstream_id = "llama-3.3-70b-versatile"
capabilities = ["chat"]

huggingface

The OpenAI-compatible Inference router (https://router.huggingface.co/v1), distinct from the self-hosted tei kind. api_key_env holds a Hugging Face token; upstream_id is a routed model id (often owner/model:provider).

[[providers]]
name = "hf"
kind = "huggingface"
api_key_env = "HF_TOKEN"
[[providers.models]]
id = "qwen"
upstream_id = "Qwen/Qwen2.5-72B-Instruct"
capabilities = ["chat"]

cloudflare

Cloudflare Workers AI. Chat and embeddings go through its OpenAI-compatible endpoint; reranking (bge-reranker-* models) goes through Workers AI’s own native POST /ai/run/{model} endpoint instead, since it is not part of the OpenAI-compatible surface - one [[providers]] entry serves all three capabilities against the same base_url. base_url is required because it embeds your account id; api_key_env holds a Cloudflare API token.

The native rerank request is { query, contexts: [{ text }, ...], top_k } (top_n is sent as top_k); the response is Cloudflare’s standard { result: { response: [{ id, score }, ...] }, success, errors } envelope, with id mapped back onto the original document index. Workers AI reports no token usage for this model; LUMEN derives a local estimate per ADR 003.

[[providers]]
name = "cf"
kind = "cloudflare"
api_key_env = "CLOUDFLARE_API_TOKEN"
base_url = "https://api.cloudflare.com/client/v4/accounts/YOUR_ACCOUNT_ID/ai/v1"
[[providers.models]]
id = "cf-llama"
upstream_id = "@cf/meta/llama-3.1-8b-instruct"
capabilities = ["chat"]
[[providers.models]]
id = "cf-rerank"
upstream_id = "@cf/baai/bge-reranker-base"
capabilities = ["rerank"]

vllm

Any self-hosted OpenAI-compatible server (vLLM, llama.cpp --api, LM Studio, SGLang, LocalAI). base_url required, API key optional. For Ollama, prefer the native ollama kind (chat + embed, see its section above); its OpenAI-compatible endpoint (http://localhost:11434/v1) also works under this kind, but you lose the native embed path and the /api/version health probe.

[[providers]]
name = "local"
kind = "vllm"
base_url = "http://localhost:8000/v1"
[[providers.models]]
id = "local-llama"
upstream_id = "meta-llama/Llama-3.1-8B-Instruct"
capabilities = ["chat", "embed"]

OpenAI chat extras on translated providers

OpenAI-compatible kinds forward every unmodeled request field verbatim, so response_format, seed, logprobs, top_logprobs, logit_bias, parallel_tool_calls, frequency_penalty and presence_penalty simply work there. The translated chat kinds rebuild the upstream request field by field, so each of these is either mapped onto a native equivalent or explicitly unsupported (issues #72, #91) - never silently lost:

¹ {"type": "json_object"} becomes generationConfig.responseMimeType: "application/json"; {"type": "json_schema"} additionally carries json_schema.schema as generationConfig.responseSchema, with JSON Schema keywords Gemini’s OpenAPI-subset schema rejects (additionalProperties, $schema) stripped recursively. Note that additionalProperties: false is therefore dropped, not enforced, on Gemini: the model may still emit extra keys (each strip is logged at debug level).

² Cohere v2 has no separate json_schema type: OpenAI’s {"type": "json_schema", "json_schema": {"schema": ...}} collapses onto Cohere’s {"type": "json_object", "json_schema": <schema>}; json_object and text pass through as-is.

³ Cohere v2 does accept a logprobs flag upstream, but its response shape is not translated back to OpenAI’s (and top_logprobs rides that same response shape), so the gateway treats both as unsupported rather than returning a malformed response.

⁴ parallel_tool_calls: false becomes Anthropic’s tool_choice.disable_parallel_tool_use: true (defaulting the choice to auto when the request carries tools but no explicit tool_choice); true is the default on both sides and needs no wire field.

What happens to an unsupported field depends on the provider’s strict flag (the same switch Ollama uses for embeddings dimensions, issue #25):

• strict = false (default): the field is dropped and a debug-level log line names the provider and field.
• strict = true: the request is rejected before any upstream call with an honest 400 (LM-1001) naming the field and provider - never a misleading 5xx.

[[providers]]
name = "anthropic"
kind = "anthropic"
api_key_env = "ANTHROPIC_API_KEY"
strict = true   # reject response_format/seed/logprobs instead of dropping

An unrecognised response_format shape (an unknown type value) is dropped with a debug log on the mapping providers, matching how unknown tool_choice shapes are handled: dropped, not guessed.

Vision (image input)

POST /v1/chat/completions accepts OpenAI’s content-parts message shape, so a user message can carry text and image parts in one array:

{
  "model": "gpt-4o",
  "messages": [{
    "role": "user",
    "content": [
      { "type": "text", "text": "What is this?" },
      { "type": "image_url", "image_url": { "url": "data:image/png;base64,iVBORw0KG..." } }
    ]
  }]
}

image_url.url is either a data:<media-type>;base64,<payload> URI (inline bytes) or a remote http(s) URL.

Per-model opt-in. A model only accepts image parts once its config declares the image modality (default is ["text"]):

[[providers.models]]
id = "gpt-4o"
capabilities = ["chat"]
modalities = ["text", "image"]   # opts this model into vision

GET /v1/models reflects this back as "modalities": ["text","image"] per model. Sending an image part to a model whose modalities lack "image" is rejected with LM-2003 (400, see docs/errors.md) before any upstream call.

Which kinds support it:

Never-fetch rule. LUMEN never dereferences a user-supplied image URL itself - doing so would be an SSRF vector (the gateway could be aimed at internal addresses) and would violate the streaming/latency pillar. A remote http(s) image_url is only ever forwarded to a provider that fetches it itself (OpenAI, Anthropic); Gemini’s inline_data field takes only inline bytes, so a remote URL routed to Gemini is rejected with LM-2004 (400) instead of the gateway silently fetching it on the caller’s behalf.

The LM-2004 pre-flight check inspects the primary provider of the model’s fallback chain. In the uncommon case where the primary accepts remote URLs (e.g. OpenAI) but a Gemini model is configured as a fallback, a request with a remote image URL passes pre-flight and, only if the primary then fails over to Gemini, surfaces as an upstream LM-3002 (502) - the gateway still never fetches the URL. Configure inline data: URIs when a Gemini fallback is in play.

Provider-native image sources (issue #12). Two provider-native reference forms are recognised in the image_url.url field, for callers whose images are already uploaded to the provider:

A provider-native reference routed to a model whose primary provider is not the reference’s own provider is rejected pre-flight with LM-2008 (400) - an honest client error instead of a confusing upstream failure. A Gemini Files API URI is also an https:// URL, but it is exempt from the LM-2004 remote-URL check: it is not a URL the provider would have to fetch, and its routing verdict belongs to the LM-2008 check.

gs:// caveat. The gateway forwards a gs:// URI to Gemini verbatim, but the Gemini Developer API (generativelanguage.googleapis.com, the default base_url of the google kind) documents fileData.fileUri for its own Files API URIs; Cloud Storage gs:// URIs are a Vertex AI capability. Against the default endpoint a gs:// reference is passed through and will be rejected by the upstream (surfacing as an upstream error naming google). It is still parsed and forwarded because the reference form is Gemini-native (mismatch routing stays an honest LM-2008), base_url may point at a Vertex-compatible gateway, and the upstream - never the gateway - is the authority on which URI forms it accepts. Upload via the Gemini Files API and pass the returned URI when targeting the Developer API.

For the mime type of a fileData part: it is included only when it can be confidently inferred from the URI’s file extension (.png, .jpg, …); otherwise it is omitted rather than guessed. Files API URIs carry no extension, and Gemini already knows the mime type recorded at upload time.

Accounting. Upstream-reported usage is authoritative and already folds in image tokens. When an upstream reports no usage at all, the local estimation fallback counts each image content part with a flat per-image heuristic (85 tokens at "detail": "low", 765 tokens otherwise) rather than counting it as zero, and the response is still flagged "estimated": true - see the ADR 003 addendum.

Fallbacks across providers

Any model can name an ordered list of fallbacks - models that back it when its provider exhausts retries or its circuit is open. Each fallback must exist and serve every capability of the model it backs (validated at boot), which lets you survive a single-vendor outage by spanning providers:

[[providers.models]]
id = "gpt-4o"
capabilities = ["chat"]
fallbacks = ["claude-sonnet-4-5"]     # different vendor, same capability

See docs/adr/005-resilience-execution.md for the resolution and circuit-breaker details, and config.example.toml for a fully worked multi-provider setup including a three-vendor rerank fallback chain.

Error codes

Every error LUMEN returns to a client carries a stable LM-XXXX code, an HTTP status, and a coarse type. The response body is always:

{ "error": { "code": "LM-1001", "message": "…", "type": "invalid_request" } }

The type is one of invalid_request, upstream_error, internal, or client_cancelled. The gateway always distinguishes these situations and never disguises one as another - in particular, an internal malfunction is never reported as a misleading 401 (a lesson from OpenRouter outages), a malformed upstream response is a 502, never a gateway 500, and a client-initiated cancel (client_cancelled) is never counted as an internal malfunction either (see LM-6xxx below).

Codes are stable: once assigned, a code keeps its meaning across releases. The code prefix groups by cause: 1xxx request, 2xxx routing, 3xxx upstream, 4xxx auth/budget, 5xxx internal, 6xxx client-cancellation.

Request errors - LM-1xxx · type: invalid_request

Routing & capability-request errors - LM-2xxx · type: invalid_request

Upstream errors - LM-3xxx · type: upstream_error

These always name the provider that failed. Retriable ones may be transparently retried on a fallback before surfacing.

For LM-3001, LM-3020 (and LM-4002/LM-4003), a Retry-After value may be advertised. The three timeouts (LM-3011 first-token, LM-3012 connect, LM-3013 total) are distinct codes purely for debugging - see §6.4 and docs/adr/005-resilience-execution.md.

How resilience shapes these codes

The 3xxx codes are what a client sees only after the resilience machinery has given up. Before surfacing, a retryable failure (LM-3001 429, LM-3003 5xx, LM-3005/LM-3012 timeouts) is retried with exponential backoff, then the request fails over to the model’s configured fallbacks. The mapping between a failure and the code that eventually surfaces:

• LM-3020 (503) - the primary’s circuit is open and no fallback remained. Skipping an open circuit is instant (no upstream call), and the response carries a Retry-After equal to the cooldown remainder.
• LM-3004 (503) - every link in the fallback chain was tried and failed (retries exhausted or circuits open all the way down).
• LM-3013 (504) - the total per-request deadline elapsed while retrying or failing over; it bounds all attempts together, so a slow chain fails here rather than hanging.
• LM-3011 / LM-3012 (504) - first-token and connect timeouts; each is a retryable failure on its own before it surfaces.

A hard upstream client error (a 4xx bad request) is never retried or failed over - a different provider would reject it too - and surfaces immediately. Whichever model ultimately served a successful request is reported in the x-lumen-model-used response header.

Auth / budget errors - LM-4xxx · type: invalid_request

Codes pinned by the spec. Enforcement happens in memory, before any upstream call - a rejected request never leaks spend to a provider.

Internal errors - LM-5xxx · type: internal

Internal errors return an opaque "internal error" message to the client; the underlying detail is written only to the server logs, never the response.

Client-cancellation - LM-6xxx · type: client_cancelled

499 is the conventional “client closed request” status (nginx). The client is normally already gone by the time this would be returned, so the status exists for logs and metrics, not for anything a client reads. It is deliberately kept out of both type: internal and the 5xx status class: a client hanging up is not a gateway malfunction, and must never inflate the internal-error metrics or alerts a real one would (issue #11, see docs/adr/006-client-cancellation-error-code.md).

Two paths produce it. A cancellation surfacing mid-stream is emitted as a terminal SSE error frame carrying this envelope. A client that simply disconnects mid-stream never sees a frame at all, but the request’s accounting record (usage_log.status and the lumen_request_duration_seconds{status="499"} sample) is settled at 499 instead of being miscounted as a 200 success. A non-streaming disconnect drops the request before any outcome is recorded and produces no sample.

Performance baseline

LUMEN’s first pillar is performance: < 1 ms added p99 off-network, ~15 MB idle RAM, streaming that doesn’t re-serialize. This document records how those promises are measured, the numbers obtained, and how to reproduce every figure. Where a target is not fully measured in a given environment, the gap is stated honestly rather than papered over.

Methodology

Two layers, because “added latency” has two very different scales:

1. In-process overhead (cargo bench) - the CPU work the gateway adds per request, with no sockets involved: the resilience executor wrapping a provider call (circuit-breaker admit → retry loop → per-attempt timeout) plus the JSON (de)serialization the OpenAI surface does. This is the honest measure of “added latency off-network”: it excludes the upstream and the network entirely. Criterion, warmed, reports median with a 95 % CI.

2. End-to-end head-to-head (bench/, docker-compose + k6) - LUMEN and LiteLLM both proxying the same zero-latency mock upstream, driven by k6. Added latency = gateway percentile − direct-to-mock percentile. This is the number a user feels; it includes one extra localhost hop. Provided as a reproducible harness (see bench/README.md). Reported at two marks per target: total request time (http_req_duration) and time to first byte (http_req_waiting: request fully written → first response byte).

3. Streaming time to first bit (cargo bench -p server --bench stream_ttfb) - the one latency k6 cannot see: how much later a client receives the first SSE chunk of a stream: true response because the gateway sits in the middle. Real sockets, full LUMEN stack (axum → router → OpenAI-kind provider), instant mock upstream; the same request is timed direct-to-upstream and via-gateway, and the difference between the two distributions is the gateway’s added streaming TTFB. The companion integration test (tests/chat.rs::first_stream_chunk_reaches_the_client_before_the_upstream_finishes) proves the first frame is forwarded while the upstream is verifiably still mid-stream (gated tail, no timing races), so this bench measures eager forwarding, not buffer-then-flush.

Environment of the recorded run

Numbers are hardware-specific; re-run the commands on your target to get yours.

Results - in-process overhead (measured here)

Total added CPU per non-streaming chat request ≈ 3.2 µs (executor + parse + serialize). Streaming passthrough adds even less per chunk: it forwards upstream Bytes verbatim with no per-chunk serde (ADR 004), so the per-chunk cost is a bounded copy plus the [DONE]/heartbeat scan, not a deserialize.

Streaming time to first bit (measured here)

Recorded with cargo bench -p server --bench stream_ttfb in the same environment as above: the span from dispatching a stream: true chat request to reading the first bytes of the SSE body, over real loopback sockets, against an instant mock upstream.

Added streaming TTFB ≈ 97 µs median (~0.1 ms): the extra wait before a client sees the first streamed token because LUMEN sits in the middle. That buys one full extra HTTP hop (accept, parse, route, provider request build, upstream connect-pooled call, headers + first-frame forward), measured end to end, and still lands an order of magnitude inside the < 1 ms pillar. The companion integration test (methodology point 3 above) guarantees the number means what it says: the first frame is forwarded while the upstream is still mid-stream, never buffered until end-of-stream.

Idle memory & binary size (measured here)

The Docker image was built from Dockerfile and smoke-tested: docker run -v config.toml -e OPENAI_API_KEY … answers /health 200 and /v1/models. At 10.6 MB it is well under the 30 MB image budget (§7.2). The amd64 image is built in CI via buildx (release.yml).

Results - loaded head-to-head vs LiteLLM (recorded baseline)

Recorded by bench/run.sh (see bench/README.md); full raw output is committed at bench/results/20260715T231135Z/.

Time to first byte for the same run (http_req_waiting; derived from the committed raw *.summary.json of that run - the metric was always recorded, its report table was added later, so the run’s own report.md predates it):

TTFB tracks total duration almost exactly in this scenario (non-streaming, tiny mock body: once the first byte is out, the rest follows within microseconds), so it carries the same caveat and the same conclusion: LUMEN delays the start of the response by ~2.6 ms at p50 on this noisy host, LiteLLM by ~317 ms.

RAM under load (docker stats, sampled mid-run): lumen ~7.6 MB, litellm ~1.03 GB.

Environment this specific run was recorded in: Darwin arm64, Docker 29.4.0, k6 v2.0.0, LUMEN at commit 51fc809, LiteLLM ghcr.io/berriai/litellm:v1.92.0@sha256:9ef6f45bc0104940571765e610c52a1d761b5ec85efcd193795281086ee61277, mockserver 5.15.0@sha256:0f9ef78c94894ac3e70135d156193b25e23872575d58e2228344964273b4af6b.

Caveat, stated honestly: this run was recorded on a shared development host (not dedicated benchmarking hardware), which is visible in the noisy direct-baseline numbers above (a 0 ms mock should not itself show ~836 ms p99; the mockserver JVM saturates a core by itself, so the direct phase measures host contention as much as transport). Treat the absolute numbers as illustrative, not authoritative. The relative comparison - lumen vs litellm, same mock, same host, same run - is the meaningful part: LUMEN added ~2.5 ms at p50 over direct (9.44 vs 6.91 ms) and its tail stayed below the noisy direct baseline, while LiteLLM’s p50 grew by ~47× (323.75 ms) and its p99 reached 6.5 s. LUMEN sustained ~25× LiteLLM’s throughput at roughly 1/140th the RAM. Re-run bench/run.sh on dedicated hardware for numbers to make capacity decisions on; see “Updating the pinned versions” in bench/README.md for how to refresh this baseline (new pinned image, new results/ directory, update the link above).

Targets

Honest summary: the off-network overhead is measured and is microseconds, comfortably inside the pillar-1 budget. The full loaded head-to-head vs LiteLLM is now a committed, reproducible baseline (pinned versions, one command, recorded result linked above) rather than just a runnable harness; the relative LUMEN-vs-LiteLLM comparison from it is solid, while the absolute and direct-vs-gateway numbers should be treated as this particular (noisy, shared) host’s numbers, not a hardware-independent claim.

Reproducing

# In-process overhead (no Docker needed):
cargo bench -p server --bench gateway_overhead

# Streaming time to first bit, direct vs via-gateway (no Docker needed):
cargo bench -p server --bench stream_ttfb

# Idle RAM + binary size:
cargo build --release -p server --bin lumen
./target/release/lumen --config config.example.toml &   # then: ps -o rss= -p <pid>

# Full head-to-head vs LiteLLM (Docker + k6, one command): see bench/README.md
bench/run.sh

ADR 001 - Bare package names, lumen_* library names

• Status: accepted
• Date: 2026-07-12

Context

The CLAUDE.md architecture lays out a six-crate workspace under crates/ (core, providers, router, auth, telemetry, server) and documents commands like cargo run -p server. That -p server selector requires the Cargo package name to be the bare server.

Naming a package core, however, is hazardous: a library crate literally named core lands in the extern prelude of any downstream crate and shadows the standard library’s ::core. This surfaces in the doctest harness, where ::core::fmt, ::core::future, etc. (referenced by expanded std/async_trait macros) fail to resolve - observed concretely as E0433: cannot find 'fmt' in 'core' while normal builds still passed.

Decision

Keep package names bare (core, providers, router, auth, telemetry, server) so the documented -p <name> commands work, but give each library crate an explicit lib name prefixed lumen_:

[package]
name = "core"

[lib]
name = "lumen_core"
path = "src/lib.rs"

Internal dependencies are wired in [workspace.dependencies] with the lumen-* key mapped to the bare package via package:

lumen-core = { path = "crates/core", package = "core" }

So: cargo run -p server works, imports read use lumen_core::…, and no crate shadows a std crate.

Consequences

• cargo run -p server -- --config … (and -p core, etc.) match the docs.
• No std-crate shadowing anywhere, including doctests.
• Slight indirection: the [workspace.dependencies] key differs from the package name. Documented here so it is not mistaken for an accident.
• The published crate names (if we ever publish) would be the bare names; we can revisit and prefix them at publish time without touching source.

ADR 002 - Per-request metadata header for logging & metrics

• Status: accepted (planned)
• Date: 2026-07-12

Context

Operators running a shared gateway need to attribute and slice observability data by dimensions the gateway cannot infer: which end-user, team, feature, environment or experiment a call belongs to. Cloudflare AI Gateway solves this with a cf-aig-metadata request header carrying a small JSON object that is then attached to each request’s log for filtering and search.

LUMEN needs the same capability, feeding both:

• structured logs (tracing) and the usage_log records, and
• Prometheus metrics.

The tension is Prometheus cardinality. Prometheus label values multiply the number of time series; putting arbitrary client-supplied metadata (e.g. a user_id) onto metric labels is an unbounded-cardinality footgun that violates pillar 1 (~15 MB idle, < 1 ms p99) - a few thousand distinct users would blow up memory and scrape cost. Cloudflare sidesteps this by indexing metadata for log search, not by turning it into metric dimensions.

We also refuse to make an observability header able to fail a real request: rejecting a chat call because someone sent malformed metadata is user-hostile.

And per the sovereignty pillar, metadata is logged, so it must never be treated as prompt content - it is operator/client labels only.

Decision

Introduce a per-request metadata header, parsed once at the edge into a small typed value carried in request extensions.

1. Header. Canonical x-lumen-metadata; also accept cf-aig-metadata as an alias so Cloudflare AI Gateway clients work unchanged. Value is a flat JSON object of string → (string | number | bool). Nested objects/arrays are rejected (dropped, see 4).

2. Bounds. At most 16 keys; key ≤ 64 bytes; value ≤ 256 bytes; whole header ≤ 4 KiB. These keep log records and memory bounded.

3. Two sinks, different rules.

   • Logs / usage_log: the full (bounded) object is attached to the request’s structured log fields and stored in a metadata column on usage_log for later filtering - the Cloudflare-style use case.
   • Prometheus: ONLY keys named in a config allowlist (telemetry.metadata_labels = ["env", "team"], default empty) become metric labels; every other key is logs-only. An allowlisted key absent from a given request gets the label value "". This makes metric cardinality a deliberate, operator-bounded decision - never client-driven.

4. Never fails the request. Missing, malformed, oversized or wrong-typed metadata is dropped with a debug!/warn! and a metadata_rejected_total counter increment; the call proceeds normally.

5. Opaque, never inspected. LUMEN does not parse metadata for meaning or PII. Documentation states plainly that metadata is logged and must not carry secrets or prompt content.

Consequences

• Metric cardinality is capped by config, not by traffic - safe by default (empty allowlist = zero new label dimensions).
• Full metadata is still available for rich filtering via usage_log, matching Cloudflare’s log-search model.
• The request path gains only a bounded header parse (no allocation when the header is absent), preserving the latency pillar.
• Implementation lands alongside usage logging (usage_log.metadata column, the batched writer, and the Prometheus label wiring in telemetry). A thin extractor in server reads the header into request extensions so chat, embeddings and rerank handlers all share it.

ADR 003 - Token accounting for every request, every capability

• Status: accepted (core promise)
• Date: 2026-07-12

Context

Token counting is a headline reason to run LUMEN: an operator fronting many providers wants one trustworthy answer to “how many tokens did this cost, per model / key / team / capability?” - without trusting each upstream to report it and without leaking prompts to a third party to find out.

The problem is that upstream usage reporting is inconsistent:

• OpenAI/Cohere/Voyage embeddings report input tokens; TEI reports none (its /embed is a bare vector array) - so a naive gateway shows 0 tokens for TEI embeddings.
• Streaming chat only carries usage if the client sends stream_options.include_usage (and some providers still omit it).
• Rerank is billed in search units by Cohere but in tokens by Jina/Voyage, and TEI reports neither.

So “count the tokens for all the APIs” cannot mean “pass through whatever the upstream says.” It means the gateway guarantees a count for every call, and is honest about whether that count is measured or estimated.

The hard constraint is pillar 1: < 1 ms added p99, no blocking the runtime. Real tokenizers (BPE) cost real CPU; running one inline on the request path would blow the latency budget.

Decision

Token accounting is a first-class, always-on output of every chat, embeddings and rerank call, produced by a two-tier strategy and surfaced in three places.

Source, in priority order

1. Upstream-reported usage - authoritative and free (already in the response body / final SSE chunk). Always preferred. estimated = false.
2. Local estimation fallback - when the upstream omits usage. estimated = true. Two levels:
   • default: a cheap, allocation-light heuristic (byte/char-based) that is safe to compute anywhere;
   • opt-in: an accurate tokenizer (tokenizers/tiktoken-style) selected per model in config, run via spawn_blocking so it never occupies a tokio worker.

Hot-path rule

The request path never runs a heavy tokenizer. Upstream usage is passed through as-is; when it is missing, accurate estimation happens off the hot path in the async usage-writer task. The response usage field carries the upstream value when present, else the cheap heuristic (flagged estimated), never a blocking BPE pass. Counting must never fail or slow a request.

What is counted, per capability

• Chat: input_tokens (prompt) and output_tokens (completion).
• Embeddings: input_tokens.
• Rerank: search_units when the provider reports them, plus a token count of query + documents for uniform observability (estimated when derived).

Three surfaces

1. Response body - OpenAI-compatible usage (chat/embeddings) unchanged.
2. Prometheus - cumulative counters, low fixed cardinality: lumen_tokens_total{capability, model, provider, direction, estimated} and lumen_rerank_search_units_total{model, provider}. Optional metadata-allowlist labels come from ADR 002 (never client-unbounded).
3. usage_log - per-request tokens_in, tokens_out, search_units, estimated, alongside cost and metadata for later slicing.

Consequences

• Every call gets a token count regardless of provider - the TEI-reports-nothing gap is closed by estimation, and the count is labelled honestly.
• The latency pillar holds: passthrough is free; BPE estimation is off-hot-path and on the blocking pool.
• The embeddings and rerank paths already surface upstream-reported usage; this ADR adds the estimation fallback and the Prometheus/usage_log counters plus streaming extraction. Cost counting (§5.4) becomes a consumer of these token counts rather than the thing that defines them.
• New error/counter surface: a tokens_estimated_total counter lets operators see how much of their accounting is measured vs estimated.

Addendum (M8 - vision / image input)

Image content parts ({"type":"image_url",...}) do not change the priority order above; they sharpen what “estimation” means when tier 2 fires.

• Upstream usage stays authoritative and untouched. OpenAI, Anthropic and Gemini all fold image tokens into their reported prompt_tokens, so a vision request with upstream-reported usage is exactly as accurate as a text-only one - no special-casing needed.
• The local estimation fallback counts text plus a flat per-image estimate. When the upstream omits usage, the heuristic estimator (estimate_chat_prompt, crates/core/src/tokens.rs) sums MessageContent::text() per message (the concatenation of text parts) plus, for every image_url part, a flat per-image token constant chosen from the part’s detail hint: "low" -> 85 tokens (OpenAI’s exact, resolution-independent low-detail cost - no dimensions needed to reproduce it); "high"/"auto"/unset -> 765 tokens, an approximation of OpenAI’s 85 + 170 * tiles tile formula for a typical ~1024x1024 image. The response is still flagged "estimated": true, so the client is never told a number is measured when it is not.
• A true per-dimension tile count is still deferred - see docs/backlog.md. OpenAI’s real high-detail formula depends on decoded pixel dimensions, which a data: URI does not carry and this gateway does not extract (the hot-path rule above - never decode/inspect image bytes on the request path - still holds). The flat 765-token constant is a documented approximation, not an attempt at per-image precision; it trades some accuracy for closing the “silently counts as 0” gap entirely.

Addendum (issue #10 - rerank token usage shape)

RerankUsage (§ “What is counted, per capability”) carries search_units and total_tokens as two independent counts, each with its own *_estimated flag (estimated for search_units, tokens_estimated for total_tokens) rather than one shared flag - a response can have a real search_units (Cohere) alongside a derived total_tokens, or the reverse (Jina/Voyage). The priority order from the “Source” section above applies per count: Jina and Voyage report usage.total_tokens in their rerank response, which the gateway passes through unflagged; when a provider omits it (Cohere, TEI, or Jina/Voyage without usage), the gateway derives total_tokens from query + documents via the existing byte-heuristic estimator, flagged "tokens_estimated": true.

Addendum (M9 - multimodal embeddings)

The same priority order applies to image content parts in /v1/embeddings: upstream usage is trusted when reported (Cohere, Voyage and Jina all fold image cost into their reported token/usage counts). When the upstream reports nothing, the local fallback estimates text parts only (image parts contribute 0 tokens) and the response is still flagged estimated: true. This undercounts image-heavy requests on a no-usage upstream; a per-image token heuristic is a backlog item (see the ROADMAP M9 note). Media volume itself is accounted separately (count + decoded bytes) via lumen_media_total / lumen_media_bytes_total and the usage_log media_count/media_bytes columns, not through the token counters.

Addendum (opt-in accurate tokenizer)

The “opt-in accurate tokenizer” tier promised above is now implemented, behind a config knob, without disturbing the default heuristic path.

• Config. A new [tokenizer] mode = "heuristic" | "accurate" selects the local estimation strategy (default heuristic). The knob is global, not per-model: accurate mode already keys off the model id internally (below), so a single switch covers a mixed fleet without per-model config.
• Encoder. Accurate mode uses tiktoken-rs (pure Rust, MIT, no OpenSSL) and picks the vocabulary by model prefix: cl100k_base for gpt-4 / gpt-3.5 / text-embedding-3, o200k_base for gpt-4o / o1 / o3 / gpt-4.1 / gpt-5. A model that matches no OpenAI-family prefix (Claude, Mistral, Llama, TEI models, …) keeps the byte heuristic - tiktoken does not describe those vocabularies, so a BPE count would be a false precision.
• Hot-path rule, upheld. The heuristic is unchanged: free, inline, used for the pre-call budget admission estimate on every request AND for the response envelope. The response usage field always carries the cheap heuristic (flagged estimated) - never a BPE pass, never a wait - exactly as this ADR’s hot-path rule states. The accurate BPE count is computed only when tier 2 fires (an upstream reported no usage) and after the response is handed off: the handler settles the envelope with the heuristic, then defers the accounting close to a spawned background task that recounts with exact BPE on the blocking pool via spawn_blocking (never on a tokio worker, repo rule 2) and finishes the record. The accurate number therefore surfaces in Prometheus and usage_log (the operator-facing accounting surfaces), not in the response envelope, and the client response is never delayed by refinement. The end-to-end latency histogram and usage_log.latency_ms are frozen at response time before the deferral, so they measure the request, not the refinement. Encoders are built once at config load, never lazily on a request.
• Scope. Applied to the paths with a post-response settlement: chat (non-streaming), embeddings, and rerank. Rerank tokens remain always gateway-estimated (uniform observability, above) whether or not the upstream billed search units; its refinement follows the same deferred path and is usually a no-op, since real rerank model ids rarely carry an OpenAI tiktoken family prefix. Streaming input counts stay on the heuristic - the stream is forwarded, never buffered (ADR 004), so there is no assembled prompt/response to BPE without violating the passthrough rule.
• Honesty preserved. Accurate local counts are still flagged estimated = true. A local count, however exact the tokenizer, is an estimate of what the upstream would bill; upstream-reported usage always wins (estimated = false). On any tokenizer failure the deferred close settles with the heuristic numbers, so counting can never fail or reject a request. Consequence to be aware of: when refinement fires, the response envelope (heuristic) and the accounting surfaces (accurate) intentionally differ - the envelope is the hot-path-safe estimate, the accounting surfaces are the billing-grade one.

ADR 004 - Zero-copy SSE streaming vs. the typed chunk trait

• Status: accepted
• Date: 2026-07-12

Context

CLAUDE.md sketches ChatProvider::chat_stream -> BoxStream<ChatChunk>: a stream of typed chunks. But the streaming spec (§4.2) demands zero-copy passthrough - when the upstream already speaks OpenAI SSE (OpenAI, Mistral, Ollama, vLLM), the gateway must forward the SSE frames as Bytes without deserializing each chunk. Deserializing a ChatChunk and re-serializing it per frame is exactly the per-token allocation overhead that makes LiteLLM 1.7–4× slower; it violates pillar 1 (< 1 ms added p99).

So the typed stream and the zero-copy requirement pull in opposite directions. Translating providers (Anthropic, Gemini) genuinely need typed chunks - they build OpenAI chunks from a foreign event schema. Passthrough providers must not pay for typing they don’t need.

Decision

Both paths converge on a single server contract: a provider yields the complete SSE response body as a Bytes stream - framing (data: …\n\n) and the terminal data: [DONE]\n\n included. The server pipes that byte stream straight into the HTTP response body; it does not re-frame, and for passthrough it does not deserialize.

Concretely, add one method to ChatProvider:

#![allow(unused)]
fn main() {
async fn chat_stream_bytes(&self, req, cancel)
    -> Result<BoxStream<'static, Result<Bytes, ProviderError>>, ProviderError>;
}

with a default implementation that adapts the typed chat_stream: serialize each ChatChunk to data: {json}\n\n and append data: [DONE]\n\n. Providers that translate a foreign schema (Anthropic) inherit the default - correct, if not zero-copy, which is fine because they must build chunks anyway.

Passthrough providers (OpenAI, Mistral) override chat_stream_bytes: set stream: true upstream, send, and on success return reqwest::Response::bytes_stream() mapped to ProviderError - the upstream’s own bytes, verbatim, [DONE] and all. No serde round-trip on the hot path.

Errors and cancellation

• Failures before the stream (non-2xx status, transport) surface as a normal Err → JSON error envelope, exactly like non-streaming. Only a mid-stream failure becomes an SSE error frame.
• The server holds the per-request cancel drop-guard inside the body stream, so a client disconnect drops the body, drops the guard and the underlying reqwest byte stream, closing the upstream connection promptly (the LiteLLM #22805 lesson). The initial send is wrapped in with_cancel too.

Usage / token accounting (ADR 003)

Pure passthrough does not deserialize, so streaming usage is sniffed off the final frame opportunistically (or estimated) in the token-accounting path - it must never block or re-serialize the passthrough path.

Consequences

• OpenAI/Mistral streaming is true zero-copy: upstream bytes → client bytes, one bounded copy through the socket, no per-chunk serde.
• Anthropic/Gemini keep the typed path and get correct (non-passthrough) SSE via the default adapter.
• The server’s streaming handler no longer uses axum’s Sse type; it writes a raw Bytes body with content-type: text/event-stream. SSE heartbeats move to an explicit injected frame rather than axum’s KeepAlive.
• core gains a bytes dependency (already ubiquitous via reqwest) for the trait’s return type.

ADR 005 - Resilience execution model (retries, fallback, circuit breaker, timeouts)

• Status: accepted (amended 2026-07-15: per-provider connect timeout and first-frame-peek streaming retry, see below)
• Date: 2026-07-13

Context

The resilience layer adds retries, multi-provider fallback chains, a per-provider circuit breaker and per-phase timeouts. The overriding constraint is pillar 1 (< 1 ms added p99) and pillar 3 (robustness): none of this may add a database or lock to the request path, and - the explicit lesson (LiteLLM #15526) - a storm of 429s from an upstream must never destabilise the gateway itself. The router crate was previously a pure resolution helper; this change turns it into an execution layer that wraps a provider call with the resilience machinery.

Three design tensions had to be resolved:

1. Where the machinery lives, given three capability traits. ChatProvider, EmbeddingProvider and RerankProvider have different signatures, so a single concrete executor cannot call all three. Duplicating retry/breaker logic per capability is unacceptable.
2. Streaming. Retry and fallback are only safe before the first byte reaches the client (spec 6.1/6.2/6.4). Once a frame is forwarded the request is committed.
3. Jitter vs. deterministic tests. Backoff needs randomness, but the acceptance criteria assert on elapsed (simulated) time.

Decision

One generic executor, capability-specific chain resolution

The executor is generic over a closure FnMut(link_index) -> Future<Result<T, ProviderError>>. It owns the resilience control flow - breaker gate, retry loop, fallback across links, total-timeout deadline - and knows nothing about chat/embed/rerank. Each handler resolves a chain (resolve_*_chain: the requested model followed by its configured fallbacks, each re-resolved for the same capability) and supplies a closure that performs the actual typed call for a given link. The chain the executor sees is metadata only (provider_name, model_id), which it uses to key the circuit breaker and to report the model that actually served (x-lumen-model-used).

Fallback chains are validated at boot: every fallback id must exist and serve the same capability as the model it backs (spec 6.2). A runtime resolution miss is therefore not expected, but is treated as a skipped link rather than a panic.

Retry classification lives on ProviderError

ProviderError::is_retryable() (5xx, connect/read timeout, 429, unreachable - never a 4xx client fault) and is_provider_fault() (does this failure indicate the provider is unhealthy, i.e. should it count against the breaker) are the single source of truth, shared by the retry loop and the breaker. A hard upstream 4xx (bad request) is neither retried nor failed over - a fallback provider would reject it too - and is returned immediately.

Backoff: pure function + injected randomness

backoff_delay(attempt, policy, retry_after, rand01) is a pure function: exponential base·2ⁿ capped at max, equal jitter (d = e/2 + e/2·rand01, so d ∈ [e/2, e]), then floored at Retry-After when the upstream sent a longer one. Production passes rand01 from a cheap lock-free splitmix64 (no dependency, no blocking, no Instant::now on the hot path); tests call the pure function with fixed fractions and assert exact bounds. The equal-jitter floor (e/2) makes “backoff delays respected” assertions robust regardless of the random draw, and the Retry-After floor makes criterion 6 (Retry-After: 3 ⇒ ≥ 3 s) hold unconditionally.

Circuit breaker: in-memory, per (provider, model)

A lock-free-ish CircuitBreaker (a Mutex<small struct> per key, never held across an .await) transitions Closed → Open (after N consecutive provider-fault failures) → Half-Open (after the cooldown) → one probe → Closed/Open. Concurrent requests that find the breaker Half-Open are refused the probe (treated as Open) so exactly one request probes. State is pushed to a Prometheus gauge lumen_circuit_state{provider,model} (0 closed / 1 open / 2 half-open) on every transition - the telemetry crate exposes a numeric setter so router depends on telemetry with no cycle. The breaker map is a DashMap, entries created on first use; bounded by the (provider × model) count, which is operator-configured and finite. Never touched off the request path by a blocking call - health checks read a snapshot.

A logical request records one breaker outcome per link (success, or one failure after that link’s retries are exhausted), so “5 consecutive failures” means five requests, not five retries within one.

Streaming: retry/fallback only at the open phase

execute_stream retries and falls back around opening the upstream byte stream (send + status check, before any body). Once the stream opens (upstream returned 2xx and we commit to forwarding), the existing to_event_stream guards (ADR 004: LM-3010 missing terminator, LM-3011 first-token, heartbeat) own the rest and never retry. This satisfies “retry only if no chunk emitted”: an open failure means nothing was forwarded, a post-open failure becomes a clean SSE error frame. (A 2xx-then-immediate-error is deliberately treated as committed - not retried - since the upstream accepted the request.)

One consequence, recorded explicitly: the circuit breaker for a streaming call only ever sees the open phase. on_success fires as soon as the byte stream opens, so a provider that opens cleanly but then dies mid-stream every time (LM-3010, handled by the frame guards and never surfaced back to the breaker) will not trip its circuit. This is the accepted trade-off of the open-phase boundary; the frame guards still give the client a clean terminal error each time.

Half-open cannot wedge. A half-open probe is normally resolved by on_success/on_failure, but a probe whose result never returns (client disconnect, the total-timeout firing mid-probe, or a non-provider-fault error that does neither) must not pin the breaker shut. The breaker records when a probe was admitted and auto-rearms: a probe still outstanding after a full cooldown is presumed lost and a fresh one is admitted. The single-probe guarantee therefore holds within a cooldown window, and recovery is always self-healing.

Timeouts and new error codes

Three timeouts, global defaults with per-model overrides:

• connect (default 5 s) - a reqwest::Client setting. Originally global only (per-provider connect would require one client per provider and lose connection pooling; deferred). Superseded by the 2026-07-15 amendment below: a provider may now override it with connect_timeout_ms, at the cost of its own (unpooled) client. A connect timeout is distinguished from a read timeout: ProviderError::ConnectTimeout → LM-3012 (504).
• first_token (default 30 s, per-model override) - reuses the streaming path; LM-3011 (504). For non-streaming it bounds the whole call; for streaming, the time to the first frame.
• total (default 600 s, per-model override) - an absolute deadline threaded through the executor bounding all retries and fallbacks together; exceeding it yields LM-3013 (504).

Circuit open with no fallback left is LM-3020 (503) carrying Retry-After (the cooldown remainder). LM-3004 (no healthy upstream) already existed and is kept for “all fallbacks exhausted”.

Health checks: optional, off the request path

A background task (default off) probes each provider that has a configured base_url (self-hosted TEI/Ollama, or any explicit override) with a short GET, storing Up/Down + latency in memory and a lumen_provider_up{provider} gauge. Providers relying on a built-in vendor URL report unknown - the gateway never hardcodes vendor endpoints. Results are exposed at /health/providers for observability; the gateway’s own /health stays completely independent of provider health (criterion 5) and does no I/O.

Consequences

• Handlers change from route.provider.chat(...) to execute_unary(chain, ..., |i| chain[i].provider.chat(...)); the resilience policy is uniform across all three capabilities and both streaming modes.
• router gains a dependency on telemetry (gauge) and on tokio/futures (it is now async). No dependency cycle.
• The circuit-breaker map and health results are process-wide in-memory state in AppState; nothing resilience-related touches SQLite on the request path.
• usage_log gains a model_used column (migration 0002) so a fallback is observable after the fact, mirroring the x-lumen-model-used header.
• Per-provider connect timeouts and true first-frame-peek streaming retries are explicitly out of scope here and recorded in docs/backlog.md. (Per-provider connect timeouts were subsequently implemented; see the amendment below.)

Amendment (2026-07-15): per-provider connect timeout

The original decision left connect global-only because a per-provider connect timeout is a reqwest::Client setting and one pooled client is shared across all providers, so overriding it per provider would mean one client per provider and the loss of cross-provider connection pooling. Issue #24 asked for it anyway (an upstream that is reliably reachable can afford a much tighter connect deadline than a flaky one; a distant self-hosted box may need a looser one), so the deferral is superseded.

Chosen design. The shared, pooled client remains the default and carries the global resilience.connect_timeout_ms. A provider that sets the new optional connect_timeout_ms (alongside the existing per-provider first_token_timeout_ms and total_timeout_ms) is given its own reqwest::Client, built once at registry construction, with that connect timeout and the same overall backstop as the shared client. Every provider that does not override keeps sharing the one pooled client, so pooling is preserved for the common case and only an explicitly-overriding provider pays for it.

Trade-off (documented, accepted). An overriding provider no longer shares the process-wide connection pool: its connections are pooled only within its own dedicated client. This is a per-provider, opt-in cost. Nothing else about the provider changes (same overall cap, same executor timeouts, same error codes).

Where it lives. ProviderSpec gains connect_timeout_ms: Option<u64>; Registry::build/reload take the overall backstop and build the dedicated clients in build_inner. Because the registry rebuilds all clients from the new specs on every hot reload, changing (adding, editing or removing) a provider’s connect_timeout_ms takes effect on SIGHUP/file-change reload with no restart, exactly like the other two per-provider timeout overrides. Config validation rejects a connect_timeout_ms of 0, matching the other overrides.

Amendment - 2026-07-15: first-frame-peek streaming retry (issue #7)

Supersedes the “2xx-then-immediate-error is deliberately treated as committed” carve-out above (the parenthetical in Streaming: retry/fallback only at the open phase) and the matching docs/backlog.md deferral. The rest of the original decision stands unchanged.

Decision

The commitment point for a streaming response moves from the open (2xx + headers) to the first content frame. After the open succeeds, the streaming closure PEEKS the first upstream frame before committing:

• first item is a content frame (Ok) - commit: the peeked frame is re-attached ahead of the untouched remainder (a single Bytes, moved not copied) and the reconstructed stream is handed to the existing to_event_stream guards. From here nothing retries (mid-stream errors still become a terminal SSE error frame - LM-3010/LM-3003 - exactly as before);
• first item is an error (Err), or the stream ends before any frame (None) - a pre-commit failure: the closure returns Err, so the executor retries, falls over to the next link, and charges the circuit breaker identically to an open failure. A None is surfaced as the new ProviderError::EmptyStream (retryable, provider-fault, mapped to LM-3010).

Boundaries and invariants preserved

• Zero-copy / bounded buffering (pillar 1, ADR 004). The peek buffers at most one frame, never the stream. The committed body is byte-identical to the upstream.
• Cancellation. The peek races the request CancellationToken; a client disconnect during the peek window returns ProviderError::Cancelled and drops the stream, aborting the upstream. Cancelled is neither retryable nor a provider fault, so no fallback is attempted and the breaker is untouched.
• Time bound. The peek runs inside the closure the executor already wraps in the per-attempt first_token timeout, so a silent upstream (headers, then no bytes) trips FirstTokenTimeout and fails over rather than hanging.
• Still non-retryable (unchanged): everything post-commit. Once a content frame is forwarded, a later mid-stream error, a missing [DONE] (LM-3010) or a first-token gap on a subsequent frame is a terminal SSE error, never a retry.

Consequence for the breaker note above

The recorded trade-off (“the breaker only ever sees the open phase”) is now narrower: a provider that opens 200 then fails on its first frame does now count against its breaker. Only failures after the first committed content frame remain invisible to the breaker (the frame guards still give the client a clean terminal error each time).

The peek lives in crates/router/src/peek.rs (generic over the frame type, unit-tested for commit / error / empty / cancel / timeout); wiremock acceptance tests are in crates/server/tests/resilience.rs.

ADR 006 - A dedicated error code for client-initiated cancellation

• Status: accepted
• Date: 2026-07-15

Context

ProviderError::Cancelled is produced when the per-request CancellationToken fires - normally because the client disconnected mid-request (ADR 004, M6). Since M1 this mapped to GatewayError::Internal("request cancelled"): HTTP 500, type: internal, LM-5001.

That mapping was flagged in docs/backlog.md at M1 (“revisit in M4”) and tracked as GitHub issue #11: a client hanging up is not a gateway malfunction, but reporting it as LM-5001/500 makes it indistinguishable from one in every place that matters operationally - the lumen_http_request_duration_seconds{status} / lumen_request_duration_seconds{status} Prometheus histograms, and any alert rule built on the 5xx or status="500" label. A busy gateway with many client cancels (slow clients, mobile networks, users navigating away mid-stream) would then look, to an operator, identical to a gateway that is actually failing.

The existing taxonomy (CLAUDE.md rule 8, docs/errors.md) is pinned to exactly three situations - client error / upstream error / internal error - each with its own type. A client cancel does not fit any of the three: it is not a rejected request (the request was valid), not an upstream fault (the provider never got a chance to fail), and not a gateway malfunction (the gateway did exactly what it should: stop work for a client that left). Silently picking one of the three to avoid extending the enum would keep reproducing the same misclassification this issue exists to fix.

Decision

A fourth ErrorType, LM-6xxx as its own code prefix

Add GatewayError::ClientCancelled (LM-6001) and ErrorType::ClientCancelled (serialized "client_cancelled"), alongside - not replacing - the existing three-way split. docs/errors.md documents it as its own section, and the code-prefix table gains 6xxx for client-cancellation. This is an additive change to the public envelope schema (type gains a fourth possible value); existing clients that switch on the three known values are unaffected as long as they have a default case, which the taxonomy already requires them to handle (new codes are added within existing prefixes routinely).

ProviderError::Cancelled now maps to GatewayError::ClientCancelled instead of GatewayError::Internal in GatewayError::from_provider.

HTTP status: 499

499 is the conventional “client closed request” status popularised by nginx - not in the IANA registry, but widely recognised in logs/dashboards and, critically, not a 5xx. The client has normally already disconnected by the time this status would be written, so it is never actually read by anyone; its only audience is server logs and the status label on the latency histograms. Any other 4xx would misleadingly imply the client’s request was at fault (it wasn’t - the request was fine, the client just left before it finished).

Telemetry: an explicit "499" label, not the "4xx" catch-all

crates/telemetry/src/latency.rs::status_str already degrades uncommon statuses to a coarse class ("4xx", "5xx", …) to keep Prometheus cardinality bounded. 499 gets an explicit arm instead, ahead of that catch-all, for the same reason LM-6001 gets its own code rather than being folded into an existing one: an operator dashboarding cancellation volume (e.g. to catch a client-side bug causing excessive aborts) needs to see it distinctly from ordinary 4xx client errors, not just confirm it isn’t a 5xx.

The stream-accounting safety net settles disconnects as 499

StreamAccounting (crates/server/src/accounting.rs) closes the per-request accounting record when a chat stream ends. Clean ends and in-band error frames settle explicitly with their real status; the Drop impl is the safety net for a body dropped before any terminal event - which is precisely a mid-stream client disconnect (or, rarely, server shutdown). That net used to hardcode 200, silently recording the most common real-world cancel as a success. It now settles at 499, and the stream wrapper settles at 200 as soon as the [DONE] terminator is observed so a client that disconnects after a clean end is still recorded as a 200.

Consequences

Precisely which cancellation paths this covers:

• Mid-stream client disconnect (the common case): the SSE body is dropped before its terminal event; StreamAccounting’s drop net now settles usage_log.status and the lumen_request_duration_seconds sample at 499 instead of a fake 200. These were never inflating 5xx alerts before - worse, they were invisible, recorded as successes. The win here is honest classification and a countable cancellation signal.
• Cooperative in-band cancellation while the stream is polled: the provider byte stream’s select! on the CancellationToken (crates/providers/src/http.rs) yields ProviderError::Cancelled, which the stream wrapper maps through GatewayError::from_provider into a terminal SSE error frame. That frame now carries LM-6001 / client_cancelled and settles the sample at 499 - previously LM-5001 / internal / 500. This was the one path that genuinely inflated status="500" samples, and alert rules built on status=~"5.." stop firing on it with no rule change.
• Not covered - non-streaming client disconnect: dropping the connection drops the whole handler and middleware future, so no status sample is recorded at all, before or after this change (the latency middleware’s own documentation notes it only observes completed requests). There is nothing to relabel on this path: it never polluted any metric, and it still produces no sample. Making it observable would need a disconnect-aware middleware, out of scope here.
• Server shutdown dropping in-flight streams is indistinguishable from a client disconnect at this layer and is also recorded as 499. Acceptable: it is rare, and “the stream was cut before its end through no fault of the gateway’s request handling” is the semantic 499 carries here.

Other consequences:

• The public error envelope’s type field gains a fourth possible value, client_cancelled. This is additive; docs/errors.md is updated as the source of truth.
• crates/core/src/error.rs’s three-way taxonomy doc comment (CLAUDE.md rule 8) now notes this fourth, orthogonal situation rather than silently breaking the “always exactly three” framing.

ADR 007 - Rate-limit and usage-log accounting refinements

• Status: accepted
• Date: 2026-07-15

Context

M5 shipped admission control (RPM/TPM quotas, hard budgets) and the usage log with four documented behaviours flagged in docs/backlog.md for revisiting (issue #26):

1. TPM debited the pre-call estimate and never adjusted it. A chat request with max_tokens = 2048 but 40 real output tokens permanently burned 2048 tokens of the per-minute window, starving later requests.
2. A request rejected by a later admission step still counted toward the earlier quotas. admit bumps RPM, then TPM, then the budget CAS; when the budget refused (402), the RPM and TPM bumps it had already made were never unwound, so a refused request consumed quota it never used.
3. The usage log recorded successful requests only. A refusal at admission (402/429) errored out of Accounting::begin before any UsageRecord was built, so per-key rejection analytics were impossible.
4. Metadata values were stringified in usage_log.metadata ({"batch":"42"}), losing the original JSON type and blocking numeric filtering.

The budget already had the right shape: reserve the estimate, then Reservation::settle to the real cost (over-reservation released, shortfall charged, real cost wins even past the limit). These refinements bring TPM and the usage log up to the same standard, without violating the hot-path rules (no blocking, no synchronous DB write on the request path).

Decision

1. TPM is settled to real usage, like the budget

Reservation now carries the tokens it debited to the TPM window (and the minute it debited them in). settle(actual_cost_micro, actual_tokens) adjusts the TPM window by actual_tokens - estimate, mirroring the budget: a smaller real count frees the window, a larger one overshoots and the next request is refused. The adjustment is a CAS loop (adjust_window) and is a no-op once the window has rolled to a new minute (the debited slot has already expired).

Asymmetry with the budget on drop. Dropping a reservation unsettled (upstream failure / cancellation before settle) refunds the budget (no money was spent) but keeps the TPM debit. TPM is a rate limiter: a request that hit the gateway counts against the per-minute rate even if the upstream call then failed. Only a successful settle, which knows the real token count, adjusts TPM. This preserves the established M5 principle (“refused/failed requests still count toward the rate”) while fixing the starvation caused by over-estimates on successful calls, which is the common case.

2. Quota bumps are unwound when a later step rejects

Within a single admit, if the TPM step refuses after RPM was bumped, the RPM bump is rolled back; if the budget step refuses after RPM and TPM were bumped, both are rolled back. A request refused inside admission therefore consumes no quota. (A request refused at its own step - e.g. RPM over the cap - never bumped that window in the first place, since bump_window checks before incrementing.) Rollback uses the same adjust_window CAS helper and the same minute guard.

3. Rejected requests produce a status-only usage row

When admission refuses, Accounting::begin enqueues a UsageRecord with the rejection status (402/429), zero tokens, zero cost, and any request metadata, then returns the error. It goes through the same bounded-mpsc UsageLogger as successful requests (try_send; a full channel drops and counts the row) - never a synchronous DB write on the request path (the M5 hot-path rule stands).

Zero tokens is the honest count: a rejected request reached no provider and produced nothing. The status column is what carries the rejection for analytics. Scope note: 401 (unknown/invalid key) is not logged - it is refused in the auth middleware before begin, where there is no key to attribute the row to. Upstream failures (5xx) after admission are already logged by the normal finish / finalize_with_status paths.

4. Metadata keeps its JSON value types

RequestMetadata stores Vec<(String, serde_json::Value)> (validated to string/number/bool) instead of pre-stringified strings. to_json - the source of the usage_log.metadata TEXT column - now emits typed JSON ({"batch":42,"canary":true}), so SQLite json_extract can filter numerically. Prometheus labels, which are always strings, stringify on the way out (label_values returns Cow<str>, borrowing the string case and only allocating for numbers/bools). The column stays TEXT; only its contents gained type fidelity, so no migration is needed.

Consequences

• The TPM window now tracks real usage for the common (successful) case, so large max_tokens reservations no longer starve a key for a full minute.
• Rejection analytics are possible per key (SELECT status, COUNT(*) FROM usage_log ... WHERE status IN (402, 429)), at the cost of extra usage-log volume during quota storms - bounded by the same channel capacity and drop counter as everything else.
• Reservation::settle gained a parameter; all call sites pass the real total token count (tokens_in + tokens_out).
• usage_log.metadata may now contain non-string JSON values; consumers that assumed all-strings should read it as typed JSON.

ADR 008 - Hot reload for auth knobs and DB provider-key rotation

• Status: accepted
• Date: 2026-07-15

Context

Hot reload (ADR-adjacent, M7 §7.3) already re-validated the config on SIGHUP or a config-file change and atomically swapped the routing table, price table and resilience policy (circuit-breaker state preserved). Two surfaces were still boot-time only and were flagged as debt (issue #20, docs/backlog.md M5/M7):

1. The [auth] operational knobs (budget-flush cadence, usage-log retention, usage-writer channel sizing) were read once and baked into background tasks.
2. A provider key stored in the encrypted DB via PUT /admin/provider-keys only took effect at the next restart: the DB-key snapshot was captured once at boot and merely re-applied unchanged on every reload, so a rotation was invisible until a restart.

The overriding constraints are the repo pillars: DB stays off the request path (pillar 3), no secret leaks (STRICT rule 5), and reload must remain atomic and non-disruptive to in-flight requests. Three questions had to be resolved that the specs do not cover.

Decision

1. Server bind address stays restart-only (explicitly out of scope)

The issue lists the bind address among “read once at boot”, but rebinding a live TcpListener under load is high-risk (dropped connections, port races, partial failure with no clean rollback) for negligible benefit over a rolling restart. We deliberately do not hot-rebind host/port. This is documented as a permanent restart-only limitation in the reload module docs and the backlog, rather than left as an implied gap.

2. DB provider keys are re-read on every reload, in the reload task

Instead of a frozen boot snapshot, ReloadTargets now carries an optional ProviderKeySource (an owned KeyStore clone + its own MasterKey handle + the configured provider names) and the backfill lives in an Arc<ArcSwap<HashMap<..>>>. Each reload runs reload_once, which:

1. async, in the reload task (never the request path): re-reads and decrypts every configured provider key from the store into a fresh map and stores it into the ArcSwap. A DB/decryption error is logged and the previous snapshot is kept, so a sick DB can never strip a working key.
2. on a blocking thread: runs apply_reload, which reads the refreshed snapshot and merges it into env-keyless specs before the (fallible) registry rebuild, then swaps the registry ArcSwap atomically.

Environment variables keep precedence (a spec with a resolved env key is left untouched), so rotation via this route only affects env-keyless providers, which matches the existing “DB back-fills providers whose api_key_env is unset” model. The registry is rebuilt wholesale (the same path SIGHUP already used); a key rotation does not change routing, only the api_key carried by rebuilt providers.

3. PUT /admin/provider-keys triggers a reload via a shared Notify

To make a rotation apply without waiting for a SIGHUP or file touch, the admin handler pings a shared tokio::sync::Notify after the DB write completes; the reloader task selects on it as a third wake source alongside SIGHUP and the file watcher. The trigger is exposed on AppState only when the reloader is actually armed, so the admin API never claims a rotation was applied when no reloader is running. The DB write is awaited before the notify, so the reload always observes the new key. Coalescing is free: Notify collapses a burst into one wake.

4. Auth knobs are a live atomic cell, not task-local constants

The runtime-safe knobs (flush_interval_ms, retention_days) live in a shared AuthKnobs (two atomics). The budget-flush task became a sleep-loop that reads the interval each cycle (a tokio::time::Interval cannot have its period changed in place), and the retention-purge task reads the window each tick. A reload overwrites the atomics, so both tasks pick up new values on their next tick with no restart. .max(1) guards a reload that disables auth (knob -> 0).

The bounded usage-log channel knobs (usage_channel_capacity, usage_batch_max, usage_flush_ms) are not made reloadable: the channel capacity is structural (fixed when the mpsc is created and the UsageLogger clones are handed to the app), so changing it means re-plumbing the writer - a restart is the honest boundary. auth.enabled and auth.db_path are likewise structural (they decide whether the whole stack and its DB connection exist).

Consequences

• Rotating a provider key is now a live operation: PUT /admin/provider-keys then requests authenticate with the new key within one reload, no restart.
• The reload’s DB read is bounded to the configured provider set and runs only in the reload task, honouring “DB off the request path”.
• Keys are never logged: ProviderKeySource holds a redacted MasterKey (zeroized on drop) and the backfill map carries raw keys only in memory, merged into ProviderSpec whose Debug already redacts api_key.
• The restart-only surface (bind address, enabled, db_path, channel sizing) is now explicitly documented rather than an accident of implementation.

ADR 009 - Shared parent budgets (budget groups)

• Status: accepted
• Date: 2026-07-22

Context

A virtual key is today the only budget boundary: budget_max lives on the key and is enforced in memory by a CAS reservation (M5, refined by ADR 007). That model cannot express a common billing shape: one prepaid pool consumed by several keys. The concrete motivator is a control plane that sells credits per customer while issuing one key per project of that customer; today it must chunk-allocate budget_max across the project keys and rebalance them from outside, which is racy and leaves credits stranded on idle keys.

Constraints inherited from the pillars and prior ADRs:

• Admission stays in per-process memory, before any upstream call; the database is never on the request path (M5).
• A request refused at admission consumes nothing: quota bumps and budget reservations made by earlier admission steps are unwound (ADR 007).
• Admin mutations apply to the DB and the in-memory state together, with no restart; hot reload re-reads DB state and never strips working state on a read error (ADR 008).
• v1 is single-instance with auth enabled; nothing here changes that (docs/operations/deployment.md, Scaling and high availability).

Decision

Introduce budget groups: a named budget pool that any number of virtual keys can belong to. Admission checks the key’s own budget (when set) AND the group’s budget (when the key belongs to one); spend settles against both.

1. Model

A group is {id, name, budget_max, budget_spent, created_at, deleted_at}. budget_max = NULL means unlimited (a pure attribution container). Groups carry budget only in this slice: no group RPM/TPM, no disabled flag, no expiry (future work, below). A key references at most one group via a nullable group_id; membership is admin-managed, never config-file state (groups and keys are DB entities, unlike providers).

budget_spent on the group is its own accumulator, flushed like key spend; it is never recomputed from member keys. Spend that member keys accrued before joining (or after leaving) the group is not retroactively moved.

2. Admission and settlement

admit order becomes: RPM bump, TPM bump, key budget reserve, group budget reserve. A refusal at any step unwinds every earlier step (ADR 007 rule), so a request refused by the group consumes no key quota and no key budget. The Reservation additionally holds the group entry and the amount reserved against it, captured at admission:

• settle(actual_cost, actual_tokens) applies the cost delta to the key AND the group (the TPM adjustment is key-only; groups have no TPM).
• Dropping unsettled refunds both budget reservations; the TPM debit stays, per ADR 007.
• A key moved to another group mid-flight settles against the group captured at admission - spend is attributed to the pool that admitted it.

Both refusals reuse LM-4001 (402): the semantic is unchanged, “a hard budget you are subject to is exhausted”. The error message distinguishes the scope (“budget exceeded for this key” vs “budget exceeded for this key’s group”) via a scope field on GatewayError::BudgetExceeded; there is no probing concern because a caller can only observe budgets it is billed against. The key-scope message is byte-identical to today’s.

3. In-memory state

AuthState gains groups: DashMap<String, Arc<GroupEntry>>. GroupEntry is the group analogue of KeyEntry’s budget half: budget_max_micro, spent_micro, dirty, same micro-USD atomics, same CAS reservation loop. KeyEntry holds group: ArcSwapOption<GroupEntry> (lock-free load on the hot path; arc-swap joins the auth crate from the existing workspace dependency). The pointer is resolved by AuthState (boot load, upsert, apply), never by KeyEntry::from_record. A group_id that cannot be resolved in memory (only reachable through out-of-band DB edits) leaves the key without live group enforcement and logs a warning - fail-open for that key rather than refusing all its traffic.

4. Flush, shutdown, reload, crash

• The periodic flusher drains dirty groups exactly like dirty keys (drain_dirty_groups alongside drain_dirty) into persist_group_budgets; the shutdown drain does the same final flush.
• Hot reload re-reads groups from the DB before re-reading keys (keys resolve group pointers), upsert-only, limits re-applied, in-memory spend preserved, DB errors keep the current tables (ADR 008 semantics).
• Crash recovery is identical to keys: enforcement lives in memory ahead of the flush, so a running process never overruns; a crash loses at most flush_interval_ms of accounting per pool, and after restart budgets reload from the last persisted state.

5. Storage and usage attribution

Migration 0007_budget_groups.sql:

• CREATE TABLE budget_groups (...) as in §1;
• ALTER TABLE virtual_keys ADD COLUMN group_id TEXT (nullable; no SQLite FK, consistent with the schema’s existing app-level referential integrity);
• ALTER TABLE usage_log ADD COLUMN group_id TEXT plus an index.

Every usage row (success AND admission refusal, per ADR 007) is stamped with the key’s group id captured at accounting begin, so per-pool reporting works even for refused traffic. GET /admin/usage gains group_by=group_id and a group_id filter, both riding the existing closed-set plumbing.

6. Admin surface

• POST /admin/groups {name, budget_max?} - create (201, the record; no secret exists for a group).
• GET /admin/groups (?include_deleted=true for tombstones) - list.
• PATCH /admin/groups/{id} {name?, budget_max?} - adjust; spend preserved.
• DELETE /admin/groups/{id} - soft delete, refused (400, LM-1001) while the group still has active member keys; the tombstone keeps usage_log attribution, mirroring key deletion, and its final spend is flushed on removal from memory.
• POST /admin/keys accepts group_id; PATCH /admin/keys/{id} accepts group_id as a tri-state field (absent = unchanged, null = leave the group, string = join) - the one deliberate divergence from the “patches cannot clear to NULL” rule, because leaving a group must not require re-minting the key. A group_id naming an unknown or deleted group is refused (400, LM-1001) before any write.
• lumen keys create gains --group-id (validated against the DB); a lumen groups offline subcommand is deferred - groups have no bootstrap chicken-and-egg problem because /admin/groups needs only the master key.

Every mutation applies to the DB and the in-memory maps together, effective on the next request, no restart - the ADR 008 contract.

Consequences

• The prepaid-credits control-plane pattern collapses to: one group per customer, one key per project, top up the group. No chunk allocation, no rebalancing, no stranded credits.
• The hot path gains one ArcSwapOption load and, for grouped keys, one more CAS loop per request - same order of cost as the existing key reservation, still allocation-free, still no locks held across await points.
• Two pools can now refuse a request; operators must read the LM-4001 message (or the usage log’s 402 rows grouped by group_id) to see which.
• VirtualKeyRecord grows group_id, so every enumerated column list (load_auth_entries, list_keys, find_by_hash, fetch_key, the integrity dump) changes in lockstep; the admin key responses expose it.
• Known admin-plane race, accepted for v1. delete_group’s member count and tombstone are separate statements, as are the membership validation and write in key create/patch, so a key racing into a group being deleted can orphan onto the tombstone. The consequence is bounded and fail-open by design: live Arc holders keep enforcing against the detached pool, and after a reload the dangling group_id resolves to no-pool-enforcement with a warning. Admin mutations are expected to be serialized by the control plane; wrapping count+tombstone (and validate+write) in BEGIN IMMEDIATE transactions is the cheap hardening if that assumption ever breaks.
• Not in this slice (recorded in docs/backlog.md): group RPM/TPM, a group disabled flag, group expiry, nested groups, a lumen_group_budget_remaining gauge, and a lumen groups CLI.

Amendment (2026-07-22): atomic grant routes

A prepaid-credits control plane tops up budgets concurrently; a read-modify-write PATCH budget_max can lose one of two racing top-ups. POST /admin/keys/{id}/grant and POST /admin/groups/{id}/grant take {"amount": <USD>} and raise the cap as an atomic increment on BOTH sides: budget_max = budget_max + ? inside SQLite, and a fetch_add on the live entry - neither side ever re-reads-then-writes, so concurrent grants all land. Spend and quota windows are untouched; the effect is immediate, no reload. Guardrails, all 400 LM-1001: the amount must be positive, finite and at most 1e12 USD (serde_json already rejects overflowing literals like 1e999 at parse time; the finite check is belt-and-braces, and the upper bound keeps repeated grants from summing the DB float toward +Inf, which would reload as an unlimited cap), an unknown or deleted id is refused, and granting to a capless (budget_max NULL) key or group is refused rather than silently meaning nothing - set a cap with a PATCH first. In memory the UNLIMITED sentinel is protected in both directions: a capless entry stays capless, and a capped entry saturates below the sentinel so a grant can never accidentally mint an unlimited budget.

Consistency contract: the database is authoritative. The grant is two independent atomic increments (DB, then memory), so three interleavings can leave the live cap diverging from the DB by at most one grant amount until the next reload or restart heals it: a hot reload or a PATCH (both store caps absolutely) racing the two steps in either direction, and a client disconnect between the DB write and the memory increment. Grants are also not idempotent: a timed-out grant may have landed - verify with a GET before retrying (an Idempotency-Key header is in the backlog).

ADR 010 - Operator console and fleet control

• Status: accepted
• Date: 2026-08-19

Context

LUMEN is administered by curl against /admin, guarded by one shared master key. Running a gateway per region (for client round-trip latency) multiplies that credential by the number of deployments and leaves no aggregate view of spend, usage or provider health.

We want a web console that manages several gateway deployments: key and budget administration, usage and cost dashboards, live operational state, and provider configuration.

Constraints inherited from the pillars and prior ADRs:

• Admission and spend stay in per-process memory, before any upstream call; the database is never on the request path (M5, ADR 007).
• Config is a file, validated then swapped atomically; an invalid reload keeps the previous configuration (ADR 008).
• The gateway does not terminate TLS and delegates it to a reverse proxy (docs/operations/deployment.md).
• v1 is single-instance with auth enabled.
• Prompts are never persisted; usage_log has no request or response columns by construction.

Decision

A console shipped as a separate deployment, not embedded in the gateway binary, managing N independently configured gateways.

1. Tenancy: a project is pinned to one gateway

A project is bound to exactly one home gateway. Its virtual key lives on that gateway and its budget is enforced there, in memory, unchanged.

This is the load-bearing decision. It means there is no cross-region key, no budget spanning gateways, and therefore no shared auth state: no Postgres backend for virtual_keys, no Redis for distributed rate limiting, and no allocation of budget slices across regions. Every console mutation targets exactly one gateway, so nothing in the design is a distributed transaction.

The cost is that moving a project between regions is a manual migration (mint a key on the destination, retire the old one, accept that usage history splits across two gateways). This is documented as an operational procedure rather than built as a feature.

2. Control direction: push over a private network

The console calls each gateway’s existing /admin API directly. No polling agent is added to the gateway, and the admin API stays the one control surface.

Push requires the console to reach the gateways, so the console runs on operator-controlled infrastructure with private network reachability (WireGuard or Tailscale). Platforms whose functions have no persistent network identity are unsuitable, because they would force gateway admin ports onto the public internet behind a credential that can mint keys.

3. Source of truth

The gateway remains authoritative for keys, budgets and live usage. The console’s database is a registry plus a cache, and stores no spend or budget column. Disagreement is surfaced as drift, never auto-healed: the console diffs GET /admin/keys against its own bindings and reports differences, because auto-reconciliation would delete keys created deliberately through the CLI.

There is one deliberate inversion. The gateway sweeps usage_log rows past its retention window, so beyond that window the console is the only remaining copy. Console retention must exceed gateway retention.

4. Human identity is delegated

The console implements no login, no password storage and no session minting. It sits behind an OIDC reverse proxy and trusts a signed identity assertion, exactly as the gateway delegates TLS to a proxy. Two roles: admin mutates, viewer reads.

5. Two new admin endpoints

GET / PUT /admin/config. Config stays file-owned: the file remains diffable, GitOps keeps working, and a gateway restarted without the console comes up identically. PUT validates the submitted TOML and builds a candidate registry from it before writing, because writing an invalid file first would break the next SIGHUP or restart with no visible cause. The write is atomic (temp file, fsync, rename) and guarded by If-Match against a content hash returned by GET, so concurrent operators cannot silently lose an edit. The previous file is kept for revert.

This endpoint can repoint a provider’s base_url and thereby redirect customer traffic. It is the highest-privilege operation in the system. Consistent with decision 4, the gateway itself has no acting identity to log - that lives in the console’s own audit_log, keyed to the signed identity the reverse proxy asserted. What the gateway records on its own side is content-level, not identity-level: a successful apply logs the old and new config content hashes at info, and a rejected apply (a stale If-Match, invalid TOML, or a config the registry cannot build) logs the full rejection detail at warn. Hashes only, never file content or secrets.

GET /admin/usage/export. Cursor-paginated raw usage_log rows. GET /admin/usage aggregates over one dimension at a time, so building a dashboard cube from it needs one call per dimension per window per gateway and still cannot answer cross-dimensional questions. Raw export lets the console build its cube once. Sovereignty is unaffected: usage_log holds no prompt or response content, and the only caller-supplied field is the ADR 002 metadata column.

Consequences

• The July fleet-state blockers (shared Postgres for auth, Redis rate limiting, budget allocation across regions) are not required for this work. They return only if decision 1 is reversed.
• lumen_usage_log_dropped_total becomes user-visible. The bounded usage channel drops entries under pressure by design, which is correct for a gateway and wrong for a dashboard that would otherwise under-report silently; the console records the counter per collection window and warns for affected periods.
• A self-hoster now needs an OIDC provider before seeing a dashboard. This is mitigated with a documented forward-auth recipe and a loopback-only development bypass, not by weakening the model.
• CLAUDE.md lists “Web UI” under what v1 does not do. This ADR supersedes that line; billing remains out of scope.

Alternatives considered

Console embedded in the gateway binary. Preserves the single-binary pillar and needs no network trust model, but cannot manage more than the gateway it ships inside, which defeats the multi-region goal.

Pull agent. Gateways poll the console for desired state. Removes the inbound admin surface and is NAT-friendly, but adds a substantial new subsystem to a gateway that deliberately does one thing, and reduces every console mutation to “queued” rather than confirmed.

Config owned by the database. Would make the console the source of truth for providers, but breaks GitOps, makes a console outage a configuration outage, and diverges from the file-and-validate-then-swap model in ADR 008.

Global keys with shared state. Real cross-region budgets, at the cost of a Postgres dependency on the auth path plus Redis for enforcement. Rejected as unnecessary once a project is pinned to one gateway.

ADR 011 - Outbound webhooks for budget events

• Status: accepted
• Date: 2026-08-25 (amended 2026-08-26: admin control surface)
• Tracking issue: #146

Context

A billing control plane integrating with LUMEN already has two of the three legs a budget loop needs:

• Control (push, backend to gateway): POST /admin/keys/{id}/grant and POST /admin/groups/{id}/grant apply atomic budget top-ups (ADR 009); PATCH adjusts limits and enable/disable.
• Reconciliation (pull, backend from gateway): GET /admin/usage/export streams invoice-grade raw usage rows (ADR 010).

The third leg, signals from the gateway to the backend, does not exist. A backend that sells prepaid credit (e.g. metering through Stripe) can only poll to learn that a customer’s budget is nearly consumed. Hard budgets make that a poor fit: the moment the pool reaches zero, the customer’s requests are refused with 402 LM-4001. An auto-recharge must fire before that moment, and polling trades latency against load in a way webhooks do not.

Constraints inherited from the pillars and prior ADRs:

• Nothing new on the request path; the database is never consulted per request and neither is any network endpoint that is not the routed provider (pillar 1, rule 4).
• Zero telemetry by default: today the gateway makes outbound calls only to configured providers. Any new outbound traffic must be strictly opt-in and carry no prompt or response content (pillar 2, usage_log’s no-content construction).
• Budget enforcement and settlement are in-memory atomics, flushed to SQLite on an interval; a crash loses at most flush_interval_ms of accounting (M5, ADR 007).
• The admin API remains the one control surface; ADR 010’s console pulls and pushes through it and reports drift rather than healing it.

Decision

An opt-in webhook sender for budget lifecycle events, delivered from the accounting layer, never from the request path.

1. Events

Three event families, all derived from state the gateway already tracks:

• budget.threshold: a key’s or group’s budget_spent / budget_max crossed a configured percentage. Thresholds are configurable (thresholds = [50, 80, 95]); each crossing is edge-triggered: one event per threshold per budget epoch, not one per request beyond it. A grant that drops consumption back under a threshold re-arms it (a new epoch begins when budget_max changes).
• budget.exhausted: the first admission refused with LM-4001 for a key or group since it last had budget. Also edge-triggered.
• key.disabled, key.rotated, key.deleted: administrative lifecycle changes, so a backend registry can stay in sync instead of discovering drift on the next reconciliation pull (ADR 010 surfaces drift; these events shrink the window in which it exists).

The payload carries accounting facts only: event type, an event id, the key or group id and name, budget_max, budget_spent, the crossed threshold, and a timestamp. Never a plaintext key, never metadata, never content.

2. Detection is free, delivery is decoupled

Budget settlement already performs an atomic fetch_add per request. Threshold detection adds a compare of the before/after values against the armed thresholds on that same settle, in memory. When a crossing is detected, the event is pushed into a bounded mpsc channel consumed by an async sender task, exactly the usage-log writer pattern (rule 4):

• A full channel drops the event and increments lumen_webhook_dropped_total. Webhooks are a convenience signal, not an accounting system; the export route remains the source of truth and the backend must reconcile against it.
• The sender delivers with at-least-once semantics: bounded exponential backoff (with jitter, capped attempts), then the event is dropped and lumen_webhook_dead_total increments. Delivery state is memory-only; a restart forgets undelivered events. This is deliberate: persisting a delivery queue would put a new writer on the DB and buy little, since the reconciliation pull already covers gaps.
• Shutdown cancels the sender without blocking: in-flight attempts are aborted through the same CancellationToken discipline as provider calls.

3. Authenticity and idempotency

Each delivery is a POST with:

• x-lumen-signature: HMAC-SHA256 over the raw body, keyed by a secret read from an env var named in config (signing_key_env, the provider-key pattern; the secret is never logged, never in errors, never in Debug).
• x-lumen-event-id: a unique id per event (not per attempt), so retries and post-crash re-fires are deduplicable by the receiver.
• x-lumen-timestamp: to let receivers reject stale replays.

Because a crash can lose up to flush_interval_ms of settled accounting, the same threshold can legitimately re-fire after a restart. Receivers must treat events as idempotent signals (the Stripe webhook contract), and grant routes are already atomic increments, so a duplicated auto-recharge trigger is absorbed by receiver-side dedup on the event id.

4. Configuration

[webhooks]
url = "https://backend.example.com/lumen/events"
signing_key_env = "LUMEN_WEBHOOK_SECRET"
events = ["budget.threshold", "budget.exhausted", "key.disabled"]
thresholds = [50, 80, 95]        # percent, for budget.threshold
channel_capacity = 1024           # bounded; full = drop + counter
timeout_ms = 5000
max_attempts = 5

No [webhooks] block means no sender task, no outbound calls, no behavior change. The block participates in hot reload like every other section (ADR 008): an invalid block rejects the reload, a changed URL or event set swaps atomically.

5. What this is not

• Not a generic eventing system: no per-request events, no usage streaming (the export route is for that), no fan-out to multiple receivers in v1 (one URL; a backend can fan out itself).
• Not guaranteed delivery: at-least-once while the process lives, dropped with a counter when the receiver is down past the retry budget. The pull API is the system of record.
• Not a replacement for ADR 010’s drift report: lifecycle events shrink the drift window; the console still diffs on its schedule.

Consequences

• A billing backend can close the loop: threshold event, Stripe charge, grant top-up, all before the customer sees a 402. Today’s alternative is polling GET /admin/usage on a tight interval.
• The gateway gains its first non-provider outbound call. The sovereignty stance is preserved by the opt-in default, but docs must state plainly that enabling webhooks makes the gateway call the configured URL.
• New metrics: lumen_webhook_sent_total, lumen_webhook_dropped_total, lumen_webhook_dead_total, delivery latency histogram.
• Testing needs a wiremock receiver: exactly-one signed event per crossing, 5xx retry with backoff, overflow drop with counter, no secret in logs, clean cancellation on shutdown (issue #146 acceptance criteria).

Amendment 2026-08-26: the webhook configuration is an admin resource

§4 above made [webhooks] a config-file section, reloadable but with three knobs that could only change across a restart (channel_capacity, signing_key_env, and the presence of the block itself). That is the wrong shape for the very caller this feature exists for: a billing control plane provisions a gateway through the admin API, and cannot restart it or edit its environment. The section stays, and it gains a control surface.

1. Routes

Master-key gated, alongside the other /admin routes:

• GET /admin/webhooks - the live settings, secret-free. Reports which source they came from and whether deliveries are signed.
• PUT /admin/webhooks - replace every setting; applied immediately and persisted, so a restart comes up identically.
• DELETE /admin/webhooks - stop emitting, persisted.
• PUT /admin/webhooks/signing-key - store the HMAC secret, sealed at rest.
• DELETE /admin/webhooks/signing-key - forget the stored secret.

There is still exactly one receiver (§5 stands): these routes edit that receiver, they do not create a collection. Fan-out remains future work.

2. Precedence: a stored row wins over the file

Settings written through PUT land in a single-row webhook_config table in the auth database. Resolution at boot and on every reload:

1. A stored row with enabled = 1 wins outright.
2. A stored row with enabled = 0 means off, whatever the file says - so DELETE is not silently undone by the next reload.
3. No row at all falls back to the [webhooks] file block (or to off).

The file therefore stays the declarative default for a GitOps deployment that never calls the API, and the API wins for a control-plane deployment. This is the provider-key rule (ADR 008) with the sources swapped: there, config-named environment variables are primary and the database fills in; here the database is an explicit operator override of a declarative default. The asymmetry is deliberate - a PUT that the next reload reverted would be a bug, whereas a provider key that the environment overrode is the documented contract.

GET names the source ("database" / "config" / "none") so drift between the file and the live setting is visible rather than inferred, in the spirit of ADR 010’s drift report.

3. Every field is editable at runtime

• url, events, thresholds, timeout_ms, max_attempts, retry_base_ms: swapped in the live policy cell, as before.
• channel_capacity: the bounded queue is rebuilt. The new queue takes new events; the previous sender task keeps its receiver, drains whatever was already queued under the settings it had, and exits when its last sender is dropped. Nothing already accepted is discarded to change a capacity.
• The signing secret: held in a swappable cell the sender reads once per event, so a rotation applies to the next delivery without restarting the task. Deliberately not per attempt: a signature covers a specific body, so re-reading the cell mid-retry would emit attempts the receiver cannot verify. Retries of an event already in flight therefore keep the secret they were signed with, and a rotation reaches the next event instead.

Enabling webhooks on a process that booted without them therefore works too: the queue, the sender task and the Prometheus collectors are created on the first enable, not at boot. A gateway that never enables webhooks exports no lumen_webhook_* series at all, which keeps the opt-in default honest on /metrics as well as on the wire.

4. The secret, and why it may cross the API

§3 above read the secret only from an environment variable named in config. That is still the primary source, and still the only one for an operator who manages secrets through their deployment system. But a control plane that provisions a gateway it does not own the environment of needs a way in, so PUT /admin/webhooks/signing-key accepts the secret in the body and seals it with AES-256-GCM under the master key - byte for byte the PUT /admin/provider-keys/{name} mechanism (ADR 008), including its threat model: the database file and LUMEN_MASTER_KEY together decrypt it, either alone does not.

Resolution order, evaluated at boot, on reload, and on every apply:

1. The variable named by signing_key_env, when set and non-empty.
2. The stored secret.
3. Neither: deliveries are unsigned. Refused as an error when signing_key_env was named (that is a broken deployment, not a choice) and allowed with a loud warning when it was deliberately omitted.

No route ever returns the secret, in any form. GET /admin/webhooks reports a boolean signed and the variable name, exactly as the provider surface reports key presence without key material.

5. What this does not change

Detection, edge-triggering, payload construction, at-least-once delivery, idempotency and the “never on the request path” guarantee are all untouched. The admin surface configures the pipeline; it is not part of it.

Future work

• Multiple receivers with per-receiver event filters.
• Quota events (quota.rpm_exhausted, quota.tpm_exhausted) once a sustained-rejection signal (as opposed to a single 429) is defined.
• A persistent outbox, only if reconciliation-by-pull proves insufficient in practice.

Backlog

Ideas surfaced during development that are intentionally out of scope for v1 (see CLAUDE.md → “What we do NOT do (v1)” and ROADMAP.md → “Backlog v2”). Recorded here so they are not lost, and so we don’t gold-plate the current milestone.

Noted while building ADR 009 (shared parent budgets)

• Grant routes are not idempotent (ADR 009 amendment). A timed-out POST /admin/{keys,groups}/{id}/grant may have landed; the docs tell billing automation to verify with a GET before retrying. A client-supplied Idempotency-Key header (dedup table keyed on it, replay the recorded response) would make retries safe; add it when a real billing integration asks.
• Groups carry budget only. Group-level RPM/TPM, a group disabled flag (pause a whole customer), group expiry, and nested groups are all deliberate non-goals of the first slice; each is a natural follow-up on the same GroupEntry shape.
• No lumen groups offline subcommand. Groups are created via POST /admin/groups (master-key gated, no virtual key needed), so there is no bootstrap chicken-and-egg; lumen keys create --group-id covers the offline key path. Add the subcommand only if a real workflow needs fully offline group provisioning.
• No lumen_group_budget_remaining gauge. Pool spend is visible via GET /admin/keys-style reads (GET /admin/groups) and the usage log; a Prometheus gauge per group would be cheap but adds a per-group metric series - decide when a dashboard actually wants it.
• Atomic budget top-up. PATCH on a key or group budget is a read-modify-write for the caller; a POST .../grant {"amount"} increment route would remove the need to serialize concurrent top-ups in the control plane.

Deferred to v2 (from the vision)

• Web admin UI
• Semantic cache
• Audio (input/output) support. Image input already shipped in v1: chat vision (M8) and multimodal embeddings (M9).
• Guardrails / moderation
• Distributed rate limiting (Redis)
• OTLP tracing export
• WASM plugin system
• Postgres backend for the auth/usage store (a postgres sqlx feature flag; the queries in crates/auth are simple enough to stay portable, so this is cheap to add if a deployment ever needs it - v1 is SQLite only)
• /v1/batches (OpenAI async batch jobs API) - explicit v1 non-goal: batch jobs imply persistent job state and scheduling, which sits poorly with the DB-off-the-request-path and single-binary pillars. Note: crates/providers/src/batch.rs is embedding request sub-batching, unrelated to this API surface.
• /v1/files (OpenAI file upload/storage API) - explicit v1 non-goal: blob storage state, same rationale as /v1/batches.

Noted while building M1

• Token-array inputs for /v1/embeddings (input as arrays of token ids) are not modelled - only string and string-batch. Resolved (issue #25). EmbedInput now models Tokens ([1,2,3]) and TokenBatch ([[1,2],[3,4]]); they pass through natively on OpenAI-compatible providers and count one token per id in the estimation fallback. Text-only providers (Cohere, TEI, Ollama, Jina, Voyage, Mistral) reject them with a 400 (LM-1001) before any upstream call.
• Rerank documents accepts only strings; Cohere also allows objects. Resolved (issue #25). See the M3 note below.
• error_type() collapses 401/402/429 into invalid_request because the public taxonomy only has three types. Fine per CLAUDE.md, but note it’s coarse.
• Acceptance criterion “boot < 100 ms” is verified manually (M1); fold a real timing assertion into the M7 criterion benchmarks rather than a flaky unit test.
• Graceful shutdown is unit-tested via an injected shutdown future; the real SIGINT/SIGTERM path (shutdown_signal) has no integration test (hard to do portably). Acceptable; revisit if signal handling grows. Resolved (issue #27). crates/server/tests/signal_shutdown.rs (#[cfg(unix)]) spawns the real lumen binary and sends it a genuine SIGTERM/SIGINT via libc::kill, asserting the same drain-then-exit-0 behaviour the injected- oneshot tests already prove for serve() itself - now proven for the actual tokio::signal path too.

Noted while building M2

• Embedding output is always a float array in v1. Base64 embeddings are decoded on the way IN (a client requesting encoding_format: "base64" won’t error), but we do not re-encode on the way OUT. Resolved (issue #25). When encoding_format: "base64" is requested, the gateway re-encodes each vector as OpenAI-style base64 at the response edge, so it works for every provider (including Ollama and TEI, which have no upstream encoding_format).
• Ollama drops the OpenAI-only dimensions field with a debug! log; a client asking for a specific dimension silently gets full-width vectors. Resolved (issue #25). A per-provider strict = true makes Ollama reject a request that sets dimensions with a 400 (LM-1001) instead of silently dropping it; the default stays lenient. encoding_format is no longer lost either (handled at the edge, above).
• LM-1002 (payload too large, 413) is emitted by RequestBodyLimitLayer as a raw 413 without our JSON error envelope. Map the tower-http rejection to GatewayError::PayloadTooLarge for a consistent body.
• Cancellation tests use real (short) wall-clock delays rather than tokio(start_paused); robust today but revisit if they flake under CI load. The HTTP-level disconnect test asserts the server stays responsive and the upstream got the request - the actual upstream abort is proven at the provider layer (conformance scenario_cancellation_aborts_upstream). Update (issue #27): this predicted flake happened - mistral_passes_embed_conformance_suite flaked once under full workspace-test parallelism. Widened the mocked upstream delay (2s → 3s) and the elapsed-time assertion (1s → 2s) in scenario_cancellation_aborts_upstream (crates/providers/tests/embeddings.rs) for more scheduler-jitter headroom without weakening what the assertion proves (still asserts the call returns in well under half the mocked delay). tokio(start_paused) would sidestep wall-clock entirely but doesn’t compose with the real reqwest/wiremock I/O this suite exercises; deferred unless the wider margin still flakes.

Noted while building M3

• Cohere v2 embed requires an input_type; the gateway can’t know query-vs- document intent by default, so it sends search_document unless overridden. Resolved (issue #22): a caller may set input_type as an extra field on the /v1/embeddings request body (search_document, search_query, classification, or clustering); an unknown value is rejected with LM-1001 before any upstream call. See docs/providers.md § cohere. A per-model default (config-side) is still open if per-request opt-in proves insufficient in practice.
• usage.search_units is only meaningful for Cohere; Jina and Voyage bill rerank in tokens. Resolved (issue #10): RerankUsage now carries a separate total_tokens/tokens_estimated pair, upstream-reported for Jina/Voyage and gateway-derived (from query + documents) for every other provider.
• Rerank documents accept string or {text} only. Cohere also allows arbitrary objects with a rank_fields selector - out of scope. Resolved (issue #25). RerankDocument::Object now keeps all fields; the request carries an optional rank_fields selector, and the gateway reduces each object document to a single ranking text at the edge (selected fields joined, or the text field when no selector), so providers still only ever see plain text. With return_documents: true, an object document echoes that reduced ranking text in document.text, not the original JSON object.
• TEI serves one model per process and ignores the request model/top_n; the gateway truncates to top_n after sorting. The configured upstream_id is informational for TEI. A future health/introspection hook could verify the configured model matches what the TEI process actually serves.
• The four hosted rerank providers default max_batch_size conservatively (Cohere 96, Jina/Voyage/OpenAI-style large, TEI 32). Revisit against real provider limits; embeddings batching already exercises these.
• Per-model embedding max_batch_size override (surfaced by issue #90). Vertex embeddings hardcode max_batch_size() = 1 because gemini-embedding-001 accepts a single instance per :predict call, but the other text-embedding-* models take up to 250. A single conservative value means a 1000-input request is ~250 sequential rounds at concurrency 4. A config override (or a larger known-safe default for the text-embedding models, keyed by model) would soften this bottleneck. Deferred: it needs per-model config plumbing that the current EmbeddingProvider::max_batch_size (provider-wide, model-agnostic) does not carry.

Noted while building M4 (slice 1 - non-streaming chat)

• Streaming disconnect test is a no-hang assertion, not an abort assertion. streaming_client_disconnect_does_not_hang_server proves the server stays responsive but not that the upstream connection was actually closed (M4 acceptance criterion 2: “upstream closed in < 100 ms”). And because the interim single-shot chat_stream awaits the full chat() before the guard is moved into the SSE body, the moved-guard path is not exercised. Strengthen in the streaming slice: assert via wiremock that the upstream request was aborted, and add a case where the client cuts after the body starts.
• Anthropic translate_request copies message roles verbatim and does not normalise user/assistant alternation or drop/merge tool/empty-content messages (spec 4.3 bullet). Fine for the interim text path; complete with tool translation in the streaming slice.
• Anthropic responses set created: 0 (the API returns no timestamp). Some OpenAI clients expect a real epoch; set SystemTime::now() if one complains.
• to_sse_body’s Chain drops the mapping closure (and thus the cancel guard) as soon as the chunk stream is exhausted, a hair before the [DONE] frame. Benign today (nothing left to cancel once the upstream is done), but the real streaming slice must not tie a live resource to guard survival through [DONE].
• Interim single-shot emits [DONE] even after a mid-stream error frame. Harmless with one item; real streaming must terminate after an error.

Noted while building M4 (slice 2 - zero-copy streaming)

• Acceptance criterion 5 (LM-3010) not yet implemented. Resolved (commit 076b909, slice 3). When the upstream closes without a [DONE] terminator and without a transport error, the gateway now appends a terminal data: {"error": {"code": "LM-3010"...}} frame then closes cleanly. The lightweight tail-watcher (EventStreamState::scan_frame in crates/server/src/chat.rs) inspects only frame boundaries - it matches a line-anchored \ndata: [DONE] marker and keeps at most DONE_MARKER.len() - 1 trailing bytes, so a terminator split across two frames is still detected and model content that merely contains the text can never spoof it. Covered by the unit guard tests in that module and the end-to-end wiremock test upstream_stream_without_done_yields_fg3010_error_frame (passthrough), plus the mid-stream error-frame tests (mid_stream_provider_error_becomes_terminal_error_frame and resilience.rs).

• Tools on Gemini: RESOLVED (issue #4). translate_request (google) now maps OpenAI tools to Gemini tools[].functionDeclarations and tool_choice to toolConfig.functionCallingConfig; assistant tool_calls become functionCall parts, role tool messages become functionResponse parts, and both the non-streaming and streaming translators surface Gemini functionCall parts as OpenAI tool_calls. Chose the full mapping over the LM-2002 rejection. Covered by unit tests and wiremock round-trips in crates/server/tests/chat.rs.

Noted while building M5

• Accurate per-model tokenizer - DONE (issue #8). ADR 003’s opt-in “accurate tokenizer via spawn_blocking” now ships behind [tokenizer] mode = "accurate": exact tiktoken-rs BPE (cl100k_base / o200k_base by model prefix) refines the local fallback AFTER the response is handed off (deferred accounting close, BPE on the blocking pool), landing in Prometheus and usage_log; the response envelope stays heuristic and the request is never delayed. Heuristic fallback for non-OpenAI models and any failure. The default stays the byte heuristic (zero cost). See the ADR 003 “opt-in accurate tokenizer” addendum. Remaining follow-up: streaming input counts stay heuristic (no buffered prompt to BPE under the ADR 004 passthrough rule).
• Streaming output estimation = data-frame count. When a stream carries no usage (rare: include_usage is auto-requested and translators always emit usage), the output-token estimate is the number of data: frames (~1 token per delta for OpenAI-style streams). Crude but honest (estimated=true).
• usage_log records successful requests only. Refusals (401/402/429) and upstream failures are visible in logs/metrics but produce no usage row. RESOLVED (issue #26, ADR 007): admission refusals (402/429) now enqueue a status-only usage row (zero tokens, status carries the rejection) through the same non-blocking channel. 401 stays unlogged (refused in the auth middleware, before accounting opens - no key to attribute); upstream failures are already logged by the normal finish path.
• Rejected requests still count toward RPM/TPM. Quota bumps are not unwound when a later admission step (budget) refuses the request. RESOLVED (issue #26, ADR 007): a request refused inside admission (TPM after RPM, or the budget after both) now rolls back the bumps it already made, so it consumes no quota. (A request refused at its own step never bumped that window.)
• DB-stored provider keys are boot-time only. PUT /admin/provider-keys takes effect at the next restart; wire it into the M7 hot-reload path. Done (post-v0.1.0): the admin route pings the hot-reload trigger and every reload re-reads provider keys from the encrypted store, so a rotation applies without a restart. Env-sourced keys keep precedence.
• Metadata values are stringified in usage_log.metadata. JSON object of strings. RESOLVED (issue #26, ADR 007): the column now stores typed JSON ({"batch":42,"canary":true}), so numeric/boolean filtering via SQLite json_extract works. Prometheus labels still stringify (labels are strings).
• TPM debits the pre-call estimate, never adjusted. Unlike the budget (reserved then settled to real usage), the tokens-per-minute window keeps the estimate. RESOLVED (issue #26, ADR 007): a successful request now settles the TPM window to the real token count, mirroring the budget - large max_tokens reservations no longer starve a key. A dropped (failed/cancelled) reservation deliberately keeps the TPM debit: a request that hit the gateway still counts against the rate limit.
• No zeroization of key material. MasterKey and the raw env string are not zeroized on drop; the zeroize crate would close the residual-memory window. Low risk (single long-lived process), noted from the M5 review.

M6 (resilience) - deferred

• Per-provider connect timeout. Done (issue #24, 2026-07-15.) A provider may set connect_timeout_ms; it then gets its own (unpooled) reqwest::Client built at registry construction, while every non-overriding provider keeps sharing the one pooled client. See the ADR 005 amendment.

• First-frame-peek streaming retry. DONE (issue #7, 2026-07-15). The commitment point moved from the open phase to the first content frame: the streaming path now peeks the first upstream frame, so a stream that opens 200 then errors or closes before any content frame fails over (and penalises the breaker) instead of committing to a terminal SSE error. Buffers at most one frame, bounded by first_token, cancellation-safe. See ADR 005, 2026-07-15 amendment; crates/router/src/peek.rs.

• Circuit-breaker map is unbounded by design. One entry per (provider, model) actually seen; bounded by the configured surface, never by client input. If a future dynamic-model feature lets clients mint arbitrary model ids, add an LRU cap.

• Health probe is a bare GET to base_url for keyed vendor kinds. TEI, vLLM and Ollama now get a real, unauthenticated liveness endpoint (DEBT-3, issue #23). Keyed vendor kinds (OpenAI, Anthropic, the OpenAI-compatible hosts, …) still get bare reachability only: their liveness routes (GET /v1/models, etc.) require an API key the probe task does not carry, and an unauthenticated call there would 401 a healthy server - a worse signal than bare reachability. Plumbing provider credentials into the probe task to unlock this is deferred to keep the probe free and side-effect-free.

• health_stays_fast_under_upstream_429_storm flaky under a 500-connection storm. Resolved (issue #27; the storm path’s kernel-level connect retry landed via PR #42). Several dev sandboxes hit a client-side panic (reqwest::Error from a TCP connect reset or a broken-pipe SendRequest) when firing 500 concurrent requests at once - a saturated OS accept backlog, not gateway behaviour. crates/server/tests/resilience.rs’s post_chat helper now retries is_connect()/is_request() transport errors (never a received HTTP response) with a short backoff before giving up, and the storm size is overridable via LUMEN_RESILIENCE_STORM_SIZE (defaults to the unchanged CI-scale 500) as a secondary escape hatch. The status-code assertions (429/503 only, /health latency bound) are unchanged - the fix is purely about not letting host-level connection churn panic the test.

M7 (release) - deferred

• Hot reload swaps routing, pricing, resilience and the safe auth knobs. SIGHUP / file-watch / admin-trigger re-validate the config and atomically swap the provider registry (ArcSwap), the price table, the resilience policy and the runtime-safe [auth] knobs (flush_interval_ms, retention_days). Still restart-only, by design: the server bind address (rebinding a live listener is high-risk), auth.enabled, auth.db_path, and the bounded usage-log channel knobs (usage_channel_capacity, usage_batch_max, usage_flush_ms) whose capacity is fixed when the channel is created.
• Hot reload re-resolves env keys fresh; DB keys are re-read each reload. A reload re-reads provider keys from the environment and, for env-keyless providers, from the encrypted DB store, so rotating a DB-stored key (PUT /admin/provider-keys) after boot takes effect on the next reload (the admin route triggers one) with no restart. A DB read error keeps the previous snapshot so a reload never strips a stored key.
• Anthropic/Gemini translation fuzzing goes only as deep as the shared SSE parser today. Fuzzing the translate_request/translate_response/stream translators directly needs a small public (or #[cfg(fuzzing)]) shim over the currently-private functions. Resolved (issue #27). Each of providers::anthropic/providers::google now has a #[cfg(fuzzing)] pub mod fuzzing shim (compiled only under cargo fuzz, which sets --cfg fuzzing across the dependency graph - zero normal-build surface change) exposing translate_request/translate_response. Four new targets (anthropic_translate_request, anthropic_translate_response, google_translate_request, google_translate_response) in fuzz/, wired into the weekly fuzz CI matrix; see fuzz/README.md “Why #[cfg(fuzzing)] shims” for the alternatives considered.
• Loaded throughput vs LiteLLM not measured in-repo. The in-process overhead (~3 µs) is benchmarked; the full p50/p99/RAM/req·s head-to-head is a reproducible bench/ harness (docker-compose + k6) run by the operator, not captured as a committed baseline. Resolved (issue #27). bench/run.sh drives the full harness end to end (pinned, digest-locked images; one command) and writes a timestamped, committed result under bench/results/. A recorded baseline is linked from docs/perf-baseline.md - read that section’s caveat before trusting the absolute numbers: it was recorded on a shared dev host, not dedicated hardware, so the relative LUMEN-vs-LiteLLM comparison is solid but the absolute figures are illustrative. Re-run bench/run.sh on real hardware for numbers to make capacity decisions on.
• Streaming time-to-first-token in the k6 harness. The head-to-head now reports non-streaming time to first byte (k6’s built-in http_req_waiting), but a streaming TTFT column (first SSE chunk of a stream: true response, per target, under load) is not measurable with stock k6: it buffers response bodies, so the first body chunk cannot be timestamped. Doing it in the harness needs a body-aware load client - the xk6-sse extension (custom k6 build via xk6) or a small purpose-built Rust/Go client - plus a mock upstream that actually paces its SSE frames (mockserver writes the whole body at once). Until then the streaming first-bit number lives off-harness in cargo bench -p server --bench stream_ttfb (single-request latency, direct vs via-gateway, no concurrent load).

Backlog debt paid down (post-v0.1.0)

• Full-config hot reload (DEBT-1) - done. Reload now swaps pricing and the resilience policy (retry/timeouts/fallbacks) as well as the routing table, preserving circuit-breaker state.
• Auth-knob hot reload + DB provider-key rotation - done. Reload also swaps the safe [auth] knobs (flush_interval_ms, retention_days) and re-reads DB-stored provider keys, and PUT /admin/provider-keys triggers a reload so a rotation applies without a restart. Server bind address stays boot-time (see the M7 note above for the exact restart-only surface).
• Key-material zeroization (DEBT-2) - done. MasterKey wipes on drop and the raw LUMEN_MASTER_KEY string is zeroized after use.
• Richer health probe (DEBT-3, issue #23) - done for the self-hosted, keyless kinds: TEI (/health), vLLM (/health) and Ollama (/api/version), each a real liveness endpoint (non-2xx = down). Keyed vendor kinds keep bare host-reachability (no reliable unauthenticated liveness endpoint); a per-kind, authenticated probe for vendor APIs remains out of scope (see the M6 entry above).

Noted while building M8 (vision - image input to chat)

• Per-image token heuristic for the estimation fallback - done (issue #9). The estimation fallback (upstream reports no usage) now adds a flat per-image constant (85 tokens for "detail": "low", 765 for "high"/"auto"/unset) instead of counting an image part as 0. A true per-dimension tile count (OpenAI’s 85 + 170 * tiles) still needs decoded pixel dimensions, which the gateway does not extract from a data: URI today and remains out of scope (no image-byte inspection on the request path). See the ADR 003 addendum.
• Anthropic/Gemini file/GCS image URIs. Only inline base64 (data: URIs) and, where the provider fetches it itself (Anthropic), remote http(s) URLs are supported. Anthropic’s source: {type: "file", file_id: ...} and Gemini’s GCS fileUri sources are not modelled; add if a caller needs pre-uploaded-file references instead of inline bytes.

Provider coverage - next candidates (post-rename)

• Tier-2 clouds need dedicated kinds (different auth/schema, not OpenAI-compatible): Azure OpenAI (deployment routing + api-version) (shipped - kind = "azure"), AWS Bedrock (SigV4, per-model schemas) (shipped - kind = "bedrock", Converse API + SigV4, crates/providers/src/bedrock/), Google Vertex AI (GCP OAuth, regional endpoints) (shipped - kind = "vertex_ai", crates/providers/src/google/vertex/).
• Azure: dedicated api_version config field - a desired fast-follow to the shipped azure kind, which currently reads the version from an ?api-version=... query string on base_url. A first-class field needs a matching ProviderSpec + crates/server/src/config.rs change. Resolved (issue #65). The provider config now takes an optional api_version field (azure-only, rejected on other kinds at boot), threaded through ProviderSpec into AzureProvider. Precedence: the explicit field wins over an ?api-version=... query string on base_url (kept for back-compat), which wins over the pinned built-in default.
• Cohere chat (Command R/R+) - we ship Cohere embed+rerank; chat is a distinct schema. Resolved (shipped - crates/providers/src/cohere/chat.rs, text chat + streaming + tools). The remaining slice is vision/image input only: the chat translation maps content with image_url: None, tracked as issue #73.
• More rerankers: Mixedbread (mxbai-rerank) (shipped - kind = "mixedbread"), Pinecone Rerank (shipped - kind = "pinecone"), NVIDIA NIM rerank (shipped - kind = "nvidia"), Together LlamaRank (shipped - kind = "together"). All four live under crates/providers/src/; the section’s “cheap differentiation for a first-class rerank gateway” goal is done.

Noted while building M8 (vision / image input)

• LM-2004 pre-flight is primary-only. A remote http(s) image URL is rejected up front (LM-2004, 400) only when the model’s primary provider can’t fetch it (Gemini). If the primary accepts URLs (OpenAI) but a Gemini model is a fallback, a fail-over to Gemini surfaces as LM-3002 (502, translation error) rather than a client 4xx - safe (never fetched, no retry loop) but a soft break of the 4xx/5xx separation (rule 8). Options if it ever bites: scan the whole chain in the pre-flight (rejects some primary-servable requests), or add a dedicated client-input ProviderError mapped to a 4xx.
• Per-image token heuristic for the estimation fallback - done (issue #9, see the note above under “M8 (vision - image input to chat)”). The remaining gap is a true dimension-based tile count, which needs image decoding this gateway deliberately does not do on the request path.
• Provider-native image URI forms. Anthropic/Gemini file & GCS URI image sources (beyond inline base64 + remote URL) are not modelled.
• Tool-role messages with image parts are silently flattened (noted while fixing issue #73). The Cohere translator gates its v2 image blocks on the user role (Cohere’s ToolMessageV2 cannot carry images), so an image part on a tool message is flattened to its text - consistent with the Anthropic translator today, but honest handling would be a 400 before the upstream call. Candidate: a shared role-aware content check in the M8 pre-flight instead of per-translator conventions.
• Remote image URLs on OpenAI-compatible-path providers whose upstream only accepts base64 (noted while wiring ollama chat, issue #63). Every kind served by the shared OpenAI provider inherits accepts_remote_image_url() = true, including ollama (its /v1 chat endpoint takes base64 data: URIs only). So a remote http(s) image URL on a vision-declared ollama model skips the LM-2004 pre-flight and fails opaquely upstream instead of as an honest client 4xx. Candidate fix: gate accepts_remote_image_url() on the provider kind (or a per-spec flag) rather than on the implementing provider type.

Contributing

LUMEN is open to contributions. The canonical, always-current guide lives at the repository root - GitHub renders it in the pull-request and issue UI:

➡️ CONTRIBUTING.md on GitHub

It covers development setup, the validation bar (cargo test / clippy pedantic / fmt), the Definition of Done, commit and PR conventions, the ADR process, and the full issue-label taxonomy.

Issues are classified along four axes (Type, priority:, area:, scope:); the canonical taxonomy with label meanings is in CONTRIBUTING.md.