LUMEN

Lightweight Unified Model ENdpoint

A universal, self-hostable LLM gateway written in Rust. One OpenAI-compatible endpoint in front of many providers - for chat, embeddings, reranking and typed decisions alike. It is designed to be light, fast and sovereign: a single static binary, zero telemetry, and prompts that are never logged by default.

What’s here

• Getting started - installation, quickstart and configuration basics.
• Chat - completions, streaming, vision and tool calling.
• Embeddings - embeddings, batching and multimodal embeddings.
• Reranking - reranking.
• Decisions (typed answers) - Decisions (TypeSafe, Perplexity, OpenAI, Cloudflare and Ollama models, POST /v1/decisions).
• Operations - token accounting & cost, metrics & dashboards, logging, usage log & multi-tenant metadata, keys, quotas & budgets, resilience tuning, deployment and upgrades.
• Examples - ready-made scenario configs.
• Reference - providers, error codes and the performance baseline.
• Architecture decisions - the ADRs that pin the design.
• Project - backlog and contributing.

The five pillars

Every trade-off is decided in this order:

1. Performance - < 1 ms added latency p99, zero-copy streaming, ~15 MB RAM idle.
2. Sovereignty - zero telemetry, prompts never logged by default, single binary.
3. Robustness - propagated cancellation, backpressure, DB off the request path.
4. Multi-capability - chat + embeddings + rerank + typed decisions are first-class citizens.
5. Token observability - every request of every capability produces a token count: upstream usage when reported, otherwise a local estimate flagged estimated. Never a silent zero. See token accounting & cost.

Want to contribute? Start with the contribution guide.

Installation

LUMEN ships as a single static binary. Pick one of three ways to get it: Docker, a prebuilt binary from a GitHub release, or a build from source.

Docker

docker run -p 8080:8080 \
  -v ./config.toml:/config.toml \
  -e OPENAI_API_KEY=sk-... \
  ghcr.io/qdequele/lumen:latest

The image sets LUMEN_SERVER__HOST=0.0.0.0 for you, so the server binds to all interfaces inside the container. The image is multi-arch: linux/amd64 and linux/arm64.

Prebuilt binary

Static musl binaries for x86_64-unknown-linux-musl and aarch64-unknown-linux-musl are attached to every GitHub release cut from a v* tag, each with a .sha256 checksum file alongside it. Verify a download before unpacking:

shasum -a 256 -c lumen-x86_64-unknown-linux-musl.tar.gz.sha256

From source

Needs a recent stable Rust toolchain (MSRV 1.94, per Cargo.toml and checked in CI against the committed Cargo.lock):

cargo build --release -p server --bin lumen

The binary lands at target/release/lumen. Run it with:

lumen --config config.toml

Validate a config without booting

lumen --check-config [--config <PATH>] validates a config file the same way the server does at boot (parsing, semantic validation and provider registry construction) and exits: 0 if valid, non-zero otherwise. It binds no listener, opens no database, and contacts no provider, so it is safe to run in a CI or deploy pipeline ahead of a real boot:

lumen --check-config --config config.toml

Next

Continue to the Quickstart, or browse the fully commented config.example.toml on GitHub.

Quickstart

Zero to a successful chat + embed + rerank request.

1. Minimal config

The ids below (gpt-4o, text-embedding-3-small, rerank-english) are the same ones used throughout config.example.toml. This minimal file needs an OpenAI key (chat + embeddings) and a Cohere key (rerank).

# config.toml - minimal quickstart config
[[providers]]
name = "openai"
kind = "openai"
api_key_env = "OPENAI_API_KEY"

[[providers.models]]
id = "gpt-4o"
upstream_id = "gpt-4o-2024-08-06"
capabilities = ["chat"]

[[providers.models]]
id = "text-embedding-3-small"
capabilities = ["embed"]

[[providers]]
name = "cohere"
kind = "cohere"
api_key_env = "COHERE_API_KEY"

[[providers.models]]
id = "rerank-english"
upstream_id = "rerank-v3.5"
capabilities = ["rerank"]

2. Run

Docker (the released image; sets LUMEN_SERVER__HOST=0.0.0.0 for you):

docker run -p 8080:8080 \
  -v ./config.toml:/config.toml \
  -e OPENAI_API_KEY=sk-... \
  -e COHERE_API_KEY=... \
  ghcr.io/qdequele/lumen:latest

From source (needs a recent stable Rust toolchain):

export OPENAI_API_KEY=sk-...
export COHERE_API_KEY=...
cargo run -p server -- --config config.toml

By default auth is off, so these requests need no Authorization header. Providers whose API-key env var is unset are only rejected when a request actually routes to them, so a partial set of keys is fine.

If you turn auth on ([auth] enabled = true), bootstrap your first virtual key with lumen keys create --name bootstrap - it runs offline against the auth database, before the server is ever started. See Keys, quotas & budgets.

3. Chat

curl -s http://localhost:8080/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Say hello in one word."}]
  }'

The response is the OpenAI chat completion envelope, including a usage object with the token count.

4. Embeddings

curl -s http://localhost:8080/v1/embeddings \
  -H 'content-type: application/json' \
  -d '{
    "model": "text-embedding-3-small",
    "input": ["the quick brown fox", "a lazy dog"]
  }'

The response carries data[].embedding for each input, plus a usage object.

5. Rerank

curl -s http://localhost:8080/v1/rerank \
  -H 'content-type: application/json' \
  -d '{
    "model": "rerank-english",
    "query": "What is the capital of France?",
    "documents": ["Paris is the capital of France.", "Berlin is in Germany."],
    "top_n": 2
  }'

Results come back sorted by descending relevance_score. documents must be non-empty: an empty list is rejected with LM-2010.

Next steps

• Chat completions
• Embeddings
• Reranking
• Decisions (typed answers)
• Token accounting

Configuration basics

Everything is one TOML file, plus LUMEN_* environment variable overrides that use __ for nesting, e.g. LUMEN_SERVER__PORT=9090. In the TOML file itself, top-level keys must appear before any [table] header. The exhaustively commented reference is config.example.toml on GitHub.

Section tour

log_format - "pretty" (human-readable, default) or "json" (production).

[server] - the HTTP server: host (bind address; use "0.0.0.0" in a container) and port (must not be 0), body_limit (max request body in bytes), first_token_timeout_ms (how long to wait for the upstream’s first sign of life before failing with LM-3011; for streaming this is time to the first SSE frame, for non-streaming it is the whole upstream call), and sse_heartbeat_ms (idle interval after which a : ping SSE comment keeps proxies from reaping a silent stream).

[auth] - virtual keys, hard budgets, quotas and the usage log. Disabled by default, in which case the gateway is an open proxy with no database at all. See Keys & budgets.

[telemetry] - which x-lumen-metadata keys become Prometheus labels on the token counters. See Usage log.

[resilience] - retries, circuit breaker, timeouts and health checks. Every value is the built-in default and the whole section is optional; see config.example.toml for the full set. Details in Resilience.

[[providers]] / [[providers.models]] - one [[providers]] block per upstream (id, upstream_id, capabilities, modalities, costs, and an optional release_date ISO date that GET /v1/models returns as release_date and as the OpenAI integer created, so clients can sort models by release). See Providers for the full provider matrix and per-provider notes. These are the foundation models: plain upstream bindings. Fallback, split and conditional routing no longer live on them (the per-model fallbacks field was removed).

[[virtual_models]] - public ids that carry routing logic over your foundation models: single, fallback, split and switch strategies, overrides, chat presets, and Jev as a reranker. A virtual model has an id, one capability, a strategy, targets, and optionally fallback_on, preset, a description shown on GET /v1/models (where it appears with "virtual": true) and listed = false to hide it from that list while it stays callable. Foundation and virtual ids share one namespace. See Virtual models. An older config with fallbacks is converted by lumen config migrate.

[image_fetch] - server-side fetching of remote image URLs for multimodal input. See Multimodal input.

API keys

API keys are never written in the config. A provider references the name of the environment variable that holds its key, via api_key_env.

Hot reload

A SIGHUP, a file watch (file mode), an admin config write, or an admin provider-key rotation triggers a reload: the new config is validated, then the provider registry, price table, resilience policy, the runtime-safe [auth] knobs (flush_interval_ms, retention_days), the webhook policy, the image-fetch policy and the tokenizer are atomically swapped.

The boot layer is read once at startup and needs a restart: [server] (bind address, body limit, stream timeouts), log_format, [telemetry] (its label allowlist becomes the Prometheus label set), auth.enabled, auth.db_path, the usage-log channel knobs (usage_channel_capacity, usage_batch_max, usage_flush_ms), [usage_events] (billing usage events for the Meilisearch Lab, ADR 015; also set by LAB_URL and LAB_INSTANCE_ID), and config_source. Details in Deployment.

Where the config lives

By default the whole config is the one TOML file the gateway was started with (config_source = "file"). Setting config_source = "db" moves everything outside the boot layer into the auth database instead, so the gateway can be configured entirely through the admin API with no config file to manage beyond a few boot keys. See Config source modes.

Viewing the live config over the admin API

GET /admin/config returns the current config document, verbatim: in file mode, the config file the gateway booted from; in db mode, the dynamic document stored in the database (an empty string before the first write). It requires the master key AND auth.enabled = true: the whole /admin/* router is only mounted when auth is on, so on a default deployment (where [auth] is disabled) this route does not exist at all and answers 404 rather than 401.

{
  "config": "<raw toml, byte for byte>",
  "hash": "<64 hex chars, BLAKE3>",
  "key_sources": { "openai": "env", "cohere": "stored", "local": "not_required" }
}

config is the document’s exact bytes, never a re-serialisation of the merged in-memory config: Config::load overlays LUMEN_* environment variables on top of it, and showing that merged view would make environment overrides look like document content, and a later write would bake them in permanently. hash is a BLAKE3 content hash of those same bytes, meant to be echoed as If-Match on the PUT /admin/config that applies a new one (ADR 010, ADR 012), so two operators editing at once cannot silently clobber each other.

To read or change one provider or one section without handling the whole document, use the granular endpoints (/admin/config/providers/{name}, /admin/config/{section}) described in Config source modes.

key_sources maps each provider in the dynamic document (the file in file mode, the stored document in db mode) to where its API key resolves from, using the same precedence as startup and hot reload: env (the provider’s api_key_env variable is set in the gateway’s environment, which wins), stored (a key stored via PUT /admin/provider-keys/{name} back-fills an unset variable), missing (no key anywhere and the provider kind needs one), or not_required (a keyless kind such as Ollama, TEI, vLLM, NIM or Bedrock). It reports the source only, never the key. If the document does not parse, key_sources is an empty object; config is still returned verbatim.

Applying a new config over the admin API

PUT /admin/config is the highest-privilege route in the gateway: it can repoint any provider’s base_url (or add a new provider entirely) and thereby redirect customer traffic to a different upstream. The granular provider endpoints can do the same. Master key required, same as every other /admin/* route, and likewise only mounted when auth.enabled = true: with auth off the route is absent and returns 404, not 401.

The If-Match contract:

• Send the submitted document as the raw request body (Content-Type is irrelevant; the body is treated as the document’s new contents).
• Send the hash from a prior GET /admin/config as the If-Match header. A missing If-Match header is rejected with 400 (LM-1001) - the request is malformed like any other missing-required-input case.
• If If-Match does not equal the current document’s hash (someone else applied a change since you last read it, or in file mode someone edited the file directly), the request is rejected with 412 (LM-1004) and nothing is written. GET /admin/config again to see what changed, then re-apply against the fresh hash.

What happens on a successful apply:

1. The candidate is validated exactly like a hot reload would: parsed, merged with LUMEN_* env vars, and used to build a candidate provider registry. A parse failure or a registry-build failure (e.g. a provider missing a required base_url) is rejected with 400 (LM-1001) and nothing is written. The error message names the offending field or setting (e.g. "server.first_token_timeout_ms must not be 0") but never a filesystem path, which is logged server-side instead.
2. A candidate that changes a restart-only (boot-layer) setting is rejected with 400 (LM-1001), naming it: the route only accepts what it can apply immediately. In db mode, a candidate that carries any boot-layer key at all is rejected the same way.
3. The document is persisted atomically. File mode: the current file is copied to a .bak sibling (e.g. lumen.toml.bak) - one generation of history, enough to revert a bad apply by hand - and the new document is renamed into place. DB mode: a new row is added to config_versions (the newest 50 are kept).
4. The hot-reload trigger fires, so the new config takes effect without a restart (see Hot reload). Every setting this route accepts is live once it returns 204.

A rejected apply (400 or 412) is guaranteed to leave the stored document byte-for-byte unchanged and to leave no temporary file behind. Concurrent PUTs are serialised gateway-side, so two operators racing the same pre-apply hash can never both land: exactly one wins, and the other sees a 412 for a hash that moved out from under it, never a silently corrupted mix of the two documents.

Security note: the master key is equivalent to host filesystem access. api_key_env accepts ANY environment variable name, not just ones a provider convention would suggest. Because PUT /admin/config lets a master-key holder add a provider with an attacker-controlled base_url and api_key_env pointed at any variable present in the gateway’s own process environment - LUMEN_MASTER_KEY itself, or any cloud credential the process happens to carry - a single crafted config plus one request to that provider’s model returns the named secret back to the caller as a Bearer token. Before remote config apply existed, reaching this required filesystem write access on the gateway host; with this route, the master key alone is enough. In practice: holding the master key is now equivalent to filesystem write access on the gateway host, plus read access to its entire process environment. Mitigate with a separate master key per gateway, exposing the admin surface on a private network only, and using the read-only config mount opt-out (see Deployment) on any file-mode gateway that does not need remote apply. In db mode the admin API is the only way to change the config, so there is no equivalent opt-out.

Validate before you boot

Run lumen --check-config --config config.toml to validate a config file without starting the server. See Installation.

Chat completions

POST /v1/chat/completions speaks the OpenAI request and response format. The model field is one of your configured model ids (the id in a [[providers.models]] block, not necessarily the upstream’s own model name - see Providers for aliasing with upstream_id).

Request

curl -s http://localhost:8080/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Say hello in one word."}]
  }'

Response

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "created": 1731000000,
  "model": "gpt-4o",
  "choices": [
    {
      "index": 0,
      "message": { "role": "assistant", "content": "Hello!" },
      "finish_reason": "stop"
    }
  ],
  "usage": { "prompt_tokens": 12, "completion_tokens": 3, "total_tokens": 15 }
}

Unknown fields pass through

Request fields LUMEN does not model as a typed struct field (tools, response_format, provider-specific extensions, …) are preserved verbatim and forwarded to the upstream untouched, rather than stripped. Provider-specific parameters keep working without waiting on a LUMEN release to add them by name.

Verbatim passthrough applies to OpenAI-compatible providers. On the translated kinds (anthropic, google, vertex_ai, bedrock, cohere), response_format, seed, logprobs, top_logprobs, logit_bias and parallel_tool_calls are mapped natively where the upstream supports them and otherwise dropped with a debug log - or rejected up front with LM-1001 when the provider sets strict = true. See the chat-extras matrix in Providers.

Routing and request errors

Full taxonomy in Error codes.

Fallbacks

If the requested id is a virtual model with a fallback strategy and the primary target fails, the request fails over automatically. The foundation model that actually served the request (primary or a fallback) is reported in the x-lumen-model-used response header, and the path taken in x-lumen-route. A virtual model can also carry a chat preset (a stored system prompt), which is applied to the request before routing. See Resilience.

Providers

Which provider kinds serve chat and their setup is in Providers.

Streaming

Add "stream": true to a /v1/chat/completions request and the response becomes text/event-stream: a series of data: {...} frames (each a chat.completion.chunk), terminated by a literal data: [DONE].

curl -s http://localhost:8080/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "gpt-4o",
    "messages": [{"role": "user", "content": "Count to 3."}],
    "stream": true
  }'

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1731000000,"model":"gpt-4o","choices":[{"index":0,"delta":{"role":"assistant","content":"1"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1731000000,"model":"gpt-4o","choices":[{"index":0,"delta":{"content":", 2, 3"},"finish_reason":null}]}

data: {"id":"chatcmpl-abc123","object":"chat.completion.chunk","created":1731000000,"model":"gpt-4o","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Passthrough design

When the upstream already speaks OpenAI-shaped SSE (the OpenAI-family kinds, vllm), the gateway forwards the upstream bytes to the client verbatim - no per-chunk deserialize/re-serialize round trip. Providers that translate a foreign event schema (anthropic, google) build typed chunks instead, which is not zero-copy but is required because they must translate the schema anyway. See ADR 004.

Heartbeats

If the stream goes idle for longer than sse_heartbeat_ms ([server] in config.example.toml), the gateway injects a : ping SSE comment to keep intermediate proxies from reaping the connection as silent.

Commitment: the first content frame, not the open

Retries and fallback stay live past the upstream opening the stream (2xx + headers): after opening, the gateway peeks the first frame before committing to the client. An upstream that opens 200 then errors, or closes, before delivering any content frame is a pre-commit failure - the gateway retries and falls over per the resilience policy (and penalises that provider’s circuit breaker) exactly like an open failure, instead of surfacing a terminal SSE error frame. A silent open (no bytes at all) still fails over via the first_token_timeout_ms bound on the peek. Only once the first content frame is forwarded does the request commit; from that point the guards below own the rest and nothing retries. See ADR 005 (first-frame-peek amendment).

Guards (post-commit)

• No first content frame within first_token_timeout_ms ([server]), across every retry/fallback attempt -> the request fails with LM-3011 (504).
• The upstream connection dies mid-stream (after commit) without a [DONE] terminator, or every link in the fallback chain fails to deliver a content frame at all -> the client receives a terminal SSE error frame carrying LM-3010 (502).

Both are documented in Error codes.

Client disconnect

If the client disconnects mid-stream, the upstream call is aborted and the request’s accounting settles at HTTP 499 (LM-6001, client_cancelled) - never counted as a 5xx internal error. See ADR 006 and Error codes.

Usage in streams

The final chunk carries usage when the upstream reports it there; otherwise the gateway falls back to a local estimate flagged "estimated": true. Every request produces a token count, never a silent zero - see Token accounting.

Vision (image input)

POST /v1/chat/completions accepts OpenAI’s content-parts message shape, so a user message can carry text and image parts in one array. It is opt-in per model: a model only accepts image parts once its config declares the image modality (modalities = ["text", "image"]; the default is ["text"]). GET /v1/models reflects the opt-in back as "modalities": ["text","image"] per model.

{
  "model": "gpt-4o",
  "messages": [{
    "role": "user",
    "content": [
      { "type": "text", "text": "What is this?" },
      { "type": "image_url", "image_url": { "url": "data:image/png;base64,iVBORw0KG..." } },
      { "type": "image_url", "image_url": { "url": "https://example.com/cat.png" } }
    ]
  }]
}

image_url.url is either a data:<media-type>;base64,<payload> inline URI or a remote http(s) URL.

Pre-flight

Sending an image part to a model whose modalities lack "image" is rejected with LM-2003 (400) before any upstream call. The check inspects the whole fallback chain, not just the primary, so a fallback missing the image modality is caught up front too.

Per-provider handling

OpenAI-family kinds, vllm and azure forward image parts verbatim - both data: URIs and remote URLs. anthropic and cohere translate both forms into their own schema (both upstreams fetch a remote URL themselves). google, vertex_ai and bedrock translate only inline data: URIs into their own schema; a remote http(s) URL routed to any of the three is rejected pre-flight with LM-2004 (400), since none of them fetches a URL itself and the gateway never fetches a chat image URL on the caller’s behalf (an SSRF vector it deliberately avoids). Full per-kind table in Providers - Vision.

Provider-native sources

Two provider-native reference forms are recognised in image_url.url, for callers whose images are already uploaded to the provider: an Anthropic Files API reference (anthropic-file:<file_id>) and a Gemini-native reference (gs://bucket/object, or a Gemini Files API URI under https://generativelanguage.googleapis.com/). A reference routed to a model whose primary provider does not match the reference’s own provider is rejected pre-flight with LM-2008 (400) instead of surfacing as a confusing upstream failure. Details, including the gs:// / Developer API caveat, are in Providers - Vision.

Token accounting

Upstream-reported usage is authoritative and already folds in image tokens. When an upstream reports no usage at all, the local estimation fallback counts each image content part with a flat per-image heuristic (85 tokens at "detail": "low", 765 tokens otherwise) rather than counting it as zero, and the response is still flagged "estimated": true. See Token accounting.

Tool calling

/v1/chat/completions accepts OpenAI’s tools request field and returns tool_calls on the response message, for every chat provider: passed through untouched for the OpenAI-family kinds, vllm and azure (via the same unknown-field passthrough that carries tools and tool_choice, see Chat completions), translated to and from the provider’s own schema for anthropic (tool_use content blocks), google/vertex_ai (tools[].functionDeclarations, toolConfig.functionCallingConfig), bedrock (toolConfig/toolUse/toolResult on the Converse API) and cohere (mostly a field rename - OpenAI-shaped tool_calls pass through largely unchanged - with tool_choice collapsing to Cohere’s REQUIRED/NONE strings; forcing one named tool has no v2 equivalent and falls back to auto).

Two-leg flow

1. Request with tools:

{
  "model": "gpt-4o",
  "messages": [{ "role": "user", "content": "What is the weather in Paris?" }],
  "tools": [{
    "type": "function",
    "function": {
      "name": "get_weather",
      "parameters": { "type": "object", "properties": { "city": { "type": "string" } } }
    }
  }]
}

2. Response with a tool call (finish_reason: "tool_calls"):

{
  "choices": [{
    "index": 0,
    "message": {
      "role": "assistant",
      "content": null,
      "tool_calls": [{
        "id": "call_1",
        "type": "function",
        "function": { "name": "get_weather", "arguments": "{\"city\":\"Paris\"}" }
      }]
    },
    "finish_reason": "tool_calls"
  }]
}

3. Follow-up request, appending the assistant’s tool call and a tool role message with the result:

{
  "model": "gpt-4o",
  "messages": [
    { "role": "user", "content": "What is the weather in Paris?" },
    { "role": "assistant", "content": null, "tool_calls": [ /* as above */ ] },
    { "role": "tool", "tool_call_id": "call_1", "content": "15C, cloudy" }
  ]
}

The model grounds its final answer in the tool result and returns a normal finish_reason: "stop" message.

Streaming

With "stream": true, tool_calls arrive as incremental deltas in OpenAI format (id and name first, then argument fragments), same as OpenAI’s own streaming tool-call shape. See Streaming.

Coverage

Translation is implemented for anthropic, google, vertex_ai, bedrock and cohere; the OpenAI-family kinds, vllm and azure pass tools/tool_choice through untouched since they already speak (or near-passthrough, for azure) the OpenAI shape. Provider setup and the full capability matrix are in Providers.

Embeddings

POST /v1/embeddings speaks the OpenAI request and response format. The model field is one of your configured model ids (the id in a [[providers.models]] block - see Providers).

Request

curl -s http://localhost:8080/v1/embeddings \
  -H 'content-type: application/json' \
  -d '{
    "model": "text-embedding-3-small",
    "input": ["the quick brown fox", "a lazy dog"]
  }'

Response

{
  "object": "list",
  "data": [
    { "object": "embedding", "index": 0, "embedding": [0.0023, -0.009, ...] },
    { "object": "embedding", "index": 1, "embedding": [0.0071, 0.014, ...] }
  ],
  "model": "text-embedding-3-small",
  "usage": { "prompt_tokens": 8, "total_tokens": 8 }
}

encoding_format

"encoding_format": "float" (the default) returns each embedding as a JSON float array. "base64" returns it as an OpenAI-style base64 string of little-endian f32 bytes instead. This is purely an output concern: LUMEN always holds vectors as floats internally, so it works uniformly even for providers with no native encoding_format, such as Ollama and TEI.

Accepted input shapes

Pre-tokenized shapes are rejected on providers that cannot consume them (see below). Content-part shapes require a model that opts into the image modality (see Multimodal embeddings).

Pre-tokenized input on text-only providers

Token-id array input ([1,2,3] or [[1,2],[3,4]]) passes through natively on OpenAI-compatible providers. Providers whose upstream API only accepts text - cohere, tei, ollama, jina, voyage, mistral - reject it before any upstream call with LM-1001 (400), naming the provider and the rejected shape. See Error codes.

Unknown fields and input_type (Cohere)

Unlike /v1/chat/completions, unknown request fields on /v1/embeddings are captured but never re-serialized into the outgoing provider body - they stop at the gateway rather than being forwarded, since a strict OpenAI-compatible upstream may reject fields it does not recognize. The one field the gateway itself reads is input_type, consumed only by the Cohere translation to override Cohere’s query-vs-document intent (search_query, search_document, classification, clustering; defaults to search_document). An unrecognized input_type is rejected with LM-1001 before any upstream call. See Providers - cohere.

Strict mode

By default a provider silently drops a request field it cannot honor (for example dimensions sent to Ollama, which has no such parameter). Setting strict = true on that provider’s [[providers]] block makes it reject such a request instead, with LM-1001 naming the field. encoding_format is always handled at the response edge and is never affected by strict.

Providers

Which provider kinds serve embed, and their batch limits, are in Providers.

Batching

An embed request with more inputs than the target provider’s batch limit is split into sub-batches, run with bounded concurrency, and reassembled in the original input order before the response is returned. This is invisible to the client: one request in, one response out, data[].index numbered against the original input regardless of how it was split upstream.

Where limits come from

Every provider kind has a built-in embed batch limit. The native kinds (openai, mistral, cohere, jina, voyage, tei, ollama, azure) each have their own limit; the OpenAI-compatible hosts all share a 2048-input limit. The exact numbers are in the Providers matrix, in the native-kinds table and in the OpenAI-compatible hosts section.

Usage

A batched request still produces exactly one usage object: prompt_tokens and total_tokens are summed across every sub-batch response before the client sees them.

Multimodal embeddings

An embed model opts into image input by declaring the image modality:

[[providers.models]]
id = "embed-multilingual"
capabilities = ["embed"]
modalities = ["text", "image"]

Once opted in, input items may be a bare string or an array of content parts ({"type":"text",...} / {"type":"image_url",...}), and both kinds of item can be mixed within one batch. Image input sent to a model without the image modality is rejected with LM-2003 (400) before any upstream call. See Embeddings for the other accepted input shapes.

Image sources

image_url.url is a data:<media-type>;base64,<payload> inline URI or a remote http(s) URL.

• data: URIs always work - the bytes travel in the request body, no fetch required.
• Remote http(s) URLs require [image_fetch] enabled = true. With fetching disabled (the default), a remote URL is rejected with LM-2005 (400) - pass a data: URI instead, or enable fetching.

The guarded fetch

When [image_fetch] is enabled, a remote URL is fetched server-side, base64-encoded, and inlined before being sent to the provider. The fetch is guarded:

• The private/loopback/link-local IP block is non-configurable (always on).
• The connection is pinned to the resolved address that was vetted against that block, so a DNS answer cannot rebind to a different address after the check (DNS-rebinding safe).
• Scheme, host, and URL-prefix allowlists (allowed_schemes, allowed_hosts, allowed_url_prefixes).
• A streamed size cap (max_bytes) and a fetch timeout (timeout_ms).
• Only image/* response content types are accepted.
• Redirects are re-validated against the same guards, not just the original URL.
• A per-request cap on the number of images fetched.

A request rejected by any of these guards fails with LM-2006 (400); the specific reason is logged server-side only, never returned to the client. A URL that passes the guards but fails at the remote host (network error, timeout, or error status) fails with LM-2007 (502). See Error codes.

Config reference

# ---------------------------------------------------------------------------
# Multimodal image fetching (M9) - OFF by default.
#
# When enabled, a remote http(s) image URL in a /v1/embeddings content-parts
# request is fetched server-side, base64-encoded, and inlined as a data: URI
# before being sent to the provider. Disabled → such a URL is rejected with
# LM-2005 (clients must pass a data: URI themselves).
#
# Guards (always on when enabled): the private/loopback/link-local IP block is
# non-configurable; the connection is pinned to the vetted resolved address
# (DNS-rebinding safe); downloads are capped and time-bounded; only image/*
# content types are accepted. Restrict `allowed_hosts` / `allowed_url_prefixes`
# to your own asset origins in production - leaving both empty with fetching on
# logs a startup warning.
# ---------------------------------------------------------------------------
[image_fetch]
enabled = false
max_bytes = 10485760          # 10 MiB per image
timeout_ms = 5000             # per-fetch timeout
allowed_schemes = ["https"]   # add "http" to allow plaintext
allowed_hosts = []            # e.g. ["cdn.example.com", ".mycompany.com"]; empty = any public host
allowed_url_prefixes = []     # e.g. ["https://cdn.example.com/images/"]; empty = no prefix restriction

Per-provider semantics

Not every multimodal provider embeds mixed input the same way: Cohere (embed-v4) and Voyage embed one combined text+image vector per item, while Jina embeds one modality per item - a mixed item is sent as its image, and its caption text is not combined into the vector. See Providers for the full per-kind notes.

Accounting

Every fetched or inlined image is counted: lumen_media_total and lumen_media_bytes_total in Prometheus, and matching columns in the usage_log table. See Metrics.

Reranking

POST /v1/rerank speaks the Cohere request and response format: query, documents, top_n. The model field is one of your configured model ids (the id in a [[providers.models]] block - see Providers).

Request

curl -s http://localhost:8080/v1/rerank \
  -H 'content-type: application/json' \
  -d '{
    "model": "rerank-english",
    "query": "What is the capital of France?",
    "documents": ["Paris is the capital of France.", "Berlin is in Germany."],
    "top_n": 2
  }'

Response

{
  "results": [
    { "index": 0, "relevance_score": 0.98 },
    { "index": 1, "relevance_score": 0.02 }
  ],
  "usage": { "search_units": 1, "total_tokens": 42, "tokens_estimated": true }
}

Results come back sorted by descending relevance_score. Each result’s index points back to the position of that document in the request’s documents array, not the sorted position.

Empty documents

documents must be non-empty. An empty list is rejected before any upstream call with LM-2010 (400). See Error codes.

Billing: search units and tokens

Rerank billing units vary by upstream: Cohere and Pinecone bill in search units (one unit is approximately one query over up to 100 documents), while Jina and Voyage bill in tokens. Set cost_per_1k_searches on a model’s [[providers.models]] block to price the search-unit case:

[[providers.models]]
id = "rerank-english"
upstream_id = "rerank-v3.5"
capabilities = ["rerank"]
cost_per_1k_searches = 2.0

Search units are counted on the lumen_rerank_search_units_total{model, provider} Prometheus counter for every request, upstream-reported when available, otherwise a gateway estimate (usage.estimated: true).

Independently of the billing unit, every /v1/rerank response also carries a usage.total_tokens count (ADR 003), for uniform token observability across capabilities: Jina and Voyage surface their upstream-reported token count unflagged; every other provider (Cohere, TEI, Pinecone, …) gets a gateway-derived query + documents heuristic flagged "tokens_estimated": true. This token count also feeds lumen_tokens_total{capability="rerank",...} and usage_log, alongside the search-unit accounting - see Token accounting & cost.

One model, two capabilities

A single model id can serve both embed and rerank if the underlying upstream model supports both, for example Cohere’s embed-v4.0:

[[providers.models]]
id = "embed-multilingual"
upstream_id = "embed-v4.0"
capabilities = ["embed", "rerank"]

Cross-vendor fallback

Like any capability, rerank can fall back across different provider kinds, through a virtual model. A three-hop chain across Cohere, Jina and Voyage survives any single vendor outage:

[[providers.models]]
id = "cohere/rerank-english"
upstream_id = "rerank-v3.5"
capabilities = ["rerank"]

[[virtual_models]]
id = "rerank-english"
capability = "rerank"
strategy = "fallback"
targets = [{ model = "cohere/rerank-english" }, { model = "jina-rerank" }, { model = "voyage-rerank" }]

The foundation model that actually served the request (primary or a fallback) is reported in the x-lumen-model-used response header, and the path taken in x-lumen-route. See Resilience.

Decision models as rerankers (Jev, Perplexity, OpenAI, Ollama, Cloudflare)

A rerank virtual model whose target carries a remap is served by a decision model. The target points at any foundation model that declares decisions (typesafe, perplexity, openai, ollama or cloudflare kinds; a typesafe model can no longer declare rerank itself). Each request becomes decision calls whose state is {"query": ...} and whose questions, by default, are one predicate (alias noul) per document, the document carried in structured instructions; the predicate’s calibrated probability that the document is relevant is the relevance_score. Jev receives the same bytes as before the rename; an OpenAI target receives the structured instructions as compact JSON text. The examples below use Jev.

[[providers]]
name = "typesafe"
kind = "typesafe"
api_key_env = "TYPESAFE_API_KEY"

[[providers.models]]
id = "jev"
upstream_id = "jev-latest"
capabilities = ["decisions"]
cost_per_1m_input = 0.042

# Optional remap fields: the question asked about every document. Unset
# fields default to a generic relevance question.
[[virtual_models]]
id = "jev-rerank"
capability = "rerank"
strategy = "single"

[[virtual_models.targets]]
model = "jev"
[virtual_models.targets.remap]
strategy = "predicate"
instructions = "Could `document` be the precedent cited in the query?"
criteria.true = "The document states the specific rule the query cites."
criteria.false = "The document is only on a similar topic."

• Besides predicate, the remap strategies are score, composite (weighted criteria) and choice, and a static context can be added; see Virtual models. strategy = "noul" is still accepted as an alias of predicate, and lumen config migrate rewrites it.
• Clients call plain /v1/rerank; ordering, top_n, return_documents, rank_fields and fallbacks (a fallback virtual model whose second target is cohere/rerank-english, say) work as for any reranker.
• Documents are packed into as few upstream calls as the target allows. The default is at most 100 documents and about 48k estimated tokens per call, 4 calls in flight. A target’s own limits tighten that: Cloudflare packs at most 64 documents per call; Ollama packs about 16k tokens per call with one call in flight (a local GPU). A listwise choice over more documents than the target’s option limit (Ollama 26, otherwise 255) is LM-1001. A document longer than about 4,096 tokens is truncated first, like Cohere’s default max_tokens_per_doc.
• Refusals: OpenAI can refuse a question. A refused predicate or score question scores that document (or that composite criterion) 0.0; a refused listwise choice fails the attempt as content_filter, so a virtual model with fallback_on = ["content_filter"] moves to its next target. Every refusal is counted in lumen_decision_refusals_total{model}, including those of a choice that then falls back.
• Because scores are calibrated probabilities, they are comparable across requests: a cut-off such as 0.5 means the same thing everywhere.
• Usage: the upstream input_tokens is usage.total_tokens (unflagged when the upstream reports it); search units are derived. The decision model’s cost_per_1m_input bills per input token.
• The remap belongs to a target of a virtual model. For per-tenant relevance rules, put a switch on the budget group in front of several such virtual models.

Providers

Which provider kinds serve rerank and their setup is in Providers.

Decisions (typed answers)

POST /v1/decisions answers typed questions about some content with calibrated probabilities, instead of generating text. A decision model gets an input (text, or text and images) and a list of questions, and returns one answer per question: a probability for a yes/no question, a distribution over options for a choice, a distribution over ordered levels for a score.

Decision models come from several vendors: TypeSafe (Jev), Perplexity (pplx-decider-*), OpenAI (gpt-6-luna), Cloudflare (Clef), local models through Ollama (Nimble, Clef, Kev) and any vendor that speaks the TypeSafe format (Liquid, Inception, Upstage). LUMEN serves all of them behind one capability, decisions, so any of them can be a target of the same virtual model (fallback, split, switch) and a reranker. The design and its trade-offs are in ADR 017.

The model field is one of your configured model ids: the id of a [[providers.models]] block declaring capabilities = ["decisions"], or a virtual model of capability decisions. Provider setup is in Providers.

Configuration

[[providers]]
name = "typesafe"
kind = "typesafe"
api_key_env = "TYPESAFE_API_KEY"

[[providers.models]]
id = "jev"
upstream_id = "jev-latest"
capabilities = ["decisions"]
cost_per_1m_input = 0.042

The old capability spelling systemone still loads (with a boot warning); lumen config migrate rewrites it.

Two request formats

/v1/decisions accepts both formats, detects which one from the body, and answers in the format it received. Clients of every vendor point at LUMEN with no code change: the OpenAI SDK, Perplexity-style and TypeSafe-format HTTP clients (TypeSafe clients that post to /v1/systemone switch the path to /v1/decisions, see Migrating).

Either format can be served by any provider kind: LUMEN translates between them. When the format of the request matches the family of the upstream (a TypeSafe-format request to Jev or Perplexity), the question bodies and state are sent verbatim, key order included, and the upstream answers come back verbatim, including unknown fields.

TypeSafe format

curl -s http://localhost:8080/v1/decisions \
  -H 'content-type: application/json' \
  -d '{
    "model": "jev",
    "state": "Help! My payouts have been failing for 3 days.",
    "questions": {
      "is_urgent": {
        "type": "noul",
        "instructions": "Does this message convey urgency?"
      },
      "department": {
        "type": "choice",
        "instructions": "Which team should handle this ticket?",
        "criteria": { "technical": "Bugs and outages", "billing": "Payments and invoices" }
      }
    }
  }'

state is the content to evaluate: a string, an object or an array. Each key of questions is your own question id, and the answer comes back under the same id.

{
  "model": "jev-1.13.0",
  "answers": {
    "is_urgent": { "type": "noul", "noul": 0.95 },
    "department": {
      "type": "choice",
      "choice": "billing",
      "probabilities": { "technical": 0.12, "billing": 0.88 },
      "confidence": 0.81
    }
  },
  "usage": { "input_tokens": 318, "output_tokens": 34 }
}

(The numbers are illustrative.) model is the versioned upstream model that answered (not your configured id). The id that actually served the request, primary or fallback, is in the x-lumen-model-used response header, as for every capability.

OpenAI format

curl -s http://localhost:8080/v1/decisions \
  -H 'content-type: application/json' \
  -d '{
    "model": "gpt-6-luna",
    "input": "Help! My payouts have been failing for 3 days.",
    "questions": [
      {
        "type": "predicate",
        "name": "is_urgent",
        "instructions": "Does this message convey urgency?"
      },
      {
        "type": "choice",
        "name": "department",
        "instructions": "Which team should handle this ticket?",
        "choices": [
          { "value": "technical", "description": "Bugs and outages" },
          { "value": "billing", "description": "Payments and invoices" }
        ]
      }
    ]
  }'

Answers come back as an array in question order:

{
  "model": "gpt-6-luna",
  "answers": [
    { "type": "predicate", "name": "is_urgent", "probability": 0.95 },
    {
      "type": "choice",
      "name": "department",
      "choice": "billing",
      "probabilities": [
        { "value": "technical", "probability": 0.12 },
        { "value": "billing", "probability": 0.88 }
      ],
      "confidence": 0.88
    }
  ],
  "usage": {
    "input_tokens": 318,
    "input_tokens_details": { "cached_tokens": 0, "cache_write_tokens": 0 },
    "output_tokens": 34,
    "output_tokens_details": { "reasoning_tokens": 0 },
    "total_tokens": 352
  }
}

When the upstream is not OpenAI, usage is completed to OpenAI’s schema so typed SDKs deserialize it, and "estimated": true is added only when LUMEN estimated the count. The OpenAI SDKs type confidence as required on choice and score answers: when the upstream gave none, LUMEN uses the highest probability.

With the OpenAI SDK, point base_url at LUMEN (use an SDK release that includes client.decisions):

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8080/v1", api_key="sk-lumen-...")  # any value when auth is off

response = client.decisions.create(
    model="gpt-6-luna",
    input="Help! My payouts have been failing for 3 days.",
    questions=[
        {"type": "predicate", "name": "is_urgent", "instructions": "Does this message convey urgency?"},
    ],
)
print(response.answers[0].probability)

name is optional but must be unique when present. safety_identifier (at most 128 characters) is forwarded to OpenAI and dropped for other vendors.

Validation and errors

LUMEN checks the documented request contract itself, before any upstream call, so a malformed question is a precise 400 instead of an opaque upstream rejection.

• LM-2011 (400): empty questions (an empty array or an empty map).
• LM-1001 (400): every other contract violation, with a message naming the offending question. A body that is not JSON, or that matches neither format; a missing model; in the OpenAI format, an unknown top-level parameter (as OpenAI itself answers), more than 128 images, a choice with fewer than 2 or more than 255 options, a duplicate name; in the TypeSafe format, a null state, a missing or unknown question type, missing instructions (unless a noul has criteria), a malformed criteria. A request that no target in the chain can take is also LM-1001, found before any call (see Cross-vendor fallback).
• LM-2003 (400): an image sent to a model whose modalities lack "image". For a virtual model, the message names the leaf model that could not take the image, not the virtual id.
• LM-2004 (400): an image that is not a data: URL. Remote http(s) image URLs are refused in both formats.

There is no question-count cap at the gateway edge; per-target caps apply when a target is chosen (Perplexity 128 questions, Cloudflare 64 questions and 4 images, Ollama choice and score 2 to 26 options, OpenAI choice at least 2 options). All codes are in Error codes.

For example, an empty choice:

curl -s http://localhost:8080/v1/decisions \
  -H 'content-type: application/json' \
  -d '{
    "model": "jev",
    "state": "Help! My payouts have been failing for 3 days.",
    "questions": { "department": { "type": "choice", "instructions": "Which team?", "criteria": {} } }
  }'

{ "error": { "code": "LM-1001", "message": "question `department`: choice needs 1 to 255 options, got 0", "type": "invalid_request" } }

Upstream failures follow the usual mapping (see Error codes):

An upstream 401 means the gateway’s upstream key is wrong: it surfaces as a 502 naming the provider, never as a misleading 401 to your client.

Images

Declare modalities = ["text", "image"] on the models that take images (the default is ["text"], so nothing changes silently):

[[providers.models]]
id = "gpt-6-luna"
capabilities = ["decisions"]
modalities = ["text", "image"]
cost_per_1m_input = 0.10

A client sends images in one of two ways at the edge, and LUMEN places them where each upstream expects them:

Upstream, the image lands in one of three places, by provider kind: inside state as image_url parts (perplexity), in a top-level images array (ollama with raw base64, cloudflare), or as OpenAI’s input_image (openai). A model without "image" in its modalities never receives an image: it is skipped in a fallback chain, or the request fails with LM-2003 when nothing is left.

Images make bodies large: one 1024x1024 PNG is about 1 to 3 MiB as base64, and a request with many images can far exceed the default 10 MiB server.body_limit. Raise it to 32 MiB (Perplexity’s own limit) for deployments that send images:

[server]
body_limit = 33554432   # 32 MiB

The symptom of a too-small limit is a 413 with LM-1002.

Cross-vendor fallback

Any decision model can be a target of the same virtual model. For example, Perplexity first (cheapest), then Jev, then gpt-6-luna:

[[providers]]
name = "perplexity"
kind = "perplexity"
api_key_env = "PERPLEXITY_API_KEY"
[[providers.models]]
id = "pplx-decider"
upstream_id = "pplx-decider-v1.1-27b"
capabilities = ["decisions"]
modalities = ["text", "image"]
cost_per_1m_input = 0.02

[[providers]]
name = "typesafe"
kind = "typesafe"
api_key_env = "TYPESAFE_API_KEY"
[[providers.models]]
id = "jev"
upstream_id = "jev-latest"
capabilities = ["decisions"]
cost_per_1m_input = 0.042

[[providers]]
name = "openai"
kind = "openai"
api_key_env = "OPENAI_API_KEY"
[[providers.models]]
id = "gpt-6-luna"
capabilities = ["decisions"]
modalities = ["text", "image"]
cost_per_1m_input = 0.10

[[virtual_models]]
id = "cross"
capability = "decisions"
strategy = "fallback"
targets = [{ model = "pplx-decider" }, { model = "jev" }, { model = "gpt-6-luna" }]

Clients send "model": "cross" in either format. Before any upstream call, each target of the chain is checked against the request, and an incompatible one is skipped: it is not an attempt for metrics or usage_log, never counts as a circuit-breaker failure, and is logged at debug without content. If no target is left, the first incompatibility is returned.

Refusals. OpenAI is the only vendor that can answer {"type": "refusal"} for a question. An OpenAI-format client receives it as is. A TypeSafe-format client receives {"type": "refusal"} under that question id, a shape the TypeSafe and Perplexity SDKs do not know and may fail to deserialize. That only happens when an operator puts an openai-kind target behind a TypeSafe-format client, as in the cross example. Every refusal is counted in lumen_decision_refusals_total{model}. In a rerank remap, a refused predicate or score scores 0.0 and a refused listwise choice fails the attempt as content_filter, which a virtual model with fallback_on = ["content_filter"] fails over on.

Token accounting and cost

Every /v1/decisions response carries a token count (ADR 003):

• When the upstream reports usage with a non-zero input_tokens, it is used as is, unflagged. A missing or malformed usage never fails a billed answer.
• Otherwise the gateway estimates input tokens with its byte heuristic (bytes / 4) over all text of the input and every question as sent, plus a flat per-image estimate, and flags the response "estimated": true. Output is estimated as 0 (answers are tiny and not billed); an upstream output_tokens count is kept when present. Admission reserves the same estimate before the upstream call.

Decision models bill input tokens only, so price them with cost_per_1m_input and no output price:

OpenAI’s cached_tokens and reasoning_tokens are passed through, not billed. Prices are the published ones as of 2026-10; set them on your models, the gateway does not fill them in.

The counts feed lumen_tokens_total{capability="decisions",direction="input"|"output",...} and, when auth is on, usage_log rows with capability = 'decisions'. Old rows written under the previous name stay systemone: GET /admin/usage?capability= accepts decisions or systemone and matches both, and group_by=capability shows old rows as systemone and new rows as decisions. Virtual keys, RPM/TPM quotas and hard budgets apply exactly as for the other capabilities. See Token accounting & cost and Keys, quotas & budgets.

Privacy. input, images, state, questions, answers and safety_identifier are request content: they are never logged, never written to usage_log, and never included in errors.

Local decision models with Ollama

Ollama 0.35 and later serves decision models on POST /v1/systemone, with no auth. Pull a model and add an ollama provider:

ollama pull nimble

[[providers]]
name = "ollama-local"
kind = "ollama"
base_url = "http://localhost:11434"   # the server root, no /v1

[[providers.models]]
id = "nimble"
upstream_id = "nimble"
capabilities = ["decisions"]

Limits to know: a choice or score takes 2 to 26 options or levels; images are accepted only by clef and clef-flash (declare their modalities accordingly); unknown request fields are stripped. For a rerank remap, an Ollama target gets about 16k tokens per call and one call at a time (a local GPU), and a listwise choice over more than 26 documents is LM-1001. A server that is not running is LM-3004 and falls back like any unavailable upstream. A model can take a while to load into VRAM on its first call: relax first_token_timeout_ms and total_timeout_ms on the provider block (see Providers).

Migrating from /v1/systemone

POST /v1/systemone, the TypeSafe-format-only route of 0.6.x, is removed in 0.7.0: it now answers 404 LM-1003 like any unknown route.

1. Send the same body to /v1/decisions. Nothing else changes in the request or the response: /v1/decisions detects the TypeSafe format and answers in it.
2. Rewrite your config. Run lumen config migrate (add --dry-run to preview): it renames the capability systemone to decisions and the remap strategy noul to predicate. The old spellings still load, with a boot warning.
3. Update dashboards. The capability label of every metric and the capability in GET /v1/models change from systemone to decisions. This is a breaking change for queries that filter on the old value. usage_log keeps old rows as they were (see above).

The TypeSafe SDKs call /v1/systemone, so they no longer go through LUMEN from 0.7.0: call /v1/decisions with the same body over plain HTTP instead. Upstream providers are unaffected: the gateway still calls TypeSafe, Ollama and the other TypeSafe-family vendors on their own /v1/systemone.

Limitations

• No streaming. /v1/decisions is request/response only.
• Jev rate limits (TypeSafe, 2026-09): 64k tokens of context per request (32k for state plus the longest question), 250k tokens/s and 1200 RPM, dynamic. Perplexity: under 262,144 input tokens, 32 MiB body, 10 RPS per org. LUMEN does not check context limits itself: a request an upstream rejects surfaces as LM-3003. To stay under upstream rate limits across tenants, set per-key RPM/TPM quotas.
• Reranking goes through /v1/rerank, not this endpoint: declare a rerank virtual model whose target carries a remap onto a decision model; see Decision models as rerankers. Per-tenant relevance rules use a switch on the budget group (design notes).

Providers

Which provider kinds serve decisions, with URLs, auth and limits, is in Providers.

Virtual models

A virtual model is a public id, such as acme/legal-rerank, that carries routing logic: ordered fallback, weighted split, conditional routing, request templates, and Jev answering /v1/rerank. It sits on top of the foundation models you declare under [[providers.models]], which stay plain upstream bindings. The design is recorded in ADR 014.

The examples below assume the foundation models of config.example.toml (openai/gpt-4o, claude-sonnet-4-5, jev, cohere/rerank-english, and so on). Every snippet is valid TOML; note that a TOML inline table { ... } must stay on one line, so long targets use [[virtual_models.targets]] sub-tables.

1. Foundation models vs virtual models

Both layers share one id namespace. Ids are non-empty and use only the characters A-Z a-z 0-9 . _ : / -; company/model is a convention, not a rule. A collision between a foundation id and a virtual id is a validation error naming both. A virtual model serves exactly one capability (chat, embed, rerank or decisions; systemone is accepted as an alias of decisions); to offer the same product name for two capabilities, define two ids.

Virtual models live in the dynamic config document (config modes): they hot-reload and are editable through the admin API (section 9). An in-flight request keeps the plan it started with.

GET /v1/models lists virtual models next to foundation models, with "virtual": true, a single-entry capabilities array, the optional description, and the modalities common to every foundation model the virtual model can reach. Foundation entries carry "virtual": false. Set listed = false to hide a virtual model from the list; it stays callable.

[[virtual_models]]
id = "gpt-4o"
capability = "chat"
strategy = "fallback"
description = "GPT-4o with a Claude backup"
listed = true
targets = [{ model = "openai/gpt-4o" }, { model = "claude-sonnet-4-5" }]

Calling an id that is neither kind is LM-2001; calling a virtual model on the wrong endpoint (say a rerank model on /v1/chat/completions) is LM-2002.

2. Strategies

Every virtual model has one strategy:

Field placement is enforced: weight only under split, when only under switch, fallback_on only on fallback and split, preset only on chat.

# 80% of traffic to OpenAI, 20% to Azure OpenAI; a failing pick moves to the other.
[[virtual_models]]
id = "acme/chat-ab"
capability = "chat"
strategy = "split"
targets = [{ model = "openai/gpt-4o", weight = 80 }, { model = "azure-gpt-4o", weight = 20 }]

Composition and limits

A target’s model may be another virtual model, so a fallback can point at a split, a switch branch at a fallback, and so on. At load (boot, reload, and every admin write) the gateway checks the following and reports the first failure with the virtual model id and the reason:

1. every model reference exists;
2. strategy shape (the table above);
3. every target serves the virtual model’s capability, or carries a remap that makes it compatible (section 7);
4. no cycles (the error prints the path, for example a -> b -> a);
5. nesting depth at most 8 virtual models from the requested id down to a foundation model;
6. at most 64 attempts per request: a fallback or split counts the attempts of every target, a switch only its largest branch, and a foundation target counts 1 (a DAG that lists the same child several times per level multiplies quickly; the error names the model and its count);
7. the contents of overrides, preset and remap.

A preset on a virtual model that is only reached through another one is ignored at request time (section 6), and loading logs a warning.

3. fallback_on triggers

fallback_on lists the failures that let a fallback or split move on to its next target.

• Default when fallback_on is omitted: ["provider_error", "rate_limited", "timeout", "circuit_open"], exactly the behaviour of the earlier per-model fallbacks.
• context_length and content_filter rely on the provider classifying its upstream error body. A provider that cannot classify an error leaves it as a plain client error, and nothing fails over. Classification is implemented for OpenAI and the OpenAI-compatible hosts, Anthropic, Google/Vertex, Bedrock, Mistral and Cohere.
• Never triggers: a malformed upstream response (LM-3002), a non-retryable upstream status (for example a TypeSafe 401 or 422, which surface as LM-3003 but are not retried), any other client 4xx, a client cancellation (which never touches a circuit breaker, ADR 006), and any failure after the first streamed byte. Once a streaming response has delivered content the request is committed; a later error becomes an SSE error frame, never a fail-over (Streaming).
• Retries still happen inside each target first (Resilience tuning), and [resilience].total_timeout_ms bounds the whole tree (LM-3013).

# Also fail over when the prompt is too long for the first model.
[[virtual_models]]
id = "acme/long-context"
capability = "chat"
strategy = "fallback"
fallback_on = ["provider_error", "timeout", "context_length"]
targets = [{ model = "openai/gpt-4o" }, { model = "gemini-long" }]

4. switch conditions

A switch target carries a when table. Attributes:

The virtual key id is deliberately not an attribute. Using an attribute the capability never has (for example has_tools on a rerank model) is a validation error.

Quote metadata.<key> in TOML. An unquoted metadata.plan = "pro" is a nested table metadata = { plan = "pro" } and is rejected; write "metadata.plan" = "pro".

Operators. A bare value means equality. Otherwise use a one-key table: in (a list), ne, gt, gte, lt, lte, or regex (compiled at load; an invalid regex is a validation error). Several keys in one when table are ANDed. any = [ {...}, {...} ] is an OR of tables and can nest. A missing attribute makes its comparison false (including ne), so the branch does not match and evaluation moves on to the next rule.

[[virtual_models]]
id = "acme/chat"
capability = "chat"
strategy = "switch"
targets = [
  { when = { group = "eu-customers" }, model = "acme/chat-eu" },
  { when = { "metadata.plan" = { in = ["pro"] }, input_tokens = { gt = 32000 } }, model = "gemini-long" },
  { when = { any = [{ has_images = true }, { has_tools = true }] }, model = "gpt-4o" },
  { model = "local-llama" },
]

The first rule sends a whole budget group to a regional model. The second needs both the metadata plan and a long input. The third sends image or tool requests to a capable model. The last target has no when and is the default. The regex operator works the same way: { group = { regex = "^tenant-[0-9]+$" } }.

5. Overrides

overrides on a target rewrites request fields for that target:

targets = [{ model = "openai/gpt-4o", overrides = { set = { max_tokens = 4096 }, default = { temperature = 0.3 }, drop = ["seed"] } }]

• set always applies; default applies only when the client did not send the field; drop removes it.
• On translated providers (Anthropic, Bedrock, Google, Cohere) a client-sent max_tokens wins over max_completion_tokens. To cap output tokens there with an override, set max_tokens.
• Overrides act on the parsed request, not on JSON paths, and each capability has an allow-list of fields:

An unknown field, a field outside the allow-list, or a value of the wrong type is a validation error.

Order of application for one request: the requested virtual model’s preset (section 6) first, then target overrides from the outermost virtual model inward, so the level closest to the foundation model wins.

6. Presets (chat)

A preset gives a chat virtual model a stored system prompt and its own sampling defaults:

[[virtual_models]]
id = "acme/support-bot"
capability = "chat"
strategy = "fallback"
targets = [{ model = "gpt-4o" }, { model = "claude-sonnet-4-5" }]
description = "Acme support assistant"
preset = { system_prompt = "You are Acme's support assistant. Answer briefly and point to the documentation when relevant.", system_prompt_mode = "prepend", overrides = { default = { temperature = 0.2 }, set = { max_tokens = 800 } } }

• system_prompt is at most 32 KiB and must not be blank. overrides uses the syntax and allow-list of section 5.
• Token accounting. The injected prompt is part of the request: it counts in usage.prompt_tokens, in the usage log and in cost, exactly as if the client had sent it. It is applied before routing, so input_tokens switch conditions count it too.
• Only on the requested id. A preset applies only when its own virtual model is the id the client sent. A preset on a virtual model reached through composition is ignored, and validation warns about it at load.
• Preset text is operator-authored config: never client-supplied, never logged, never echoed in errors.

7. Decision models as rerankers

A remap on a target of a rerank virtual model converts POST /v1/rerank into decision calls. The target must point directly at a foundation model that serves decisions: any of the typesafe (Jev and other TypeSafe-format vendors), perplexity, openai, ollama and cloudflare kinds. A typesafe model can no longer declare rerank itself.

The state sent upstream is { "query": <query> }, plus "context" when set. Each question’s instructions are structured as { "document": <document>, "question": <instructions> }. Jev receives exactly these bytes; an OpenAI target receives them as compact JSON text.

context is a static, operator-authored string sent in the state next to the query, for rules that hold for every request of this virtual model (“US federal case law. Prefer holdings over dicta.”).

Batching and limits

• predicate, score and composite are batched: by default at most 100 documents and about 48k estimated tokens per upstream call, with 4 calls in flight. Each target’s limits tighten that: Cloudflare packs at most 64 documents per call, Ollama about 16k tokens per call with one call in flight. For composite the per-call budget counts every question of every document.
• choice must fit one call, because its scores are relative and cannot be merged across calls: at most 255 documents (26 on Ollama) and within the per-call token budget. Otherwise the request fails with LM-1001 (400) naming the limit, before any upstream call. A choice over one document returns relevance 1.0 without an upstream call.
• Documents are truncated to about 4,096 estimated tokens.

Refusals. An OpenAI target can refuse a question. A refused predicate or score scores that document (or that composite criterion) 0.0. A refused choice fails the attempt as content_filter: with fallback_on = ["content_filter"] on the virtual model, the request moves to the next target, otherwise it surfaces as LM-2013. Every refusal increments lumen_decision_refusals_total{model}, including a refused choice that falls back.

Accounting. usage.total_tokens is the sum of the input_tokens the decision model reports over all calls (upstream-reported, not estimated). Cost is the decision model’s cost_per_1m_input; search_units stay derived from the document count. x-lumen-model-used names the decision model.

Jev with a Cohere fallback. A remap target and a plain reranker in one fallback give a cross-capability fallback. A TypeSafe 529 Overloaded is a provider_error, so the default triggers already send the request to Cohere. This composite example is also in config.example.toml:

[[virtual_models]]
id = "acme/legal-rerank"
capability = "rerank"
strategy = "fallback"

[[virtual_models.targets]]
model = "jev"
[virtual_models.targets.remap]
strategy = "composite"
context = "US federal case law. Prefer holdings over dicta."
[[virtual_models.targets.remap.questions]]
instructions = "Is `document` on the topic of the query?"
weight = 0.3
criteria.true = "It discusses the same legal question."
criteria.false = "It is about something else."
[[virtual_models.targets.remap.questions]]
instructions = "Does `document` state the rule the query cites?"
weight = 0.7
criteria.true = "It states the specific rule."
criteria.false = "It only mentions related topics."

[[virtual_models.targets]]
model = "cohere/rerank-english"

Per-tenant relevance rules combine remap with switch on the budget group:

[[virtual_models]]
id = "acme/rerank"
capability = "rerank"
strategy = "switch"
targets = [{ when = { group = "law-firm" }, model = "acme/legal-rerank" }, { model = "jev-rerank" }]

The other strategies in one line each:

[[virtual_models]]
id = "acme/graded-rerank"
capability = "rerank"
strategy = "single"

[[virtual_models.targets]]
model = "jev"
remap = { strategy = "score", instructions = "How well does `document` answer the query?", levels = ["off topic", "related", "partial answer", "full answer"] }

[[virtual_models]]
id = "acme/pick-rerank"
capability = "rerank"
strategy = "single"

[[virtual_models.targets]]
model = "jev"
remap = { strategy = "choice", instructions = "Which document best answers the query?" }

score with 4 levels gives 0, 1/3, 2/3 and 1. Remap text is operator-authored config: never client-supplied, never logged, never echoed in errors.

8. Observability

Tokens and cost are booked to the foundation model that served, so a fallback that fires costs what the fallback model costs (token accounting).

9. Admin API

All routes are under the config admin surface (config modes) and work in both config_source modes.

{id} is percent-encoded when it contains /:

curl -s -H "Authorization: Bearer $LUMEN_MASTER_KEY" \
  http://localhost:8080/admin/config/virtual_models/acme%2Flegal-rerank/plan

The /plan body shows preset and remap text, since it is an admin route behind the master key; it is never logged.

Dependency safety. Deleting a provider, a foundation model or a virtual model that another virtual model still references is LM-1001, naming the dependent. Every write is validated with the same rules as boot, so an invalid virtual model (a cycle, a missing reference, a bad when) is LM-1001 naming the model and the reason.

10. Migrating from fallbacks

Per-model fallbacks, the [providers.models.rerank] block, and the rerank capability on typesafe models are removed. A config that still uses one fails validation naming the model, and prints the converted snippet.

lumen config migrate -c config.toml --dry-run   # print the result, change nothing
lumen config migrate -c config.toml             # rewrite (file mode keeps config.toml.bak)

lumen config migrate [-c PATH] [--dry-run] rewrites the document. In file mode it backs the file up to <file>.bak and replaces it atomically; in DB mode it writes a new config_versions row. Formatting and comments outside the edited tables are kept, and the result is validated before anything is written.

The rename rule. To keep clients working unchanged, the virtual model takes over the public id and the foundation model is renamed:

• A foundation model M on provider P with fallbacks = [F1, F2] becomes the foundation model P/M, and a virtual M is created with strategy = "fallback" and targets = [P/M, F1, F2]. The renamed model keeps sending its old upstream id: the migrator pins upstream_id to the old value when the model had none.
• Only models with fallbacks are renamed. If P/M already exists, the migrator stops with an error naming it.
• If M declares several capabilities, one id can carry only one virtual model: the virtual M gets the first declared capability, and the others are printed as needing a decision. Until you add a second virtual model, a call to M for another capability returns LM-2002; it still works through P/M, without fallback. --dry-run shows this before anything is written.
• A typesafe model R declaring rerank (with or without a converter block) becomes a virtual R with capability = "rerank", strategy = "single", and one target with a predicate remap that copies the converter’s fields, pointing at the provider’s decisions foundation model with the same upstream_id. If none exists, the migrator creates P/<upstream_id> with capabilities = ["decisions"]. When the prices of the removed reranker differ from that model’s, a note asks you to review them.
• The capability spelling systemone (model capabilities, virtual model capability) becomes decisions, and remap.strategy = "noul" becomes predicate (ADR 017). Both old spellings keep loading without the migrator and log a boot warning.

Visible side effect. usage_log.model_used and the Prometheus model labels of renamed foundation models change, for example gpt-4o becomes openai/gpt-4o. The id clients send does not change. Update dashboards and queries that filter on the old value.

Before and after:

# before (no longer valid)
[[providers.models]]
id = "gpt-4o"
capabilities = ["chat"]
fallbacks = ["claude-sonnet-4-5"]

# after
[[providers.models]]
id = "openai/gpt-4o"
upstream_id = "gpt-4o"
capabilities = ["chat"]

[[virtual_models]]
id = "gpt-4o"
capability = "chat"
strategy = "fallback"
targets = [{ model = "openai/gpt-4o" }, { model = "claude-sonnet-4-5" }]

See also ADR 014 and the Error codes.

Token accounting & cost

EVERY request of every capability (chat, embed, rerank) produces a token count. Never a silent zero: upstream usage is used when a provider reports it, otherwise the gateway falls back to a local estimate flagged "estimated": true. This is a central reason LUMEN exists, not an afterthought - see ADR 003.

Why it has to be guaranteed

Upstream usage reporting is inconsistent across providers: TEI’s /embed returns a bare vector array with no usage at all; streaming chat only carries usage when the client asks for it (and some providers omit it even then); rerank is billed in search units by Cohere and Pinecone but in tokens by Jina/Voyage, and TEI reports neither. A gateway that only passes through whatever the upstream says would show 0 tokens for a large slice of traffic. LUMEN closes that gap: a count is guaranteed for every call, and the count is always honest about whether it was measured or estimated.

Where it surfaces

1. Response body - the OpenAI-compatible usage object on chat and embeddings responses, and usage.search_units plus usage.total_tokens on rerank responses (rerank carries both a search-unit and a token count, each independently flagged - estimated for search_units, tokens_estimated for total_tokens - see Reranking - Billing).
2. Prometheus - lumen_tokens_total (every capability, including rerank’s token count) and lumen_rerank_search_units_total (rerank’s search-unit count), both carrying an estimated (or upstream-reported) signal. See Metrics & dashboards.
3. usage_log (when [auth] is enabled) - per-request token counts, cost and the estimated flag, alongside status and metadata for later slicing. See Usage log & multi-tenant metadata.

Cached and reasoning token breakdown

When an upstream reports a token breakdown, the gateway surfaces it instead of discarding it (issue #99), on the same three sinks and under the same “never zero, never invented” rule:

• Response body - an OpenAI-compatible usage.prompt_tokens_details (cached_tokens) and usage.completion_tokens_details (reasoning_tokens). For Anthropic prompt caching, cache_read_input_tokens maps to cached_tokens (a cache read/hit, same as OpenAI) and cache_creation_input_tokens rides a distinct cache_creation_tokens field in prompt_tokens_details (a cache write, which has no OpenAI equivalent).
• Prometheus - lumen_token_breakdown_total{capability, model, provider, kind} with kind one of cached, reasoning, cache_write. It is a subset of lumen_tokens_total, split out so cache and reasoning usage can be summed on their own.
• usage_log - the nullable cached_tokens, reasoning_tokens and cache_write_tokens columns.

A breakdown is only ever an upstream fact: when the upstream reports none, all three surfaces omit it (None/NULL/no series), and a locally estimated count carries no breakdown at all.

How estimation works

Source priority, in order:

1. Upstream-reported usage. Authoritative and free - already present in the response body or the final SSE chunk. estimated = false.
2. Local estimation fallback, when the upstream omits usage. estimated = true. The default is a cheap, allocation-light byte/char heuristic that is safe to run anywhere, including the request path; an accurate tokenizer is an opt-in, off-hot-path option run via spawn_blocking.

The hot-path rule holds regardless: the request path never runs a heavy tokenizer. Upstream usage is passed through as-is; when it is missing, the cheap heuristic fills in the response usage field, flagged estimated, and never blocks or slows the request. Streaming chat estimates the same way when the upstream sends no usage at all in its final chunk.

Images. Upstream-reported usage already folds in image cost (OpenAI, Anthropic and Gemini all report image tokens as part of prompt_tokens), so a vision request with upstream usage is exactly as accurate as a text-only one. When the upstream reports nothing, the two capabilities diverge:

• Chat counts each image content part with a flat per-image estimate (85 tokens at "detail": "low", 765 tokens otherwise) instead of counting it as zero. See Vision (image input).
• Embeddings still estimates image parts as zero tokens when the upstream reports no usage; this undercounts image-heavy requests on a no-usage upstream, and a per-image heuristic for embeddings is a backlog item. Media volume itself (count and decoded bytes) is tracked separately via lumen_media_total / lumen_media_bytes_total and the usage_log media_count/media_bytes columns, not through the token counters. See Multimodal embeddings.

Either way, a locally-estimated count is always flagged "estimated": true

• the client is never told a number is measured when it is not.

Cost

Per-model prices feed cost accounting and hard budgets:

[[providers.models]]
id = "gpt-4o"
capabilities = ["chat"]
cost_per_1m_input = 2.5
cost_per_1m_output = 10.0

Rerank models price by search unit instead:

[[providers.models]]
id = "rerank-english"
capabilities = ["rerank"]
cost_per_1k_searches = 2.0

A model without prices set costs 0, so hard budgets never bite on it. See Keys, quotas & budgets for how cost feeds budget enforcement.

Metrics & dashboards

GET /metrics exposes every gateway metric in Prometheus text exposition format. It is unauthenticated by design - restrict it at the network layer (firewall, reverse proxy, service mesh) rather than expecting the gateway to gate it. See SECURITY.md.

Metrics reference

lumen_tokens_total, lumen_rerank_search_units_total, lumen_media_total and lumen_media_bytes_total also gain one extra label per key listed in telemetry.metadata_labels - see Usage log & multi-tenant metadata for the allowlist mechanics and a multi-tenant query example.

Per-request cost is not a Prometheus series: it is a usage_log-only figure, computed from the cost_per_1m_* / cost_per_1k_searches prices in config. See Token accounting & cost.

What to alert on

Several failure modes are deliberately silent on the request path: the gateway sheds work instead of failing requests, and the only trace is a counter. “The drop is visible in your dashboards” is only true if something watches these:

The monitoring rig ships these as ready-made Prometheus rules in monitoring/prometheus/alerts.yml - copy them into your own Prometheus and tune the windows. Routing them to a pager needs an Alertmanager, which the rig intentionally leaves out.

See it live: the monitoring rig

monitoring/ is a one-command Docker Compose stack that runs the gateway against real providers with Prometheus scraping it and a pre-provisioned Grafana dashboard covering every metric above: token rates by provider/model/capability/direction, rerank search units, media accounting, circuit-breaker state and the internal counters. ./smoke.py exercises chat, streaming, embeddings and rerank per provider and asserts a non-zero token count on each; ./traffic.py generates sustained randomized traffic, tagged with multi-tenant metadata, so the dashboard has something to show. It is the fastest way to see all of this live - see monitoring/README.md.

Logging

Logs are the third observability leg next to metrics and the usage log. The gateway writes structured logs to stdout only (no file appender, no rotation of its own - that is the supervisor’s or the log pipeline’s job).

What is never logged

Prompts, responses and provider secrets are treated as radioactive:

• Request and response content is never logged, at any level, including debug and trace. This is the sovereignty pillar, not a default you can toggle. The nearest thing is an upstream’s own error message, which is logged only after request echoes are cut out of it (see Upstream error messages).
• Provider API keys and the master key never appear in logs or errors; the redacting Debug implementations keep them out of debug output, and tests enforce it.

What is logged is metadata: request ids, models, providers, HTTP status, latency, operational events (boot, config reloads, circuit-breaker transitions, flush failures), the upstream’s own error message on a failed call (below), and the client-supplied x-lumen-metadata header when present (a bounded JSON object of up to 16 keys, whole header capped at 4 KiB). Operators must not include secrets or prompt content in the metadata header, as it is logged in full and stored in the usage_log table.

Upstream error messages

When an upstream answers a client error (4xx other than 429) whose JSON body carries a message string, the gateway logs one warn line, upstream returned an error, with provider, status and upstream_error, under the request span (same request_id as the request failed line that follows). For example, a 400 from Anthropic carries Anthropic’s error.message. Without such a string there is no line, only the status in request failed. The rules (ADR 016):

• Only the vendor’s message string is logged (error.message, message, error_description, error, detail). A body without one, a non-JSON body, or a structured validation report is never logged, since it can quote the request.
• The message is cut where a validation error starts echoing the request ('input', "input", input_value, input=, as pydantic-based hosts such as vLLM do), so request content does not reach the log.
• Probable credentials are replaced by <redacted> (known key prefixes, long opaque tokens, and the value printed after a word such as key, token or password), then the message is cut to 512 characters.
• It never reaches the client. The client gets the usual error envelope: LM-2012 or LM-2013 when the body classifies as a context-length or content-policy refusal, LM-3003 otherwise.

To drop the line entirely, filter its target at the subscriber, for example RUST_LOG=info,lumen_providers::mapping=error.

Format

log_format is a top-level config key:

log_format = "json"   # "pretty" (default) or "json"

• pretty: human-readable, colored, for local development.
• json: one JSON object per line with event fields flattened, for production pipelines (Loki, CloudWatch, journald + a parser, …).

The format is boot-time only (logging initializes before anything else runs, right after config parses).

Level and filtering: RUST_LOG

Filtering uses the standard tracing env-filter syntax via the RUST_LOG environment variable. When RUST_LOG is unset, the gateway runs at info.

RUST_LOG=debug lumen --config config.toml            # everything, verbose
RUST_LOG=warn lumen --config config.toml             # quiet: warnings and errors
RUST_LOG="info,lumen=debug,lumen_server=debug" lumen ...   # gateway binary + HTTP layer
RUST_LOG="info,lumen_providers=debug" lumen ...      # debug for provider translation

Targets follow Rust module paths: the binary crate is lumen (its per-request events use the lumen::http and lumen::usage targets); the libraries are lumen_server, lumen_providers, lumen_router, lumen_auth, lumen_telemetry, lumen_core (ADR 001 naming).

Where debug is specifically useful: dropped OpenAI extras on translated providers. In lenient mode, a request field with no equivalent on the target provider (see chat completions) is dropped with a debug line naming the field. If a client swears it sent logprobs and nothing happened, this is where the evidence is.

Rejected metadata needs no debug: malformed x-lumen-metadata never fails a request; it is dropped with a warn line (visible at the default level) and a lumen_metadata_rejected_total increment - see the usage log.

Docker and systemd

Both capture stdout natively:

docker logs <container>                  # or a logging driver
journalctl -u lumen.service -f           # systemd unit from the deployment page

For production pipelines set log_format = "json" (the monitoring rig runs this way) and let the collector parse one object per line.

Usage log & multi-tenant metadata

When [auth] is enabled, every request writes a row to the usage_log table: token counts, cost, status, the model that actually served the request, and metadata - never message content. Prompts and responses are never logged, by default and in the usage log alike.

The route column

model is the id the client requested and model_used the foundation model that served. When the requested id is a virtual model, the nullable route column (migration 0010) also records the path the request took, for example acme/chat>acme/chat-eu>mistral-large: the same value as the x-lumen-route response header. It is NULL for a direct call to a foundation model and for rows written before the column existed.

Never on the request path

Usage-log writes never block a request. Each accounted call pushes an entry onto a bounded async channel (usage_channel_capacity); a separate batched writer task drains it and writes to the database in batches (usage_batch_max entries, or at least every usage_flush_ms). If the channel is ever full, an entry is dropped rather than backpressuring the request path, and lumen_usage_log_dropped_total increments so the drop is visible on /metrics.

Rows age out on their own: retention_days purges usage_log rows older than that many days.

Querying it over HTTP

GET /admin/usage (master-key gated, like every /admin/* route) returns aggregates over these rows - filtered by key, budget group (ADR 009), model, provider, capability and time window, grouped by the dimension you choose. Every row carries the key’s group_id at admission (refusal rows included), so per-pool reporting covers refused traffic too. Because rows arrive through the bounded channel above, requests from the last flush interval may not be visible yet. See Keys, quotas & budgets.

Exporting raw rows: GET /admin/usage/export

GET /admin/usage aggregates over one dimension at a time. A control plane building its own multi-dimensional view (per-tenant AND per-model AND per-day, say) needs the underlying rows instead of a fixed aggregate shape (ADR 010). GET /admin/usage/export returns them directly, cursor-paginated, master-key gated like every other /admin/* route.

Query parameters (all optional, unknown parameters are rejected with 400 LM-1001):

The response shape:

{
  "since": 1785600000,
  "until": 1785686400,
  "rows": [ { "id": 1042, "key_id": "...", "model": "...", "...": "..." } ],
  "next_cursor": 1043
}

since and until are the EFFECTIVE window for this call: either what the caller passed, or the resolved default (24 hours before until, until itself defaulting to “now”). Each row has the same columns as the usage_log table (id, key_id, group_id, model, model_used, route, provider, capability, token/media/cache counters, cost, latency, status, metadata, ts). As with every other usage surface, rows never carry prompt or response content, only accounting fields.

Pagination is by primary key, not offset, so an export cannot skip or repeat a row when new requests land mid-export: keep requesting with cursor set to the previous page’s next_cursor until next_cursor comes back null, which signals the window is exhausted. A full page is not by itself proof that more data exists; the exhausted signal is always a null cursor, even if that means one extra call returning zero rows at the very end.

Pin the window explicitly when paginating. With no explicit since/ until, both default relative to “now” and are recomputed independently on every call - so a multi-page export that never passes since/until is filtering each page against a window that keeps sliding forward while it pages through. Read since and until off the FIRST response and pass those same two values back on every subsequent page, instead of relying on the defaults again; this is exactly why the response echoes them.

limit is capped at 10000 per page; a request above the cap is rejected (400) rather than silently clamped, so a caller always knows the page it got back was the size it asked for.

curl -s -H "Authorization: Bearer $LUMEN_MASTER_KEY" \
  "http://localhost:8080/admin/usage/export?since=2026-08-01T00:00:00Z&limit=500"

x-lumen-metadata

Clients may attach a per-request metadata header, canonically x-lumen-metadata (alias cf-aig-metadata, for drop-in compatibility with Cloudflare AI Gateway clients). The value is a flat JSON object of string/number/bool values, bounded so log records and memory stay bounded: at most 16 keys, each key at most 64 bytes, each value at most 256 bytes, and the whole header at most 4 KiB.

The full (bounded) object is attached to structured logs and stored in the usage_log metadata column for later filtering. It is opaque - LUMEN never parses it for meaning or PII - and it is logged, so it must never carry secrets or prompt content.

Missing, malformed, oversized or wrong-typed metadata never fails the request: it is dropped with a log line and a lumen_metadata_rejected_total increment, and the call proceeds normally. See ADR 002.

Prometheus label allowlist

Only keys listed in telemetry.metadata_labels become Prometheus labels on the token/media counters; every other key stays logs-only:

[telemetry]
metadata_labels = []              # e.g. ["team", "env"]

The default is empty, which means client-supplied metadata can never mint a single new Prometheus time series - metric cardinality is a deliberate, operator-bounded decision, never a client-driven one. An allowlisted key absent from a given request gets the label value "". Keep the value sets you allowlist bounded: every distinct combination of allowlisted values is its own time series.

Multi-tenant recipe

Send org/team/project identifiers in the metadata header, allowlist the keys you want to slice by, and query per tenant on /metrics. With [auth] enabled, requests also need a virtual key:

curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $LUMEN_KEY" \
  -H 'content-type: application/json' \
  -H 'x-lumen-metadata: {"org_id":"acme","team_id":"rag","project_id":"docs-chat"}' \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hi"}]}'

[telemetry]
metadata_labels = ["org_id", "team_id", "project_id"]

Tokens per organization over the last 24 hours:

sum by (org_id) (increase(lumen_tokens_total{org_id!=""}[24h]))

The monitoring/ rig’s traffic.py script simulates exactly this pattern across several tenants and its dashboard has a dedicated multi-tenant panel row - see monitoring/README.md.

Disconnect accounting

A client that disconnects mid-stream is not a gateway failure and must not be recorded as a fake success. The request’s accounting settles at 499 (LM-6001) instead: usage_log.status and the lumen_request_duration_seconds{status="499"} sample both reflect the disconnect, kept out of both the internal-error class and the 5xx status class so a client hanging up never inflates internal-error alerts. See Error codes.

Keys, quotas & budgets

Auth is off by default: with [auth].enabled = false the gateway is an open proxy, with no database at all. Turn it on to get virtual keys, hard budgets and RPM/TPM quotas.

Enabling

[auth]
enabled = true
db_path = "lumen.db"   # SQLite file, created if missing

When enabled, the LUMEN_MASTER_KEY environment variable is required: 64 hex characters (32 bytes). It serves two roles - the bearer token of the /admin/* API, and the AES-256-GCM key that seals provider keys stored at rest. It is never logged and never stored.

What you get

With auth on, every /v1/* request is checked against a virtual key:

• Virtual keys are stored only as BLAKE3 hashes; the plaintext is returned exactly once, at creation, and never again.
• Hard budgets (budget_max, in USD) and RPM/TPM quotas (rpm_limit, tpm_limit) are enforced in memory, before any upstream call - a rejected request never spends and never reaches a provider.
• The database is never on the request path. Budget spend is flushed from memory to SQLite on flush_interval_ms (default 10000). While the process is running, enforcement itself lives in memory ahead of the flush, so budget overruns cannot occur. A crash loses at most that much accounting; after a restart, budgets reload from the last persisted state, so unflushed usage lost in a crash can permit spend beyond the intended budget until the gap closes.
• That database is the only copy of the key hashes, stored provider keys, budget state and usage ledger: see Backups for how not to lose it, and Scaling and high availability for why in-memory enforcement means one instance in v1.

Refusals

See Error codes for the full taxonomy.

Bootstrapping the first key (CLI)

On a fresh deployment there is no usable client key yet, and the admin API requires a running server. lumen keys closes that loop: it runs offline, straight against the SQLite file at auth.db_path - no server needed.

export LUMEN_MASTER_KEY=<64 hex chars>   # same gate as the /admin API
lumen keys create --config config.toml --name team-search \
  --budget-max 50 --rpm-limit 60 --tpm-limit 100000
lumen keys list --config config.toml

keys create prints the record plus the one-time plaintext key as a JSON object on stdout (the same shape as POST /admin/keys, shown below) and never logs it. --budget-max, --rpm-limit, --tpm-limit and --expires-at are optional, exactly like their JSON counterparts; keys list prints the records only (no hashes, no plaintext).

If the server is already running, keys list is safe, but a key created by the CLI only joins the live in-memory key table at the next restart or config reload - prefer POST /admin/keys against the running server in that case.

The admin API

Every route under /admin/* is mounted only when [auth].enabled = true, and every route is gated by the master key (Authorization: Bearer <LUMEN_MASTER_KEY>). Changes apply to the database and the in-memory state together, so they take effect immediately with no restart.

Budget groups

A budget group is a shared pool that several keys draw from (ADR 009). The pattern it exists for: prepaid credits per customer, one key per project of that customer. Without groups you would chunk the customer’s credit across the project keys and rebalance from outside - racy, and credit strands on idle keys. With groups: one group per customer, its keys join it, and a top-up is a single grant.

Create the pool first:

curl -s http://localhost:8080/admin/groups \
  -H "Authorization: Bearer $LUMEN_MASTER_KEY" \
  -H 'content-type: application/json' \
  -d '{"name": "acme-corp", "budget_max": 500.0}'

name is required; budget_max (USD) is optional - omit it for an unlimited pool that only attributes usage. A group has no plaintext and no hash, so unlike key creation there is no one-time secret: the 201 response is just the record, and GET /admin/groups lists the same shape.

{
  "id": "...",
  "name": "acme-corp",
  "budget_max": 500.0,
  "budget_spent": 0.0,
  "created_at": 1752537600,
  "deleted_at": null
}

A group carries budget only: no group-level RPM/TPM, no disabled flag, no expiry (each member key’s own limits still apply; group-level limits are backlog).

Membership. A key belongs to at most one group, via group_id on POST /admin/keys or PATCH /admin/keys/{id}:

curl -s http://localhost:8080/admin/keys \
  -H "Authorization: Bearer $LUMEN_MASTER_KEY" \
  -H 'content-type: application/json' \
  -d '{"name": "acme-project-search", "group_id": "<group id>", "budget_max": 100.0}'

On the patch, group_id is tri-state: leave the field out and membership is unchanged, send JSON null and the key leaves its group, send a string and it joins that group. This is the one deliberate exception to “patch fields cannot clear to NULL” - leaving a group must not require re-minting the key. A group_id naming an unknown or deleted group is refused with 400 LM-1001 before anything is written. Key records expose group_id wherever records appear. lumen keys create takes --group-id <ID> too, but the group must already exist: there is no offline lumen groups subcommand - create groups through the admin API, which needs only the master key.

Enforcement works exactly like a key budget, one level up:

• Admission checks both. The key’s own budget_max (when set) AND the group pool, in memory, before any upstream call. Either refusal is a 402 LM-4001; the message says which scope refused (“budget exceeded for this key” vs “budget exceeded for this key’s group”).
• Refunds mirror per-key budgets (ADR 007): a request refused at admission consumes nothing from either pool, and actual cost settles against both on completion.
• Flush and crash semantics are identical to key budgets: pool spend flushes to SQLite on flush_interval_ms, enforcement lives in memory ahead of the flush, and a crash loses at most one flush interval of pool accounting.

Pool spend is the group’s own accumulator, never recomputed from its members: spend a key accrued before joining (or after leaving) is not moved retroactively.

To top up a customer, raise the cap - pool spend is preserved, and the new cap binds every member key on its next request, no restart (name can be patched the same way; an unknown or deleted id returns 404 LM-1003):

curl -s -X PATCH http://localhost:8080/admin/groups/<id> \
  -H "Authorization: Bearer $LUMEN_MASTER_KEY" \
  -H 'content-type: application/json' \
  -d '{"budget_max": 1000.0}'

A PATCH sends an absolute cap, so top-ups that can race (an automated billing flow) should use grant instead - it adds atomically and never loses a concurrent update.

DELETE /admin/groups/{id} is a soft delete, like keys, and it is refused (400) while the group still has active member keys: move them out (PATCH with "group_id": null or another group) or delete them first. Silently dropping members out of pool enforcement would be worse than the error. The tombstone keeps usage_log attribution, and its final pool spend is flushed on removal.

Attribution. Every usage_log row - successes and 402/429 refusal rows alike - carries the key’s group id (when it has one), so per-customer reporting includes the traffic the pool refused. GET /admin/usage (below) takes a group_id filter and group_by=group_id; under that grouping, rows from ungrouped keys aggregate under an empty group name.

Lab-linked groups

A control plane such as the Meilisearch Lab marks the groups and keys it owns with two opaque fields (ADR 015):

• account_ref on a group (POST/PATCH /admin/groups): the control plane’s account id. A key in a group that has an account_ref is billable.
• external_ref on a key (POST/PATCH /admin/keys): the control plane’s key id.

Both are 1 to 128 characters when set, PATCH with null clears them, and both are returned by GET and filterable on the list routes (GET /admin/groups?account_ref=..., GET /admin/keys?external_ref=...). While [usage_events] is configured, account_ref must also be a UUID (400 LM-1001 otherwise).

With [usage_events] on, PATCH /admin/keys/{id} with group_id and PATCH /admin/groups/{id} with account_ref first flush pending spend, so it is billed to the account in force when it was spent. If that flush fails the call answers 500 LM-5001 and changes nothing: retry it. A database outage therefore blocks key moves and account_ref changes while billing is on. See Lab integration for the billing rule, event format and delivery.

Granting budget (top-ups)

POST /admin/groups/{id}/grant adds to a pool’s cap atomically. It exists for the prepaid-credits flow above: a customer buys credits, and the billing control plane grants their pool.

curl -s -X POST http://localhost:8080/admin/groups/<id>/grant \
  -H "Authorization: Bearer $LUMEN_MASTER_KEY" \
  -H 'content-type: application/json' \
  -d '{"amount": 25.0}'

The body is {"amount": <USD>}, the amount to add to budget_max; spend and quota windows are untouched. The 200 response is the updated group record, the same shape as GET /admin/groups. The key half is POST /admin/keys/{id}/grant: same body, same rules, and the 200 response is the updated key record (the record only - no plaintext, nothing was minted).

Why not a PATCH? A patch sends an absolute cap, so two racing top-ups can lose one: both workers read 500, both add 25, both write 525, and one customer payment vanishes. A grant is an atomic increment on both sides - budget_max = budget_max + ? inside SQLite, a fetch_add on the live in-memory entry - so concurrent grants all land. Like every admin change it takes effect on the very next request, no restart.

Three refusals:

• A non-positive, non-finite or oversized amount (400 LM-1001). Zero and negatives are rejected; overflowing literals like 1e999 never even parse; and a single grant is capped at 1e12 USD so repeated grants can never sum the stored cap toward infinity (which would read back as unlimited).
• Unknown or deleted id (404 LM-1003), like every other admin write.
• A capless target (budget_max null, 400 LM-1001): there is no cap to raise, and silently doing nothing would be worse than the error. Set a cap first (PATCH {"budget_max": ...}), then grant.

One retry caveat for billing automation: a grant is not idempotent. A call that timed out or disconnected may still have landed in the database, and blindly retrying would credit the cap twice - verify with GET /admin/groups (or /admin/keys) before retrying.

Create a key

curl -s http://localhost:8080/admin/keys \
  -H "Authorization: Bearer $LUMEN_MASTER_KEY" \
  -H 'content-type: application/json' \
  -d '{
    "name": "team-search",
    "budget_max": 50.0,
    "rpm_limit": 60,
    "tpm_limit": 100000
  }'

name is required; budget_max, rpm_limit, tpm_limit and expires_at (unix seconds) are all optional - omit any of them for “unlimited”. A blank name, a negative value, or a budget_max above 1e12 USD is refused with 400 LM-1001 naming the field (0 is accepted: a key that may spend or send nothing). Group budgets follow the same rule. Names are labels, not identifiers: two keys may share one, so look keys up by id. The response is the only place the plaintext key ever appears:

{
  "key": "sk-lumen-...",
  "id": "...",
  "name": "team-search",
  "group_id": null,
  "budget_max": 50.0,
  "budget_spent": 0.0,
  "rpm_limit": 60,
  "tpm_limit": 100000,
  "expires_at": null,
  "disabled": false,
  "created_at": 1752537600,
  "deleted_at": null
}

Store key now - it is never shown again. PATCH /admin/keys/{id} takes the same budget/quota fields (plus disabled) to adjust an existing key; fields left out of the patch are unchanged, the same value rules as on creation apply, and an unknown id returns 404 LM-1003.

Rotate a key (POST /admin/keys/{id}/rotate)

A lost or leaked key does not have to be replaced by a new one: rotation mints a fresh secret for the same key record.

curl -s -X POST http://localhost:8080/admin/keys/<id>/rotate \
  -H "Authorization: Bearer $LUMEN_MASTER_KEY"

The response has the exact same shape as creation - the new plaintext in key, shown exactly once, plus the record. Everything else is preserved: the id (so usage_log attribution is unbroken), the name, budgets, accrued spend and quotas. The swap is applied to the in-memory table too, so the old plaintext stops authenticating on the very next request and the new one works without a restart. An unknown or deleted id returns 404 LM-1003.

Delete a key (DELETE /admin/keys/{id})

curl -s -X DELETE http://localhost:8080/admin/keys/<id> \
  -H "Authorization: Bearer $LUMEN_MASTER_KEY"

Returns 204 No Content. Deletion is a soft delete by design: usage_log rows reference the key id, so removing the row would orphan the key’s usage history. Instead the row is tombstoned (deleted_at is set) and kept for attribution and audit. A deleted key:

• stops authenticating immediately (the in-memory table is updated, no restart needed), and never loads again at boot;
• disappears from GET /admin/keys; pass ?include_deleted=true to list tombstones (the audit view);
• rejects any further PATCH, DELETE or rotate with 404 LM-1003, like an unknown id - it cannot be resurrected by accident.

Retention of the tombstoned rows follows your usage-log retention policy: they are plain rows in the same database, with no plaintext and no hash that still authenticates. If you only want to pause a key, use PATCH {"disabled": true} instead - that one is reversible.

Store a provider key (PUT /admin/provider-keys/{name})

curl -s -X PUT http://localhost:8080/admin/provider-keys/openai \
  -H "Authorization: Bearer $LUMEN_MASTER_KEY" \
  -H 'content-type: application/json' \
  -d '{"key": "sk-..."}'

{name} is the provider’s configured name (as in [[providers]]). The body is {"key": "<provider api key>"}; a successful call returns 204 No Content. The key is sealed with AES-256-GCM under LUMEN_MASTER_KEY before it touches disk, for providers whose api_key_env is unset or empty. The call pings the hot-reload trigger after sealing the key, so the reloader re-reads provider keys from the encrypted store (off the request path) and rebuilds the provider registry - a rotated key takes effect without a restart. Environment-sourced keys keep precedence over a stored key. See Deployment - Hot reload.

{name} must be a provider the current config document defines; any other name is refused with 404 LM-1003 and nothing is stored (a provider added with PUT /admin/config/providers/{name} can take its key right after that call returns). DELETE /admin/config/providers/{name} also forgets the provider’s stored key, so a provider recreated later under the same name starts without one.

Check a provider key (POST /admin/providers/{name}/check)

Verifies a configured provider’s live credentials (the ones the routing table uses right now, so a key just rotated with PUT /admin/provider-keys/{name} is the one checked) without running a chat, embedding or rerank call. The gateway sends one request to a free, authenticated endpoint of that provider kind and never an inference route, so no tokens are spent.

curl -s -X POST http://localhost:8080/admin/providers/openai/check \
  -H "Authorization: Bearer $LUMEN_MASTER_KEY"

{
  "provider": "openai",
  "kind": "openai",
  "key_valid": true,
  "reachable": true,
  "http_status": 200,
  "latency_ms": 142,
  "endpoint": "GET https://api.openai.com/v1/models",
  "detail": "key accepted"
}

key_valid is a tri-state:

The verdict is data: the call answers 200 whatever the key’s state, so a rejected provider key is never mistaken for a rejected master key. An unknown {name} is 404 LM-1003. The response never contains the key, and endpoint drops any user:password@ a base_url carries. Each check is capped at 10 seconds.

What each kind probes:

A passing check proves the credentials authenticate, not that inference will succeed. These still only show up on a real request: an exhausted quota or credit balance, a model the key is not entitled to, a missing Bedrock model-access grant, a wrong Azure deployment name, and RPM limits. For Bedrock, an AccessDeniedException on the listing still reports key_valid: true (the credentials authenticated; the IAM policy just lacks bedrock:ListFoundationModels), with a detail saying so.

Usage & spend reporting (GET /admin/usage)

Aggregates the usage log per key, budget group, model, provider or capability - the HTTP query surface over the same rows the batched writer persists:

curl -s "http://localhost:8080/admin/usage?group_by=provider&since=2026-07-15T00:00:00Z" \
  -H "Authorization: Bearer $LUMEN_MASTER_KEY"

Query parameters (all optional):

The response echoes the effective window and grouping, then one aggregate per group - request counts split by status class, token totals, the estimated-vs-upstream split (ADR 003), rerank search units, media counts and cost:

{
  "since": 1784073600,
  "until": 1784160000,
  "group_by": "provider",
  "truncated": false,
  "groups": [
    {
      "group": "openai",
      "requests": 1204,
      "requests_ok": 1180,
      "requests_client_error": 20,
      "requests_server_error": 4,
      "tokens_in": 803211,
      "tokens_out": 121408,
      "tokens_total": 924619,
      "estimated_requests": 17,
      "upstream_requests": 1187,
      "search_units": 0,
      "media_count": 3,
      "media_bytes": 402133,
      "cost": 12.41
    }
  ]
}

Groups are ordered by cost (highest first) and capped at limit; when more groups matched, truncated is true and the returned groups are the most expensive ones. A window that matches nothing is a normal 200 with an empty groups array. Invalid filters, timestamps, group_by values or limits are 400 LM-1001.

Two accounting notes:

• Recent requests may lag. Usage rows travel through the bounded channel and its batched writer (see Usage log), so requests from the last flush interval (usage_flush_ms, default 2 s) may not appear yet - and entries dropped under a jammed channel (lumen_usage_log_dropped_total) never will.
• upstream_requests counts rows whose numbers are exact, which includes admission refusals (402/429): they consumed zero tokens, and zero is exact. estimated_requests counts rows whose token counts were locally estimated per ADR 003.
• Provider attribution on refusals. Rows served by a provider carry the provider that actually served them (under a fallback this may differ from the primary). Admission-refusal rows (402/429) never reached a provider; they carry the requested model’s primary provider, so per-provider reports still see the traffic that was headed there.

One encoding note: an RFC3339 +HH:MM offset contains a +, which in a query string means a space - percent-encode it as %2B (until=2026-07-15T03:00:00%2B02:00), or use Z/unix seconds.

Outbound webhooks for budget events

Everything above is pull: a control plane asks LUMEN what happened. Webhooks are the push half (ADR 011), and they exist for one problem in particular. A hard budget refuses with 402 LM-4001 the instant the pool empties, so a prepaid-credits backend that only polls will always learn about the exhaustion after the customer has been refused. A budget.threshold event at 80% is the trigger for an auto-recharge that lands as a POST /admin/keys/{id}/grant before that ever happens.

Absent by default. With no webhook configured, LUMEN makes no outbound call to anything but its providers, and does not even export a lumen_webhook_* metric. Enabling it is a deliberate choice, and it requires auth.enabled = true (every event describes a virtual key or a budget group).

There are two ways to configure one, and they compose: the admin API for a control plane, and the config file for a GitOps deployment.

Creating a webhook through the API

This is the path a billing backend wants: nothing to restart, nothing to edit on the host. Three calls, all gated by LUMEN_MASTER_KEY.

1. Store a signing secret. Generate one, keep your copy - LUMEN seals it and will never hand it back.

export WEBHOOK_SECRET="whsec_$(openssl rand -hex 24)"

# The secret reaches curl on stdin, never in an argument list (any local
# process can read those), and `jq` builds the JSON so a secret containing a
# quote or a backslash cannot corrupt the body. `printf` is a shell builtin,
# so it does not put the value in the process table either.
printf '%s' "$WEBHOOK_SECRET" | jq -Rs '{secret: .}' |
  curl -s -X PUT http://localhost:8080/admin/webhooks/signing-key \
    -H "Authorization: Bearer $LUMEN_MASTER_KEY" \
    -H 'content-type: application/json' \
    --data-binary @-

204. The secret is encrypted with AES-256-GCM under the master key before it touches the disk, exactly like a stored provider key.

2. Create the webhook.

curl -s -X PUT http://localhost:8080/admin/webhooks \
  -H "Authorization: Bearer $LUMEN_MASTER_KEY" \
  -H 'content-type: application/json' \
  -d '{
    "url": "https://backend.example.com/lumen/events",
    "events": ["budget.threshold", "budget.exhausted", "key.disabled"],
    "thresholds": [50, 80, 95],
    "channel_capacity": 1024,
    "timeout_ms": 5000,
    "max_attempts": 5,
    "retry_base_ms": 500
  }'

url is the only required field; every other one has the default shown above. Because this route decides where the gateway sends requests, the receiver must be publicly routable: a url whose host is a loopback, private, link-local (for example the 169.254.169.254 cloud metadata endpoint), unique-local or other reserved address, a localhost name, or a hostname that resolves to any such address (or does not resolve) is refused with 400 LM-1001. To deliver to an internal receiver, list its host in the LUMEN_WEBHOOK_ALLOWED_HOSTS environment variable (comma-separated; an entry matches exactly, or as a domain and its subdomains when written .suffix, for example LUMEN_WEBHOOK_ALLOWED_HOSTS=billing.internal,.corp.example), read once at boot. A listed host is accepted without resolving it. The check runs when the URL is set, not on every delivery, so it does not cover a hostname re-pointed afterwards or a receiver that answers with a redirect. The same check applies when PUT /admin/config/webhooks or a whole-document PUT /admin/config introduces or changes the [webhooks] receiver; a receiver already in the document (written by the operator on disk) is not re-checked on unrelated edits. Note that /admin/config* remains the most privileged surface: a provider’s base_url is not restricted the same way, so expose it only to callers you would trust with the host. The 200 response is the new state:

{
  "enabled": true,
  "source": "database",
  "settings": { "url": "https://backend.example.com/lumen/events", "...": "..." },
  "signed": true,
  "signing_key_stored": true,
  "updated_at": 1787691194
}

The new policy is in force from that moment: the very next request that crosses a threshold enqueues an event. Delivery itself stays asynchronous and best-effort, so a full queue or a receiver that stays down past the retry budget still drops it (see the guarantees below). The settings are stored, so a restart comes up identically.

3. Check it.

curl -s http://localhost:8080/admin/webhooks \
  -H "Authorization: Bearer $LUMEN_MASTER_KEY"

signed: true is the field to watch. If it is false, deliveries carry no x-lumen-signature and your receiver cannot tell them from anyone else’s traffic.

Editing is the same PUT: it replaces every setting, so send the whole document rather than a fragment (there is no PATCH - a partial update of a delivery policy is how you end up retrying against a URL you meant to change). Every field is editable at runtime, channel_capacity included: LUMEN rebuilds the queue and lets the previous sender drain what it already had.

Rotating the secret is another PUT /admin/webhooks/signing-key. It applies to the next delivery attempt, with nothing restarted. Retries of an event already in flight keep the signature they were created with, so a rotation never makes a pending retry unverifiable.

Deleting:

curl -s -X DELETE http://localhost:8080/admin/webhooks \
  -H "Authorization: Bearer $LUMEN_MASTER_KEY"

Emission stops immediately, and the decision is stored: a [webhooks] block in the config file will not quietly re-enable it on the next reload. DELETE /admin/webhooks/signing-key forgets the secret separately.

Declaring a webhook in the config file

For a deployment where the config file is the source of truth and no control plane calls the API:

[webhooks]
url = "https://backend.example.com/lumen/events"
signing_key_env = "LUMEN_WEBHOOK_SECRET"
events = ["budget.threshold", "budget.exhausted", "key.disabled"]
thresholds = [50, 80, 95]
channel_capacity = 1024
timeout_ms = 5000
max_attempts = 5
retry_base_ms = 500

signing_key_env names the environment variable holding the secret; the secret itself never appears in the file. A named-but-unset variable with no stored secret is a boot error: a billing integration silently downgraded to unsigned deliveries is worse than a refused start.

Which one wins. Settings written through PUT /admin/webhooks are stored in the database and take precedence over this block, so a runtime change is not undone by the next reload. GET /admin/webhooks reports source as "database" or "config" so the two are never ambiguous. If you manage the file, avoid the PUT (or expect the file to become decorative); if you manage the API, the block is just the boot-time default.

The events

Both budget events fire for keys and for budget groups; the payload’s scope says which, and subject_id is the id you would pass to the matching grant route.

Edge-triggering

A threshold fires once per budget epoch, not once per request past it. Cross 80% on Tuesday and every request for the rest of the week is silent. A grant that buys headroom re-arms the thresholds it drops below: top a $100 key that has spent $85 up to $300, and 85/300 = 28% re-arms 50% and 80% for the new epoch. A cap reduction does not un-fire what already fired.

budget.exhausted works the same way: the first refusal signals, the next thousand do not, and a grant re-arms it.

One consequence worth planning for: a restart re-arms from the last flushed spend, so an event can legitimately fire twice if the crash window (auth.flush_interval_ms, 10 s by default) swallowed the settle that first crossed it.

The payload

{
  "id": "evt_9f2c1b7ad04e4a1c8f3b6e2d5a90c714",
  "event": "budget.threshold",
  "scope": "key",
  "subject_id": "3b14b24efc6dc198000cdf5506ddea6d",
  "subject_name": "team-search",
  "budget_max": 100.0,
  "budget_spent": 82.0,
  "threshold": 80,
  "ts": 1787691194
}

Accounting facts only. Never a plaintext key, never client metadata, never prompt or response content - the same no-content construction as usage_log. budget_max is omitted for an uncapped subject, and threshold only appears on budget.threshold.

Verifying a delivery

Every POST carries four headers:

Verify against the raw bytes you received, before any JSON round-trip:

import hashlib, hmac

def verify(raw_body: bytes, header: str, secret: str) -> bool:
    expected = hmac.new(secret.encode(), raw_body, hashlib.sha256).hexdigest()
    return hmac.compare_digest(expected, header)

The secret comes from whichever source is configured: the environment variable named by signing_key_env wins when it is set and non-empty, and the secret stored through PUT /admin/webhooks/signing-key fills in otherwise. No route ever returns it, in any form; GET /admin/webhooks reports the booleans signed and signing_key_stored and the variable name, exactly as the provider surface reports key presence without key material.

Delivery guarantees, and what they are not

At-least-once while the process lives. A retryable failure (5xx, 429, 408, or a network error) backs off exponentially with jitter up to max_attempts, reusing the same x-lumen-event-id every time. Any other non-2xx is treated as permanent and not retried - the receiver has said this event will never be accepted. Nothing is persisted, so a restart forgets undelivered events.

Receivers must be idempotent on x-lumen-event-id. Retries, and the post-restart re-fire described above, both re-send the same id.

Not an accounting system. A full queue drops events rather than slow a request down, and a receiver that stays down past the retry budget loses them. GET /admin/usage/export remains the source of truth: reconcile against it on a schedule and treat webhooks purely as a latency optimisation over polling.

Never on the request path. Detection is a compare on the atomic budget settle that already happens per request; delivery is a non-blocking try_send into a bounded queue drained by a background task. A dead receiver cannot add a millisecond to a customer’s request.

Watching it work

These series appear the first time a webhook is enabled, not at boot, so a gateway that never uses them stays quiet on /metrics too.

Sustained dropped or dead means your billing loop is running blind: fall back to the export route until the receiver is healthy again.

What a reload can change

A config reload re-resolves the same precedence: a stored row still wins, so a reload never reverts a PUT. When the file block is what is in force, every field is re-applied, and removing the block stops emission.

How a field is applied depends on the field. url, events, thresholds, timeout_ms, max_attempts and retry_base_ms are retuned in place, on the queue and sender task already running - so a retarget cannot lose what is already queued. channel_capacity cannot be resized in place, so it replaces both: new events go to the new queue while the previous sender finishes delivering the events it had already accepted, then exits. Either way nothing already accepted is discarded.

The one thing a reload cannot do is see a new environment variable: a running process cannot observe a change to its own environment. Pointing signing_key_env at a variable that was not set when the gateway started needs a restart, or the sealed-secret route instead.

Operator notes

Per SECURITY.md, protect LUMEN_MASTER_KEY and the SQLite database file together: either one alone is not enough to read a stored provider key, but both together decrypt it. Treat them as a single secret.

Lab integration: account refs and billing usage events

The Meilisearch Lab is a control plane that manages accounts, keys and one credit ledger across several products. LUMEN integrates with it through two contracts only, never through Lab internals: the Lab drives the admin API to create and top up keys, and LUMEN pushes billing usage events back so the Lab can charge for what was spent. Nothing here sits on the request path, and a gateway without the [usage_events] block (and without the LAB_URL and LAB_INSTANCE_ID variables) behaves exactly as before. The wire shapes follow the Lab’s platform contract v2. The decision and its alternatives are in ADR 015 and its 2026-10-09 amendment.

The lease model

• One budget group per Lab account per gateway. The group is the account’s lease: its budget_max is the slice of the account’s shared credit wallet this gateway may spend. An account that uses several gateways has one group on each, and each gateway reports its own events.
• account_ref on the group holds the Lab account id. external_ref on a key holds the Lab key id. Both are opaque text, 1 to 128 characters when set. Set them on POST and PATCH; PATCH with null clears one. Both are returned by the GET routes and filterable on the list routes: GET /admin/groups?account_ref=... and GET /admin/keys?external_ref=....
• account_ref must be a UUID while [usage_events] is configured, because the Lab rejects any other account id. A violation is 400 LM-1001 naming the field. Without the block, any 1 to 128 character value is accepted.
• A key is billable when it has a group and that group has an account_ref. Every other key is an operator key: never billed, behaving exactly as today.

The Lab’s loop, using routes that already exist:

1. POST /admin/groups with account_ref the first time the account enables LUMEN on this gateway.
2. POST /admin/keys with group_id and external_ref per customer key.
3. Keep the lease in step with the account’s credits through GET /admin/groups/{id} and PATCH /admin/groups/{id} (or grant), see “The lease sync” below. Admission is unchanged: an exhausted lease answers 402 LM-4001, the correct answer for a customer who is out of credits. See Keys, quotas & budgets.
4. Scope every call it proxies for a user with X-Lumen-Account-Ref, see “One gateway, many accounts” below.

Configuration

The Lab mints one id and one secret per reporting deployment (platform contract v2, section 3.3). Three environment variables, the same names Scrapix and glutony read, are all a gateway needs:

LAB_URL=https://lab.meilisearch.com      # events go to {LAB_URL}/internal/events
LAB_INSTANCE_ID=<uuid>                   # this gateway's instance id
LAB_INSTANCE_SECRET=<64 hex>             # shown once by the Lab; never logged

LAB_URL and LAB_INSTANCE_ID overlay the [usage_events] block (and create it when the file has none); LAB_INSTANCE_SECRET is the default secret_env. The block spells the same thing out in TOML:

[usage_events]
url = "https://lab.meilisearch.com"      # LAB_URL
instance_id = "<uuid>"                   # LAB_INSTANCE_ID
secret_env = "LAB_INSTANCE_SECRET"       # env var holding the instance secret
source = "eu-1"                          # optional label for the boot log
batch_size = 500                         # 1..=500
timeout_ms = 5000                        # 100..=60000

• Opt-in and restart-only. The block is part of the boot layer (Config source modes): a reload that changes it is refused, like any boot-layer key.
• Requires [auth] enabled = true. Otherwise the gateway refuses to boot ([usage_events] requires auth.enabled = true). This includes a block created by the environment alone: a gateway with auth off that inherits LAB_URL from an env file shared with Scrapix or glutony refuses to boot, so keep those variables out of a standalone gateway’s environment.
• Both halves are needed. LAB_INSTANCE_ID without any url refuses to boot with an error naming LAB_URL, and a url without an instance id refuses to boot naming LAB_INSTANCE_ID.
• The secret must be set. The gateway refuses to boot when the variable named by secret_env (or signing_key_env) is unset, empty or only whitespace. The error names only the variable. The secret is never logged, never in an error and never in Debug output.
• Every live group’s account_ref must already be a UUID. The admin API only enforces the UUID shape while the block is on, so a group given some other ref before billing was enabled would produce events the Lab rejects. With the block configured, the gateway checks every live group at boot and refuses to start if one has a non-UUID account_ref; the error lists the offending group ids and nothing else. To fix it: boot without [usage_events] (and without LAB_URL / LAB_INSTANCE_ID), PATCH /admin/groups/{id} each listed group with its Lab account UUID (or null if it is not a Lab account), then restore the block and restart.
• With no block (and neither LAB_URL nor LAB_INSTANCE_ID set) there are no outbox rows, no sender task, no lumen_usage_events_* metric and no call to the Lab.
• Removing the block strands pending events. Rows still in the outbox stay there unsent while the block is absent (they go out once it is back). Drain first: wait for lumen_usage_events_pending to reach 0 (stop billable traffic if new events keep arriving), stop the gateway, confirm SELECT COUNT(*) FROM usage_outbox WHERE delivered_ms IS NULL AND dropped_ms IS NULL is 0 (the shutdown flush may have written a last event), then remove the block.

Instance identity

At boot, with the block configured and before the sender starts, the gateway calls GET {LAB_URL}/internal/instances/me once, with Authorization: Bearer <instance secret> and X-Lab-Instance-Id (contract v2, section 3.6), and logs the answer (instance_id, region, and the source label). Engines are hosted by Meilisearch only (decision A of the contract): the instance kind is hosted, and every event the gateway reports is billed by the Lab.

• Boot is refused when the Lab answers 401 or 403 (the credentials are wrong: check LAB_INSTANCE_ID and the secret), or answers 2xx with a body that is not this gateway’s identity: a malformed body, an instance_id other than the configured one (compared case-insensitively), a product other than lumen (the Lab would skip every event with product mismatch) or a kind other than hosted.
• Anything else only warns (any other status, for example a 404 from a Lab that does not serve the endpoint yet or a 5xx, a connect error, a timeout, or a 2xx whose body stalls or breaks mid-read): the gateway logs one warning and boots. Events are durable and the sender retries them.
• An unreachable Lab delays serving. The call is bounded by timeout_ms (default 5000 ms) and runs after the listener is bound but before the gateway starts serving: while a Lab does not answer, the socket accepts TCP connections but HTTP requests, /health included, wait up to that long. HTTP readiness probes need that margin; TCP probes pass at once.
• Billing attribution is unchanged from ADR 015: a key is billed when its group has an account_ref; every other key is an operator key.

One gateway, many accounts

A hosted gateway serves many Lab accounts through one master key, so the gateway enforces the account boundary itself (contract v2, section 8.3). The Lab sets X-Lumen-Account-Ref: <account uuid> on every /admin/* call it proxies for a user. The UUID is matched in its canonical lowercase form whatever case the caller sends: every write stores a UUID account_ref in lowercase, scoped or not (migration 0014 lowercased the ones stored before), while a ref that is not a UUID is kept exactly as written. With that header:

• Webhooks are platform-only rather than filtered. The budget webhook receiver is one gateway-wide setting that carries every account’s budget events, so no account may read or change it. A per-account receiver is a possible later change.
• On the account routes, a header that is not exactly one UUID (malformed, or sent twice) is 400 LM-1001, never read as “unscoped”. The master key is checked before the scope, so a missing or wrong key is still 401 LM-4004.
• Case does not matter for UUIDs. The header and every stored UUID account_ref are lowercase, so an uppercase header, an uppercase ref sent by an unscoped call or an uppercase ?account_ref= filter all match the same groups.
• Usage follows the group’s current account. The usage routes filter on the account_ref a group has now: if an operator moves a group to another account, its usage history moves with it.
• Without the header the master key keeps its unscoped, platform-operator behaviour, so an operator’s scripts and the Lab’s own provisioning calls are unchanged. The scope check and the write are separate store calls; only an unscoped master-key call running at the same moment can race them.

The lease sync

The Lab keeps each account’s lease in step with its credits (contract v2, section 8.2) through three routes, all accepting the scoping header:

An account at zero credits has a zero remaining lease and the gateway refuses its keys itself (402 LM-4001).

The billing rule

Billing rides on budget settlement, so it is exactly what the budget enforced.

• Settled cost, not spend. Each key carries a settled-cost counter, raised in settle by the real cost of a finished request. Reservation estimates held by in-flight requests are never billed.
• Watermark. Each key also carries a billing watermark: the part of its settled cost already turned into events, persisted in virtual_keys.billed_micro. At every budget flush, a billable key’s event is the settled cost since the watermark. The delta is never negative, and a flush with nothing new settled emits nothing.
• Atomic with the spend. The event row is written in the same SQLite transaction that persists the key’s spend and moves the watermark. After a failed commit nothing changes in memory and the next flush retries with a larger delta.
• Non-billable keys never bill their past. A key with no group, or in a group without account_ref, has its watermark kept equal to its settled cost. If it later joins a Lab-linked group, or its group gains an account_ref, only spend after that point is billed. Migration 0011 back-fills every existing key’s watermark from its current spend, so enabling [usage_events] on a running gateway bills only new spend.
• Money is integer micro-USD. Events carry provider_cost_micro_usd; the bill is never computed from the floating-point budget_spent column. The Lab prices it, with the unit counts, into credits.
• One flush, one lock. The periodic flush, the shutdown flush and the final flush of a deleted key share one function and one async lock, because two overlapping flushes would bill the same delta twice.
• Deleted keys are retired, not dropped. A deleted key stays in every flush until a flush that includes it has committed and no in-flight request still holds it. Its last spend, even from a request that is still streaming, is billed, and a failed final flush is retried.
• Crash window: one flush. A crash loses at most one flush interval of settled cost, from the bill and from the persisted budget alike (ADR 009 section 4). The interval is [auth] flush_interval_ms (default 10000 ms).

Changing billability flushes first, and fails closed

With [usage_events] configured, these two admin calls first flush pending spend, so it is billed to the account in force when it was spent:

• PATCH /admin/keys/{id} with group_id in the body (including null);
• PATCH /admin/groups/{id} with account_ref in the body (including null).

If that flush fails, the call answers 500 LM-5001 and changes nothing. Retry it. The consequence for operators: while billing is on, a database or outbox outage blocks key moves and account_ref changes. Serving requests and top-ups are unaffected.

Residual: spend from a request admitted before the change but settled after it follows the new membership. Clear a group’s account_ref only after the account is closed.

A PATCH /admin/keys/{id} that changes only external_ref does not flush first: it cannot change billability or the account. Spend settled before the change but not yet flushed is reported under the new api_key_id, on the same account, with no effect on the money billed.

The event

One usage.recorded event per billable key per flush that has new settled cost, in the Lab-owned v2 shape (contracts/vendor/lab/lab-events.schema.json):

{
  "id": "0192f3c1-7c2e-7b1a-9f00-3c9d2e4a5b61",
  "type": "usage.recorded",
  "occurred_at": "2026-09-30T15:04:05.123Z",
  "account_id": "<group.account_ref>",
  "api_key_id": "<key.external_ref or null>",
  "product": "lumen",
  "data": {
    "operation": "gateway",
    "units": { "requests": 412, "tokens_in": 90120, "tokens_out": 6000, "tokens_estimated": 310 },
    "provider_cost_micro_usd": 1834,
    "description": "key <lumen key id> 2026-09-30T15:03:55.120Z..2026-09-30T15:04:05.123Z"
  }
}

• id is a UUIDv7 minted once, when the outbox row is built just before the flush transaction writes it, so every retry carries the same id and the Lab deduplicates on it.
• operation is always gateway: one event aggregates every capability of the key over the window, because LUMEN keeps one watermark per key. (The Lab schema’s enum for lumen also lists chat, embed, rerank and systemone; that list predates the decisions rename of ADR 017, and LUMEN sends gateway only.)
• units are raw counts the Lab prices: settled requests, input and output tokens, and tokens_estimated, the tokens (in plus out) of the requests whose counts were local estimates (ADR 003). A crash can lose units; money is only ever computed from the watermark.
• provider_cost_micro_usd is the settled cost since the watermark, in integer micro-USD, passed through and marked up by the Lab. Never the floating-point budget_spent column, never a reservation.
• description is key <lumen key id> <window start>..<window end>. occurred_at and the window end are the flush time; the window start is the previous flush that billed this key (for its first event, the moment the key was loaded into memory at boot or created).
• The instance is not in the body: the Lab takes it from the authenticated X-Lab-Instance-Id header. The lease snapshot (group), source, key_id and window that ADR 015 carried in data are gone.
• The schema’s usageData forbids additional properties, so nothing else is sent. The schema is the Lab’s, vendored byte for byte and never edited here; the serializer is tested against the vendored copy. The CI job lab-contract-drift compares it with meilisearch/lab main and fails on drift when the LAB_REPO_TOKEN secret is set; without the secret it skips with a notice, and while the Lab has not published its copy yet (404) it passes with a warning.

Delivery

One sender task per process, only when [usage_events] is set.

• Every 2 seconds (and immediately again while a full batch was just acknowledged), it selects up to batch_size pending rows whose retry time has come, oldest first, and sends POST {url}/internal/events with body {"events":[...]}.
• Headers: Content-Type: application/json, X-Lab-Instance-Id: <LAB_INSTANCE_ID>, X-Lab-Timestamp: <unix seconds> and X-Lab-Signature: sha256=<hex>, where <hex> is the HMAC-SHA256, under the instance secret, of <X-Lab-Timestamp>.<raw body>. The timestamp is taken fresh for every attempt. The Lab rejects a timestamp more than 300 s from its clock: keep the gateway’s clock synced.
• A 2xx response carries {"accepted":["<id>", ...]}. Only ids listed in accepted are marked delivered. Ids not listed stay pending (see the drop rule below).
• Anything else (connect error, timeout, 401, any other non-2xx status, a body that is not the expected JSON) leaves every row of the batch pending. Each failed row waits min(2^attempts, 300) seconds before its next try, starting at 2 seconds and doubling, capped at 5 minutes, plus up to 1 second of jitter.
• A 401 is logged as an error (check LAB_INSTANCE_ID and the secret), counted, and retried forever. A wrong secret never takes the gateway down.
• Rows are dropped in exactly one case. An event a reachable Lab keeps out of accepted is permanently rejected after 24 h (contract v2, section 3.5). The clock starts at the first 2xx answer that leaves the event out (usage_outbox.first_skipped_ms, set once and never reset); the next such answer at least 24 h later drops it: the sender sets usage_outbox.dropped_ms, logs an error naming the event id and increments lumen_usage_events_dropped_total. The body stays in the table until the 7-day purge, so the amount can still be reconciled by hand. A Lab that is unreachable, times out or answers a non-2xx status never starts or resets the clock and never drops anything, however long it lasts: those rows wait. This deliberately differs from webhooks, which are signals: these events are the bill. Delivered and dropped rows are purged after 7 days, checked hourly.
• Redirects are not followed: the signed body reaches the configured endpoint only. A 3xx counts as a failed delivery with reason status.
• Shutdown: the final budget flush writes its events, then the sender gets one delivery attempt bounded to 5 seconds. Whatever is left is durable and goes out after the next boot.
• All SQLite access for delivery runs on the sender task, never on the request path.

Metrics and the alert

not_accepted means the Lab answered 2xx but left that id out of accepted, and counts only rows that will be retried (a dropped row counts in lumen_usage_events_dropped_total alone); store means the outbox itself could not be read or updated. A rising lumen_usage_events_dropped_total means the Lab refuses these events for good (an account this instance may not report for, an unknown account): read the error log for the ids.

The starter alert LumenUsageEventsStuck in monitoring/prometheus/alerts.yml fires when lumen_usage_events_oldest_pending_seconds > 900 for 5 minutes. Serving is unaffected while it fires, but every bill is delayed. Look at lumen_usage_events_failed_total by reason: auth is a secret mismatch, connect and timeout are reachability, status is a Lab-side error, and not_accepted is the Lab skipping events. A second starter alert, LumenUsageEventsDropped, fires on increase(lumen_usage_events_dropped_total[1h]) > 0: an event was dropped under the 24 h rule and was never billed; follow the runbook below to reconcile it.

Runbook: an event the Lab never accepts

If delivery works for new events but the alert stays on, one row is probably being refused (typically not_accepted keeps rising by the same small count), and it pins lumen_usage_events_oldest_pending_seconds. Left alone, it is dropped 24 h after the first refusal (see the drop rule in “Delivery”) and the alert clears; the steps below settle it sooner or reconcile it afterwards. The outbox is the usage_outbox table of the auth database ([auth] db_path). Inspect the oldest pending rows read-only:

sqlite3 -readonly lumen.db "SELECT id, attempts, created_ms, next_attempt_ms, first_skipped_ms, body \
  FROM usage_outbox WHERE delivered_ms IS NULL AND dropped_ms IS NULL ORDER BY created_ms, id LIMIT 5"

A non-null first_skipped_ms is when the Lab first answered without the id (unix ms): the row is dropped at the first refusal 24 h after it. List what was already dropped (and is kept until the 7-day purge) with:

sqlite3 -readonly lumen.db "SELECT id, first_skipped_ms, dropped_ms, body \
  FROM usage_outbox WHERE dropped_ms IS NOT NULL"

Look at body (the exact event sent) and at the Lab’s logs for that id to learn why it is refused. Then either:

• fix the cause on the Lab side (an account it does not know yet, an instance not allowed to report for it) within the 24 h: the row is retried unchanged and clears on its own; or
• settle it by hand when the Lab must never take it, for example because the body itself is wrong (a row’s body never changes once written): reconcile the amount with the Lab first (charge or waive it there), then mark that one row delivered, so it is purged after 7 days like any other:

sqlite3 lumen.db "UPDATE usage_outbox SET delivered_ms = CAST(strftime('%s','now') AS INTEGER) * 1000 \
  WHERE id = '<event id>' AND delivered_ms IS NULL AND dropped_ms IS NULL"

A dropped row was never billed: reconcile its amount with the Lab by hand before the 7-day purge removes it. Never delete a pending row: it is the only record of that bill.

Upgrading a build from main with pending v1 events

No tagged release shipped the first usage-event shape (v1). Builds from main since the first ADR 015 usage-events commit, after 0.6.1, may still hold pending outbox rows in that shape, and a v2 Lab cannot validate them. A v1 body has no data.operation. Those builds also predate the dropped_ms column (migration 0013 adds it when the new build boots), so set the rows aside in two steps.

1. Stop the old gateway, then inspect the v1 rows still pending and keep the output, it is the only record of that spend:

   sqlite3 -readonly lumen.db "SELECT id, attempts, created_ms, body FROM usage_outbox \
     WHERE delivered_ms IS NULL AND json_extract(body, '$.data.operation') IS NULL \
     ORDER BY created_ms, id"
   

2. Still before upgrading, park them so the new sender never selects them:

   sqlite3 lumen.db "UPDATE usage_outbox SET next_attempt_ms = 9223372036854775807 \
     WHERE delivered_ms IS NULL AND json_extract(body, '$.data.operation') IS NULL"
   

3. Upgrade and start the new build, then mark the parked rows dropped (now in milliseconds), so they leave lumen_usage_events_pending and are purged after 7 days like any dropped row:

   sqlite3 lumen.db "UPDATE usage_outbox SET dropped_ms = CAST(strftime('%s','now') AS INTEGER) * 1000 \
     WHERE delivered_ms IS NULL AND dropped_ms IS NULL AND json_extract(body, '$.data.operation') IS NULL"
   

The gateway never sends these rows, so the spend they carry (data.cost_micro_usd in each body) must be reconciled with the Lab by hand.

Sovereignty

Enabling this block makes the gateway call the configured URL. Events carry no prompt, no response, no request metadata and no model id: only the account and key refs, the LUMEN key id, unit counts, micro-USD cost and the window. The boot-time identity call sends the instance credentials (the id and the bearer secret) and nothing else. Per-model detail stays in usage_log and GET /admin/usage/export, which remain best-effort and are not the billing source. Without the block, this block adds no outbound call (webhooks, if configured, are separate).

What the Lab must provide

LUMEN carries no Lab code. The Lab side of platform contract v2 is:

1. Mint the instance id and secret (the hosted-engine rake task, bin/rails lab:hosted_engine:create PRODUCT=lumen ...) and serve GET /internal/instances/me.
2. Accept product: "lumen" with operation: "gateway" and the four unit names in its own contracts/lab-events.schema.json (the file LUMEN vendors), and price them in saas/config/pricing.yml (LUMEN’s money is provider_cost_micro_usd).
3. Verify X-Lab-Instance-Id, X-Lab-Timestamp and X-Lab-Signature on POST /internal/events and answer {"accepted": [...]}.
4. Validate each event of a batch on its own and leave an invalid one’s id out of accepted; never answer non-2xx for a whole batch because of one event. A non-2xx never starts the gateway’s 24 h drop clock, so one bad event would stall billing for every event behind it.
5. Set X-Lumen-Account-Ref (the canonical lowercase account UUID) on every proxied /admin/* call and never expose the platform-only routes to users.
6. Run SyncLumenLeasesJob against GET /admin/groups/{id} and PATCH /admin/groups/{id} (or grant), see “The lease sync”.

Config source modes: file and DB (ADR 012)

The dynamic half of LUMEN’s configuration - providers, routing, resilience, pricing, tokenizer, image fetch, the two hot-reloadable [auth] knobs, and [webhooks] - lives in exactly one place at a time: either the boot TOML file (today’s default) or a table in the auth SQLite database. The config_source boot key picks which. Both modes expose the identical admin API, so a client cannot tell which one it is talking to.

The two modes

config_source = "file"   # default
# config_source = "db"

"file" (default). The whole config - boot layer and dynamic layer together - is the one TOML file the process was started with. PUT /admin/config and every granular write stage a new copy, back up the previous one to <file>.bak, and rename atomically; an external edit (a human, GitOps) is picked up by the file watcher on the next reload. This is unchanged from before ADR 012 - every existing deployment needs no change.

"db". The dynamic layer lives in the config_versions table of the same SQLite database [auth] already uses. Requires auth.enabled = true; LUMEN refuses to boot otherwise (the admin plane is the only writer, and the database is the store). The boot TOML file may then contain only boot-layer keys - see below - since the DB is the dynamic layer’s sole source of truth and a second copy in the file would be exactly the kind of drift ADR 012 exists to prevent.

Boot layer vs. dynamic layer

A handful of settings are read once, at process start, and can never change without a restart - the boot layer. Everything else is the dynamic layer, owned by whichever ConfigSource is selected and editable through the admin API without a restart.

Every dynamic-layer setting takes effect on the next hot reload, which an admin write triggers itself - so everything the admin API accepts is live as soon as the call returns.

In file mode this split is invisible day to day: the one file holds both layers, exactly as before ADR 012. In db mode the boot file may hold only the left-hand column’s keys: [server], log_format, [telemetry], the boot-layer [auth] keys, [usage_events], and the top-level config_source key itself. Any dynamic-layer key in that file - a [[providers]] block, a [resilience] table, auth.flush_interval_ms or auth.retention_days - is a boot error naming the offending key, never a silently-ignored second source. The error message tells you the fix: remove the key from the boot file, or set config_source = "file".

The same split is enforced in the other direction on every admin write, and just as strictly: in db mode, PUT /admin/config and every granular write refuse a candidate that carries any boot-layer key at all - server.*, log_format, telemetry.*, usage_events, config_source, or a boot-layer [auth] key - independent of what value it names, even one identical to the field’s own built-in default. A dynamic document may never carry a boot-layer key; the only way to change one is to edit the boot file and restart. See If-Match and the boot-layer guard below for why an equal-to-default value cannot be waved through.

The read side enforces the same rule: in db mode, every load of the stored document (at boot, and on every reload) refuses one that carries a boot-layer key, whatever its origin - a hand edit of config_versions or a restored backup included. Boot fails naming the key; a reload keeps the running config and logs the rejection. The gateway can therefore never come up with auth silently disabled, pointed at another database file, or bound to a different port than the boot file says.

First boot in DB mode

A fresh config_source = "db" deployment with nothing ever written to config_versions boots into an empty-but-valid dynamic config: no providers, every dynamic section at its built-in default. GET /health is 200, /v1/* answers the normal LM-2001 (no such model) rather than refusing to start, and the log carries:

config_source = "db" and no config stored yet; PUT /admin/config to install one

This is what makes a fleet bootstrap a pure API flow: boot the process with just [server]/[auth]/config_source = "db", then PUT /admin/config with the real dynamic document once the process is reachable.

The admin API surface

Every route below is mounted only when [auth].enabled = true and gated by the master key, exactly like the rest of /admin/*. All of them work identically in both modes, and every mutating one requires If-Match.

Two admin surfaces are deliberately untouched by any of this: PUT /admin/provider-keys/{name} (encrypted secrets, see Keys, quotas & budgets) and /admin/webhooks* (ADR 011’s DB-row-wins precedence over a [webhooks] config block is unchanged, and applies to the document regardless of which ConfigSource backs it). GET /admin/config/webhooks reads the [webhooks] block from the document alone (null when absent) - a different thing from GET /admin/webhooks, which reports whichever source is actually live.

If-Match and the boot-layer guard

Every mutating call - the whole-document PUT and every granular one - requires an If-Match header carrying the hash from a prior GET:

curl -s http://localhost:8080/admin/config \
  -H "Authorization: Bearer $LUMEN_MASTER_KEY"
# {"config": "...", "hash": "b1946ac9...", "key_sources": {"openai": "env"}}

curl -s -X PUT http://localhost:8080/admin/config \
  -H "Authorization: Bearer $LUMEN_MASTER_KEY" \
  -H 'If-Match: "b1946ac9..."' \
  --data-binary @new-config.toml

• Missing If-Match is 400 LM-1001.
• Stale If-Match (the document changed since you read it - another operator’s apply landed first, or in file mode a human edited the file directly) is 412 LM-1004. Re-GET and re-apply.
• A candidate that changes a boot-layer key (server.*, log_format, telemetry.*, auth.enabled, auth.db_path, the usage-log channel knobs, usage_events, config_source itself) is 400 LM-1001, naming the changed key(s): "restart-only keys changed: server.port; edit the boot config file and restart". In file mode an unchanged boot-layer block still passes, since the candidate there is the whole file.
• A db-mode candidate that carries any boot-layer key at all is 400 LM-1001, unconditionally - naming the key: "unexpected key 'auth.enabled' in dynamic document: boot-layer keys are restart-only in config_source = \"db\" mode; set them in the boot config file instead". This is stricter than the diff above, and deliberately so: the stored document in db mode never carries boot keys, so every boot field on that side already resolves to its built-in default, and a candidate that sets one EXPLICITLY to that same default ([auth] enabled = false, [auth] db_path = "lumen.db", [server] port = 8080, …) would otherwise produce no diff against the current document and pass straight through. But a restart merges the stored dynamic document OVER the boot file (Config::load_with_dynamic), so that stored key would silently win at the next boot regardless of whether its value ever differed from the default - refusing to boot at all if it disables auth, pointing at the wrong database, or silently rebinding the listen port. A well-formed db-mode candidate simply carries no boot-layer keys at all; this check does not compare values, it refuses the key outright.
• A candidate that fails validation (bad TOML, an unknown field, a an invalid virtual model such as a cycle or a dangling target, …) is 400 LM-1001, naming the field, the virtual model or the dependent model.

Every write runs under one process-wide lock, so two concurrent applies can never interleave and defeat the If-Match check. The candidate is fully re-validated (parse, semantic checks, a throwaway provider-registry build) before anything is persisted - the same validation a restart would run - so a granular edit can never leave behind a document a restart would refuse to load. On success the hot-reload trigger fires (when one is armed), so the change is live without a restart.

Granular edits preserve the rest of the document

PUT /admin/config/providers/{name} and PUT /admin/config/{section} patch the current document with toml_edit rather than re-serializing it from a parsed struct, so in file mode every comment and formatting choice outside the table being touched survives untouched. In db mode the document is machine-owned and this is harmless either way.

Deleting a provider, a foundation model or a virtual model that a virtual model still targets is refused - 400 LM-1001, naming the dependent model - by the same validation pass every other write goes through. Fetching an unknown provider name, virtual model id or {section} is 404 LM-1003, the same style every other per-entity admin lookup in LUMEN uses.

The auth section: two knobs only

GET/PUT /admin/config/auth covers exactly the two dynamic [auth] knobs - flush_interval_ms and retention_days, which the background tasks read live - never enabled, db_path or the usage-log channel knobs (usage_channel_capacity, usage_batch_max, usage_flush_ms), which are boot-layer. A PUT body naming any of those is rejected 400 (deny_unknown_fields on the request type), not silently ignored. Unlike every other section, which the write replaces wholesale, auth merges field-by-field into the existing [auth] table, so the boot-layer keys survive untouched even though the request body never mentions them:

curl -s -X PUT http://localhost:8080/admin/config/auth \
  -H "Authorization: Bearer $LUMEN_MASTER_KEY" \
  -H 'If-Match: "b1946ac9..."' \
  -H 'content-type: application/json' \
  -d '{"flush_interval_ms": 5000, "retention_days": 30}'

DB-mode storage and retention

In db mode, every successful write inserts one immutable row into config_versions (never updates one in place); the current document is whichever row has the highest id. The newest 50 versions are retained - older rows are pruned automatically on each write. There is no admin endpoint to list or roll back to a previous version in this release (see Backlog); the rows exist and are queryable directly against the SQLite file if you need to recover one by hand.

Secrets never enter the document in either mode: api_key_env and signing_key_env are environment-variable names, never values, so a config_versions row is exactly as sensitive as the config file is today - no more, no less.

Migrating between modes

Switching modes is an operational procedure, not an API call: it always requires a restart, since config_source is itself boot-layer.

File to DB

1. GET /admin/config against the running file-mode process and save the response’s config field (the response is a JSON {"config", "hash"} envelope; the TOML document is the config string, not the raw body):

   curl -s http://localhost:8080/admin/config \
     -H "Authorization: Bearer $LUMEN_MASTER_KEY" | jq -r .config > saved.toml
   

   In file mode this is the WHOLE config file, byte for byte (boot layer and dynamic layer together, see the admin API table above) - it still needs step 4 below before it is a valid db-mode candidate.

2. Edit the boot TOML: set config_source = "db", and strip every dynamic-layer key ([[providers]], [[virtual_models]], [resilience], [tokenizer], [image_fetch], [webhooks], and auth.flush_interval_ms / auth.retention_days) - the boot file must contain only boot-layer keys once db mode is selected. Confirm auth.enabled = true.

3. Restart. The process boots with an empty stored document (first boot in db mode, see above) - /health is healthy, /v1/* answers LM-2001 until the next step.

4. Before the PUT, strip every boot-layer key from the document saved in step 1: [server], log_format, [telemetry], [usage_events], config_source, and the boot-layer [auth] keys (enabled, db_path, usage_channel_capacity, usage_batch_max, usage_flush_ms). Keep auth.flush_interval_ms and auth.retention_days if the file set them. The saved document is the file-mode GET’s WHOLE file, and the db-mode boot-layer guard (see If-Match and the boot-layer guard) now refuses a candidate carrying any boot-layer key unconditionally, even one whose value equals its built-in default - PUT-ing step 1’s document as-is is refused 400 LM-1001.

5. PUT /admin/config with the stripped document from step 4, using the empty document’s hash as If-Match (the GET /admin/config a fresh db-mode boot returns before any write).

DB to file

1. GET /admin/config against the running db-mode process and save the response’s config field (jq -r .config, as in the file-to-DB step 1) - the dynamic document alone, no boot keys.
2. Merge it with the boot-layer keys the process is currently running with ([server], log_format, [telemetry], [usage_events], and the boot-layer [auth] keys) into one TOML file.
3. Set config_source = "file" (or remove the key - "file" is the default) in that same file.
4. Point --config (or the existing config path) at the merged file and restart.

Either direction is a clean cutover: nothing reads from both sources at once, and the old source (the boot file’s dynamic keys in the file→db direction, or the config_versions table in the db→file direction) is simply no longer consulted after the restart - it is not deleted, so reverting is the same procedure run in the other direction.

See also

• ADR 012 - Config source abstraction for the design rationale.
• Keys, quotas & budgets for [auth] itself, and for the encrypted provider-key and webhook surfaces this document deliberately does not cover.
• Deployment - Hot reload for what a reload swaps and what stays restart-only outside the config-source boot key.
• Error codes for the full LM-1001/LM-1003/LM-1004 taxonomy.

Resilience tuning

LUMEN survives flaky upstreams without becoming flaky itself. A request goes through, in order: retries, then fallback to the next target of a virtual model, then the circuit breaker deciding whether to even try a given provider. None of this touches the database on the request path. Retries and the circuit breaker are always active with sane defaults; fallback exists only where you declare a virtual model, and background health checks are opt-in. All of it lives under [resilience] - see ADR 005 for the design.

Retries

Applied only to retryable upstream failures: 5xx, connect/read timeouts, and 429. A client 4xx is never retried - a fallback provider would reject it too. Backoff is exponential with equal jitter, and honors an upstream Retry-After header as a floor (a Retry-After: 3 guarantees at least a 3-second wait). While streaming, commitment happens at the first content frame, not at the open: after the upstream opens (2xx + headers), the gateway peeks the first frame before committing. An upstream that opens 200 then errors or closes before delivering any content still retries and falls over, charging the circuit breaker like an open failure, instead of surfacing a terminal SSE error frame - so an immediately-dead stream is still retried even though it opened. Once the first content frame reaches the client the request is committed and a later mid-stream error becomes a clean SSE error frame instead of a retry. See ADR 005 (first-frame-peek amendment) and Streaming.

[resilience]
retry_max_attempts = 3   # total attempts per provider incl. the first (>= 1)
retry_base_ms = 200      # base backoff wait after the first failure
retry_max_ms = 5000      # ceiling on the exponential backoff term

Set retry_max_attempts = 1 to disable retries entirely.

Fallback chains

Fallback is declared on a virtual model, not on the foundation model. A fallback virtual model lists targets to try in order, and fallback_on says which failures move on to the next one (default: a 5xx, a 429, a timeout or an open circuit; opt in to context_length and content_filter). Targets can be other virtual models, so chains compose. Chains are validated at boot and on every reload: every target must exist and serve the same capability, with no cycles, so a runtime resolution miss never happens in practice. Whichever foundation model actually served a request is reported in the x-lumen-model-used response header, and the path taken in x-lumen-route, so a caller (and the usage log) can see when a fallback fired.

[[virtual_models]]
id = "gpt-4o"
capability = "chat"
strategy = "fallback"
fallback_on = ["provider_error", "rate_limited", "timeout", "circuit_open"]
targets = [{ model = "openai/gpt-4o" }, { model = "claude-sonnet-4-5" }]

Circuit breaker

Tracked per (provider, model) pair:

circuit_failure_threshold = 5     # consecutive failures that trip it open
circuit_cooldown_ms = 30000       # time spent open before a half-open probe

After circuit_failure_threshold consecutive provider-fault failures, the circuit opens. While open, that link is skipped instantly - no upstream call - straight to the next fallback, or answered with 503 LM-3020 (plus Retry-After set to the cooldown remainder) if none remains. After circuit_cooldown_ms, a single half-open probe decides whether to close again. State is exported as lumen_circuit_state{provider,model} (0 closed, 1 open, 2 half-open).

The three timeouts

A provider’s own connect_timeout_ms override is picked up on hot reload like the other resilience knobs, since the registry rebuilds its clients from the new config.

[resilience]
connect_timeout_ms = 5000
total_timeout_ms = 600000   # 10 minutes

Health checks

Off by default. When health_check_enabled = true, a background task probes, on health_check_interval_ms, every provider that has an explicit base_url (self-hosted TEI/Ollama, or any explicit override) - providers on a built-in vendor URL are never probed and report unknown, since the gateway hardcodes no vendor endpoints. Results are published at GET /health/providers and the lumen_provider_up{provider} gauge. The list follows the live config: a provider added or removed by a hot reload (including one requested by PUT/DELETE /admin/config/providers/{name}) appears as unknown or disappears once that reload has run, and, when health checks were enabled at boot, a new provider with a base_url is probed from the next interval on (health_check_enabled and health_check_interval_ms themselves are read at boot: changing them takes a restart). This is independent of the gateway’s own liveness: GET /health never depends on provider state and does no I/O.

These probes never send provider credentials, so a revoked key still reads up. To check a key on demand, use POST /admin/providers/{name}/check.

health_check_enabled = false
health_check_interval_ms = 30000

How this shapes the error codes you see

Retries, fallback and the circuit breaker all happen before a 3xxx error ever reaches a client - by the time one surfaces, the resilience machinery has already given up. See Error codes: “How resilience shapes these codes” for the full mapping from a given upstream failure to the code you’ll see.

Deployment

Docker

docker run -p 8080:8080 \
  -v ./config.toml:/config.toml \
  -e OPENAI_API_KEY=sk-... \
  ghcr.io/qdequele/lumen:latest

The image is built from a multi-stage Dockerfile: a static musl binary copied onto a distroless/static non-root base - no shell, no libc in the final image, just the gateway. It is multi-arch (linux/amd64 and linux/arm64). The image sets LUMEN_SERVER__HOST=0.0.0.0 for you, so the server binds to all interfaces inside the container; mount your config at /config.toml (the image’s default CMD).

Bare binary

Static musl binaries for x86_64-unknown-linux-musl and aarch64-unknown-linux-musl are attached to every GitHub release cut from a v* tag - a single self-contained file, no runtime dependencies, which makes it systemd-friendly as a single process:

lumen --config /etc/lumen/config.toml

Bind the host/port via [server] in the config file, or the LUMEN_SERVER__HOST / LUMEN_SERVER__PORT environment variables.

systemd unit

A minimal hardened unit. The two numbers that matter: TimeoutStopSec must exceed the gateway’s 30 s drain window (see Shutdown and restarts), and ReadWritePaths must cover the auth database directory when auth is enabled (set an absolute auth.db_path; the default lumen.db is relative to the working directory).

[Unit]
Description=LUMEN gateway
After=network-online.target
Wants=network-online.target

[Service]
User=lumen
Group=lumen
WorkingDirectory=/var/lib/lumen
ExecStart=/usr/local/bin/lumen --config /etc/lumen/config.toml
# Provider API keys and LUMEN_MASTER_KEY, mode 0600, never in the config.
EnvironmentFile=/etc/lumen/env
# SIGHUP = config hot reload (see below). systemd's default stop signal is
# SIGTERM, which is the graceful-drain path.
ExecReload=/bin/kill -HUP $MAINPID
Restart=on-failure
# The gateway drains in-flight requests for up to 30 s on SIGTERM, then
# runs a final accounting flush with a bounded wait of up to 5 s; give the
# whole clean path headroom before systemd escalates to SIGKILL.
TimeoutStopSec=40
NoNewPrivileges=true
ProtectSystem=strict
ReadWritePaths=/var/lib/lumen

[Install]
WantedBy=multi-user.target

TLS and the reverse proxy

LUMEN intentionally does not terminate TLS. Put a reverse proxy (nginx, Caddy, your load balancer) in front of it, and leave HSTS to that proxy. The gateway speaks plain HTTP and should not be exposed directly to the internet without one.

Caddy needs two lines (automatic HTTPS, streams flush correctly by default):

gateway.example.com {
    reverse_proxy 127.0.0.1:8080
}

nginx needs response buffering off, or SSE streams arrive in bursts instead of token by token, and a read timeout longer than your slowest stream:

location / {
    proxy_pass http://127.0.0.1:8080;
    proxy_http_version 1.1;
    proxy_buffering off;
    proxy_read_timeout 300s;
}

Every response does carry a conservative set of default security headers:

• X-Content-Type-Options: nosniff
• X-Frame-Options: DENY
• Referrer-Policy: no-referrer
• Content-Security-Policy: default-src 'none'

Surface control

• Restrict /admin/* and /metrics at the network layer (firewall, reverse proxy, service mesh) as appropriate for your deployment. /admin/* requires the master key, but /metrics is unauthenticated by design - see SECURITY.md.
• GET /health is safe to point a liveness probe at: it never depends on provider state and does no I/O. It answers {"status": "ok", "version": "<gateway version>"}, unauthenticated, so a control plane can show which build it talks to.
• Webhook receivers set through PUT /admin/webhooks must be public. Internal hosts (loopback, private, link-local, localhost, or names that resolve to them) are refused unless listed in LUMEN_WEBHOOK_ALLOWED_HOSTS (comma-separated hosts, .suffix for a domain and its subdomains, read at boot). Set it when the receiver runs next to the gateway, for example LUMEN_WEBHOOK_ALLOWED_HOSTS=lab in a Compose network. See Outbound webhooks.

Scaling and high availability

The honest answer to “can I run two replicas behind the load balancer?” is: it depends on whether auth is enabled, and v1 does not paper over that.

With [auth].enabled = false (the default), the gateway is a stateless proxy: no database, no keys, no budgets. Run as many replicas as you like; nothing breaks. Two per-instance caveats remain: circuit breakers and health probes are per-process (each replica discovers a bad upstream on its own), and /metrics is per-instance (scrape every replica and aggregate in PromQL - the counters sum correctly).

With [auth].enabled = true, v1 is single-instance by design. Hard budgets and RPM/TPM quotas are enforced in per-process memory (that is what keeps the database off the request path), and spend is flushed to a per-node SQLite file. Budget-group pools (ADR 009) are enforced the same way, in the same per-instance memory, so the single-instance constraint applies to a shared pool identically. Behind a load balancer, N replicas each enforce the full budget and quota independently: a $100 hard budget becomes an effective $100 x N, an rpm_limit of 60 becomes 60 x N, and the usage ledger splits into N disjoint database files. Nothing crashes - the guarantees silently stop meaning what they say, which is worse.

Until then, the supported shapes with auth enabled are:

• One active instance. A supervisor (systemd, a single-replica Kubernetes Deployment with strategy: Recreate) restarts it; the drain semantics below bound the restart blip to seconds.
• Active/passive: a standby instance behind a failover VIP or LB health check, sharing nothing. On failover the standby starts from its own (empty or restored) database; budgets re-enforce from the last flushed state of whatever database it opens.

A shared Postgres backend for the auth/usage store and distributed rate limiting are the v2 items that lift this constraint - see the backlog.

Hot reload

A SIGHUP, a file-watch event (file mode), an admin config write (PUT /admin/config or a granular /admin/config/* write), or an admin provider-key rotation (PUT /admin/provider-keys/{name}) triggers a config reload: the new config is validated first, and only then are the provider registry, price table, resilience policy, the runtime-safe [auth] knobs (flush_interval_ms, retention_days), the webhook policy, the image-fetch policy and the tokenizer atomically swapped in. Every reload also re-reads DB-stored provider keys, so a key rotated via the admin API takes effect without a restart even without an explicit trigger call; a DB read error keeps the previous snapshot rather than stripping a working key. In-flight requests are unaffected. If the new config is invalid, it is rejected - the old config keeps serving, and lumen_config_reload_failures_total increments so the failed reload is visible in your dashboards.

The boot layer stays boot-time only and needs a real restart: [server] (bind address, body limit, stream timeouts), log_format, [telemetry] (its label allowlist becomes the Prometheus label set, fixed when the metrics are registered), auth.enabled, auth.db_path, the bounded usage-log channel knobs (usage_channel_capacity, usage_batch_max, usage_flush_ms), usage_events (the billing outbox sender is wired at startup; the LAB_URL, LAB_INSTANCE_ID and LAB_INSTANCE_SECRET variables feed it, see Lab integration) and config_source - rebinding a live listener, re-registering metrics or resizing a running channel is out of scope for a live swap. The admin config API refuses a change to any of them (400 LM-1001) rather than accepting a value that would not apply; edit the file and restart instead.

PUT /admin/config (ADR 010) applies a new config document remotely instead of an operator editing the file by hand. In file mode (the default) it stages the submitted bytes next to the real config file, validates the staged copy, then backs up the current file to .bak and renames the staged file into place before triggering the same reload path as above. This means the gateway process needs write permission on the config file’s directory, not just the file itself (the staged file and the .bak sibling are both new files created next to it, and the final apply is a rename within that directory). A read-only config mount (a common hardening choice, e.g. a Kubernetes ConfigMap volume or an immutable container layer) disables this route: the staging write fails and the request is rejected with an internal error (LM-5001), which is the correct refusal - the alternative would be a silent apply that never actually took effect. GET/reading the config still works read-only; only the PUT needs the extra permission. Relatedly, if the config path is a symlink (a pattern some ConfigMap-mount setups and manual atomic-deploy scripts use), an apply’s rename REPLACES the symlink itself with a regular file - the same rename that lands the new document in place cannot also preserve “the path is a symlink pointing elsewhere”; a setup that depends on the config path staying a symlink across reloads is not compatible with applying through this route. None of this applies in db mode (Config source modes): the document lives in the auth database, so the process needs no write access to the config file, and a read-only config mount is fine. See Applying a new config over the admin API for the full request contract, and its security note on what holding the master key implies once this route exists.

Shutdown and restarts

What each signal does:

The drain window is a built-in constant, not configurable, and a clean stop can spend up to 5 more seconds on the final accounting flush below. Tune your supervisor against the whole path: systemd TimeoutStopSec=40 (the unit above), Kubernetes terminationGracePeriodSeconds: 40. A supervisor that kills sooner turns graceful restarts into the crash case below; if draining ever exceeds 30 s, the gateway logs a warning and exits anyway rather than hanging.

Accounting across a stop, when auth is enabled:

• Clean shutdown attempts a final accounting flush with a bounded wait. After the listener drains, the gateway performs a final budget flush and waits up to 5 seconds for the usage-log writer to drain its channel. Usage rows still in the channel when the wait expires are lost, as are rows from database write failures (both logged as warnings rather than blocking exit). In the common case, a clean shutdown loses no accounting.
• A crash loses at most flush_interval_ms (default 10 s) of budget accounting and whatever usage-log rows were still in the bounded channel. Budget enforcement itself lives in memory ahead of the flush, so a running process never allows overruns. After a restart, however, budgets reload from the last persisted state: unflushed usage lost in a crash can permit spend beyond the intended budget until the gap closes.

Rolling restarts through the reverse proxy work as expected: mark the instance down (or just send SIGTERM), let the 30 s drain finish the in-flight streams, start the new binary. With auth enabled, avoid running old and new concurrently for long - see Scaling and high availability.

Backups

Everything durable lives in one SQLite file: auth.db_path (default lumen.db). It is the only copy of the virtual-key hashes, the encrypted provider keys stored via the admin API, the budget state, and the entire usage_log ledger. With auth disabled there is no database and nothing to back up.

• Live backup (server running): the database runs in WAL mode, so use SQLite’s online backup rather than copying the file:

  sqlite3 /var/lib/lumen/lumen.db ".backup '/backups/lumen-$(date +%F).db'"
  

  A live backup can trail reality by up to flush_interval_ms (default 10 s) of budget accounting - the in-memory spend not yet flushed.

• Cold backup (consistent snapshot): stop the gateway first (a clean SIGTERM attempts a final flush, see above), then copy lumen.db together with its -wal and -shm sidecar files if present. The backup reflects the state at shutdown, though in-flight or unflushed usage rows from the bounded channel may not be included.

• Restore needs the matching LUMEN_MASTER_KEY. Stored provider keys are encrypted under it; a restored database without the same master key serves virtual keys and history fine, but every stored provider key is undecryptable (re-enter them via the admin API). Per SECURITY.md, the key and the database are a pair: back up and protect them together, but never in the same place.

• Virtual keys are stored as BLAKE3 hashes and the plaintext is shown only once at creation: a lost database is unrecoverable key-wise. Clients keep their plaintext keys, but the gateway no longer knows them; they must be re-created. Back up on a schedule that matches how much usage_log history you are willing to lose.

Validate configs in the pipeline

Run lumen --check-config in CI or your deploy pipeline before a real boot. It performs the same parsing, semantic validation and provider registry construction the server does at startup, then exits 0 if the config is valid and non-zero otherwise - without binding a listener, opening a database, or contacting a provider. See Installation.

Upgrades

An upgrade is: replace the binary (or pull the new image), restart. This page covers the two things that make that boring in practice: the database migration story and what version numbers promise.

Before upgrading

1. Read the target release’s section in CHANGELOG.md. Release-specific upgrade notes live there (for example, 0.2.0 carries a one-line repair for databases created by 0.1.0, after a migration file’s checksum changed).
2. Back up the database if auth is enabled - see Backups. Upgrades run schema migrations automatically, and the backup is your downgrade path.
3. Validate your config against the new binary before booting it for real: lumen-new --check-config --config /etc/lumen/config.toml. A removed or renamed config key fails here, in the pipeline, instead of at restart time.

Upgrading to virtual models (config change)

The release that introduces virtual models removes per-model fallbacks, the [providers.models.rerank] block and the rerank capability on typesafe models. Run the converter before rolling out:

lumen config migrate -c /etc/lumen/config.toml --dry-run   # preview
lumen config migrate -c /etc/lumen/config.toml             # rewrite, keeps a .bak
lumen --check-config --config /etc/lumen/config.toml

Clients keep sending the same ids, with one exception: a migrated model that declared several capabilities keeps only the first one under its old id. Calls to that id for the other capabilities return LM-2002 until you add a virtual model for them (or point those clients at <provider>/<id>, which serves every capability without fallback). The migration prints a note for each such model, so read the --dry-run output and settle them before rolling out. usage_log.model_used and the Prometheus model labels of renamed foundation models change (for example gpt-4o becomes openai/gpt-4o), so update dashboards and queries that filter on them. Migration 0010 adds the nullable usage_log.route column. See Migrating from fallbacks.

Schema migrations run themselves

When auth is enabled, the gateway applies its embedded, numbered SQLite migrations automatically at boot (one per schema change, listed in crates/auth/migrations/). There is no separate migrate command and nothing to run by hand.

• Forward-only. There are no down-migrations. Rolling back to an older binary against a database a newer binary already migrated is not supported: the older binary refuses to start when it finds applied migrations it does not know. The downgrade path is the pre-upgrade backup.
• Integrity-checked. Each applied migration’s checksum is verified at boot; a mismatch is a hard, named error rather than a silent divergence.
• With [auth].enabled = false there is no database and this whole section is moot.

The restart itself

SIGTERM the old process (or let your supervisor do it), start the new one. In-flight requests get up to 30 seconds to finish and a clean shutdown attempts a final accounting flush with a bounded wait (up to 5 seconds) - the mechanics and the supervisor timeouts to pair with them are in Shutdown and restarts. With auth enabled, prefer stop-then-start over running old and new side by side; two live instances double-enforce budgets and quotas for as long as they overlap (see Scaling and high availability).

What version numbers promise

LUMEN is pre-1.0 and follows SemVer’s 0.x rules: a minor bump (0.2 -> 0.3) may contain breaking changes; a patch bump does not. Every breaking change is called out explicitly in the CHANGELOG. Per surface:

• HTTP API: the OpenAI-compatible (/v1/chat/completions, /v1/embeddings) and Cohere-compatible (/v1/rerank) surfaces track those upstream formats; gateway-specific behavior changes are CHANGELOG items.
• LM-xxxx error codes are stable identifiers. Codes are never renumbered or reused; new ones are added and documented in the error reference.
• Config format: keys can be added in any release; removals or renames are breaking changes (CHANGELOG + caught by --check-config, which rejects unknown keys).
• Metrics: renaming or re-labeling an exported series is a breaking change for your dashboards and is called out in the CHANGELOG.

Model release watcher

LUMEN connectors are per provider, not per model. A new model on a provider LUMEN already supports only needs a [[providers.models]] entry; a new provider, or a new API shape, needs code in crates/providers. The model release watcher tells maintainers which of the two each new release is, so neither goes unnoticed.

It is a scheduled GitHub Actions workflow (.github/workflows/model-watch.yml) running a standard-library Python script (scripts/model-watch/model_watch.py). It needs no secrets: every source is public, and it files issues with the workflow’s own GITHUB_TOKEN. It is a repository maintenance tool; it never runs inside the gateway.

What it does

Daily (06:17 UTC) and on manual dispatch, it:

1. Fetches three public catalogs: models.dev, LiteLLM’s model_prices_and_context_window.json, and OpenRouter’s /api/v1/models plus /api/v1/embeddings/models. LiteLLM’s openai_like/providers.json is read as a hint for which of its providers speak the OpenAI format.
2. Diffs them against the committed snapshot, scripts/model-watch/snapshot.json: per source, provider id to model id to capability (chat, embedding or rerank; image, audio and video models are ignored).
3. Classifies every addition against what LUMEN supports, read at run time from crates/providers/src/kind.rs (the ProviderKind variants, is_openai_compatible, default_base_url, supports_embeddings) and the capability tables in docs/providers.md:
   • (a) config only: a new model on a provider LUMEN supports. If the kind does not serve that capability yet (an embedding model on an embed-less kind, say), it is listed as a capability gap instead.
   • (b) OpenAI-compatible provider: a provider LUMEN does not know whose catalog entry declares an OpenAI-compatible base URL. Candidate for a new ProviderKind variant in the OpenAI-compatible family.
   • (c) needs a connector: a provider with a native API (or one the catalogs cannot vouch for), or any rerank model from an unsupported provider, since rerank has no OpenAI wire format.
4. Files issues, labelled per CONTRIBUTING.md:
   • one issue per (b)/(c) provider, titled Provider candidate: <name> [model-watch:provider:<id>], with the catalog ids, base URL and a next-steps checklist. Labels: enhancement, area: providers, priority: medium when it brings embedding or rerank models (otherwise priority: low), and the matching scope: labels. A later run comments on the open issue instead of opening another.
   • one rolling issue, New models on supported providers [model-watch:supported-models], for every (a) addition, grouped by LUMEN kind. Each run adds a dated comment.
5. Commits the refreshed snapshot to the branch it ran on, only after every issue was filed.

Providers that were already in the catalogs when the snapshot was first taken are the baseline: the watcher stays quiet about their new chat models (many are aggregators reselling the same models), and only speaks up when they add an embedding or rerank model.

Triage

• Close a provider issue once handled, or as wontfix to silence it. The watcher never reopens a closed provider issue.
• Close the rolling issue after the batch is triaged; the next run with new models opens a fresh one.
• Teach it a mapping: when a catalog spells a supported provider differently (together_ai for together), add the id to scripts/model-watch/providers.json. Ids are normalized (lowercase, separators removed) and a provider whose base URL equals a kind’s default_base_url is matched automatically. A capability only described in prose in docs/providers.md (Together and Cloudflare rerank) goes under extra_capabilities there.
• A new ProviderKind is picked up by the next run with no watcher change, as long as it is also added to the docs/providers.md tables (the run summary warns about a kind missing from them).

Running it locally

python3 scripts/model-watch/model_watch.py --dry-run                 # what would be filed today
python3 scripts/model-watch/model_watch.py --dry-run --show-bodies   # with full issue bodies
python3 -m unittest discover -s scripts/model-watch                  # unit tests

--dry-run never writes to GitHub or to the snapshot. With GITHUB_TOKEN and GITHUB_REPOSITORY set, it also reads existing watcher issues, so it reports comment and skip where a real run would. --source-dir DIR reads <source>.json files from DIR instead of fetching (offline replay). --init rebuilds the snapshot from today’s catalogs without filing anything, for example after a deliberate catalog id migration.

Failure handling

• A catalog changes shape or is unreachable: that source is skipped, its previous snapshot section is kept, the healthy sources are still processed, and the run goes red. Adapt the matching parse_* function.
• A catalog shrinks below half its previous size, or below a sane minimum: treated the same way, so an outage page is never recorded as a mass removal (which would re-announce everything on recovery).
• Too many new issues in one run (more than --max-new-issues, default 10; a dispatch input): nothing is filed and the snapshot is left as is. This is almost always a catalog renaming its ids. Inspect with --dry-run, then either add aliases or re-dispatch with a higher limit.
• Issue filing fails midway: the snapshot is not written, so the next run re-detects the same additions (an open issue may receive the same comment twice).
• Schedule paused: GitHub disables scheduled workflows after 60 days without repository activity. Re-enable it from the Actions tab; the committed snapshot makes the first run after a pause catch up in one batch.

Why the snapshot is committed

The snapshot is the watcher’s only state, and it is committed back to main rather than kept as a workflow artifact. Artifacts expire (90 days at most), and a lost snapshot means either refiling the whole catalog or silently skipping a window of releases. A committed file survives pauses, is reviewable with git log -p, and lets anyone reproduce a run locally with --dry-run. The cost is one small bot commit on days the catalogs change; main only blocks deletions and force-pushes, so the workflow’s token can push it, and commits made with that token do not trigger other workflows.

Future work

Not built yet: a later phase could hand type (c) issues to a Claude Code GitHub Action running the provider-integrator agent to draft a connector PR, always for human review. The watcher only detects and triages; it never writes provider code.

Examples

Runnable scenario configs live in examples/ at the root of the repository. Each directory is self-contained: a config.toml (the gateway config), a README.md (what it demonstrates and any prerequisites), and a run.sh (the requests to fire once the gateway is up).

Every scenario follows the same two-terminal recipe: start the gateway with the scenario’s config in one terminal, then fire its run.sh in another. Provider keys are never written into a config file; each config reads them from the environment variable named by its api_key_env field.

Every config.toml in examples/ passes lumen --check-config in CI.

minimal-chat

The smallest possible LUMEN config: one provider (OpenAI), one model (gpt-4o), chat only.

Demonstrates: non-streaming chat via POST /v1/chat/completions, and the same endpoint with "stream": true.

Env vars: OPENAI_API_KEY.

# terminal 1
export OPENAI_API_KEY=sk-...
cargo run -p server -- --config examples/minimal-chat/config.toml

# terminal 2
./examples/minimal-chat/run.sh

examples/minimal-chat on GitHub

self-hosted

A fully keyless config: no cloud provider, no API key anywhere. Chat and embeddings come from Ollama, reranking from TEI. Everything runs offline once the models are pulled.

Demonstrates: chat against Ollama’s OpenAI-compatible endpoint, embeddings against Ollama’s native endpoint, and reranking against a local TEI server.

Env vars: none. Requires Ollama running locally with llama3.2 and nomic-embed-text pulled, and (optionally) TEI serving BAAI/bge-reranker-large on port 8081.

# terminal 1
cargo run -p server -- --config examples/self-hosted/config.toml

# terminal 2
./examples/self-hosted/run.sh

examples/self-hosted on GitHub

multi-provider-fallback

Cross-vendor chat fallback: a gpt-4o virtual model serves from OpenAI (openai/gpt-4o) first, then from claude-sonnet-4-5 (Anthropic) when OpenAI fails.

Demonstrates: the x-lumen-model-used response header reporting whether the primary or the fallback served a request, and how the circuit breaker trips after repeated failures on the primary. See Resilience tuning.

Env vars: OPENAI_API_KEY, ANTHROPIC_API_KEY (the Anthropic key is only used if the fallback actually fires).

# terminal 1
export OPENAI_API_KEY=sk-...
export ANTHROPIC_API_KEY=sk-ant-...
cargo run -p server -- --config examples/multi-provider-fallback/config.toml

# terminal 2
./examples/multi-provider-fallback/run.sh

examples/multi-provider-fallback on GitHub

rag-pipeline

The two calls behind a typical RAG pipeline, wired to two different providers: embeddings via OpenAI (text-embedding-3-small) at index time, reranking via Cohere (rerank-english) at query time.

Demonstrates: POST /v1/embeddings embedding a small document corpus, then POST /v1/rerank re-scoring the same documents against a query with "top_n": 2. See Embeddings and Reranking.

Env vars: OPENAI_API_KEY, COHERE_API_KEY.

# terminal 1
export OPENAI_API_KEY=sk-...
export COHERE_API_KEY=...
cargo run -p server -- --config examples/rag-pipeline/config.toml

# terminal 2
./examples/rag-pipeline/run.sh

examples/rag-pipeline on GitHub

multi-tenant-analytics

Per-tenant cost and usage attribution: [auth] enabled with a virtual key per tenant carrying a hard budget, and [telemetry].metadata_labels turning the x-lumen-metadata header into Prometheus labels.

Demonstrates: creating a virtual key through the admin API (POST /admin/keys, master-key bearer), tagging requests with x-lumen-metadata, and slicing lumen_tokens_total on /metrics by org_id. See Usage log & multi-tenant metadata and Keys, quotas & budgets.

Env vars: OPENAI_API_KEY, LUMEN_MASTER_KEY (64 hex characters, e.g. openssl rand -hex 32).

# terminal 1
export OPENAI_API_KEY=sk-...
export LUMEN_MASTER_KEY=$(openssl rand -hex 32)
cargo run -p server -- --config examples/multi-tenant-analytics/config.toml

# terminal 2
export LUMEN_MASTER_KEY=...   # same value as terminal 1
./examples/multi-tenant-analytics/run.sh

--check-config on this scenario does not need LUMEN_MASTER_KEY set: the master key is a secret read from the environment at actual server startup, never part of the config file (and the config loader explicitly ignores it).

examples/multi-tenant-analytics on GitHub

HTTP API reference

The gateway’s HTTP contract is one OpenAPI 3.1 document, docs/openapi.yaml (its info.version is the gateway version): the OpenAI-compatible /v1 surface (chat completions, embeddings, rerank, decisions, models), the operational routes (/health, /health/providers, /metrics) and every master-key /admin route a control plane drives.

• A test in crates/server/src/app.rs fails the build when a mounted route is missing from the document or the document names a route that is not mounted, so the file always describes the running router.
• With [auth] enabled = true the gateway serves the same document as JSON at GET /openapi.json (master key), which is how the Meilisearch Lab vendors it. Like the other platform routes it answers 403 LM-4005 to a call scoped with X-Lumen-Account-Ref (see Lab integration).
• Every error is the {"error": {"code": "LM-xxxx", ...}} envelope of Error codes.

Render it with any OpenAPI viewer, for example npx @redocly/cli@1 build-docs docs/openapi.yaml (writes a static redoc-static.html).

Providers

LUMEN ships twenty-seven built-in provider kinds - sixteen native integrations (their own request/response translation, including deployment-routed azure, SigV4-signed bedrock and decisions-only typesafe) plus eleven OpenAI-compatible hosts that reuse the OpenAI path with a per-kind base URL. Each [[providers]] block in your config selects one with a kind string and gives it a unique name (your own label). Each [[providers.models]] block under it exposes a model to clients:

[[providers]]
name = "my-openai"        # your label; must be unique
kind = "openai"           # selects the built-in implementation
api_key_env = "OPENAI_API_KEY"   # NAME of the env var holding the key
# base_url = "https://…"  # optional override (required for self-hosted kinds)

[[providers.models]]
id = "gpt-4o"             # the id clients send (owned entirely by you)
upstream_id = "gpt-4o-2024-08-06"   # what LUMEN sends upstream (defaults to `id`)
capabilities = ["chat"]   # any of "chat", "embed", "rerank", "decisions"
release_date = "2024-08-06"   # optional; lets clients sort GET /v1/models by release

Rules that apply to every provider:

• API keys are never in the config. api_key_env names an environment variable; LUMEN reads it only when a request actually routes to that provider. A hosted provider whose env var is unset fails only at use, not at boot - a partial set of keys is fine.
• release_date is metadata only. An ISO date ("2024-08-06", or the unquoted TOML date 2024-08-06), validated at load. GET /v1/models returns it as release_date and as the OpenAI integer created (Unix seconds, midnight UTC); both are omitted for an undated model. It never affects routing.
• Model ids are globally unique across all providers. A collision aborts startup and names both offending providers. Several ids may map to one upstream_id (versioned aliasing).
• capabilities must match the kind. A model can only declare capabilities its provider kind implements (table below); this is validated at boot.
• base_url is an optional override for hosted kinds, and required for the self-hosted kinds (tei, ollama), which are keyless.
• Batching: an embed request with more inputs than the provider’s batch limit is split into sub-batches, run with bounded concurrency, and reassembled in the original order. The limits below are built in.
• Multimodal embeddings (M9): declare modalities = ["text", "image"] on a model to accept image content parts on /v1/embeddings. input items may be strings or arrays of parts ({"type":"text",...} / {"type":"image_url",...}). Images are passed as data: URIs, or - with [image_fetch] enabled - as remote http(s) URLs the gateway fetches under SSRF/resource guards and inlines. Image input to a model without "image" is rejected with LM-2003; a remote URL with fetching disabled is LM-2005. Cohere (embed-v4) and Voyage embed a combined text+image vector per item; Jina embeds one modality per item (a mixed item is sent as its image, its caption text is not combined). See the multimodal-embeddings design spec for the full guard list.

The typesafe kind serves decisions only; reranking a decision model goes through a virtual model remap (see Decision models as rerankers). perplexity and cloudflare (OpenAI-compatible table below) also serve decisions. The full decisions matrix is below. The together kind (in the OpenAI-compatible table below) additionally serves rerank (LlamaRank) natively; see its section for the model config.

OpenAI-compatible hosts (chat + embed via the OpenAI path). The Embed column reflects what each host actually serves upstream: groq, deepseek, perplexity and xai expose no /embeddings endpoint (perplexity and cloudflare do serve decisions, see Decisions by provider kind), so a model declaring embed on those kinds is rejected at config load (it could only ever 404 at request time) - unless the provider sets a custom base_url, which is taken to mean an operator-run proxy that may serve embeddings:

Self-hosted or catalog-dependent kinds (vllm, huggingface, cloudflare) stay permissive: the operator controls what their endpoint serves. If one of the embed-less hosts above later ships an embeddings API, point a kind = "openai" provider at it with a base_url override, or file an issue to update the capability table.

All embed-serving OpenAI-compatible kinds use a 2048-input embed batch limit. Anything that speaks the OpenAI wire format but isn’t listed can still be used via kind = "openai" with a base_url override.

cloudflare additionally serves rerank (not shown in the table above, which covers only the chat/embed OpenAI-compatible path): its BAAI bge-reranker-* models are served through Workers AI’s native /ai/run/{model} endpoint rather than an OpenAI-compatible one. See ### cloudflare below.

openai

• kind: openai · capabilities: chat, embed, decisions
• Auth: api_key_env (e.g. OPENAI_API_KEY), sent as a bearer token.
• base_url: optional; defaults to OpenAI’s public API. Set it to point at any OpenAI-compatible endpoint.
• Embed batch limit: 2048 inputs per upstream call.
• Decisions: gpt-6-luna on POST {base_url}/decisions (public beta). A choice needs at least 2 options and a request at most 128 images. Set cost_per_1m_input = 0.10 and modalities = ["text", "image"]. See Decisions.

[[providers]]
name = "openai"
kind = "openai"
api_key_env = "OPENAI_API_KEY"

[[providers.models]]
id = "gpt-4o"
upstream_id = "gpt-4o-2024-08-06"
capabilities = ["chat"]

[[providers.models]]
id = "text-embedding-3-small"
capabilities = ["embed"]

[[providers.models]]
id = "gpt-6-luna"
capabilities = ["decisions"]
modalities = ["text", "image"]
cost_per_1m_input = 0.10

mistral

• kind: mistral · capabilities: chat, embed (OpenAI-compatible).
• Auth: api_key_env (e.g. MISTRAL_API_KEY), bearer token.
• base_url: optional override.
• Embed batch limit: 512.

[[providers]]
name = "mistral"
kind = "mistral"
api_key_env = "MISTRAL_API_KEY"

[[providers.models]]
id = "mistral-small"
upstream_id = "mistral-small-latest"
capabilities = ["chat"]

anthropic

• kind: anthropic · capabilities: chat only.
• Auth: api_key_env (e.g. ANTHROPIC_API_KEY). LUMEN authenticates with the x-api-key / anthropic-version headers, not a bearer token.
• Translation: OpenAI ⇄ Anthropic is bidirectional, including tools and streaming events, so clients keep using the OpenAI wire format. parallel_tool_calls: false maps to tool_choice.disable_parallel_tool_use; response_format, seed, logprobs, frequency_penalty and presence_penalty have no Messages API equivalent - see chat extras and the strict flag.
• base_url: optional override.

[[providers]]
name = "anthropic"
kind = "anthropic"
api_key_env = "ANTHROPIC_API_KEY"

[[providers.models]]
id = "claude-sonnet-4-5"
upstream_id = "claude-sonnet-4-5-20250929"
capabilities = ["chat"]

google

• kind: google · capabilities: chat, embed (Gemini).
• Auth: api_key_env (e.g. GEMINI_API_KEY). The key rides the x-goog-api-key header, never the URL.
• Translation: OpenAI ⇄ Gemini, including streaming (streamGenerateContent). response_format JSON mode maps to generationConfig.responseMimeType / responseSchema, seed to generationConfig.seed, and frequency_penalty / presence_penalty to generationConfig.frequencyPenalty / presencePenalty; logprobs and parallel_tool_calls have no mapping - see chat extras.
• Embeddings: served through models/{model}:batchEmbedContents (gemini-embedding-001, text-embedding-004, …). One inner request per input; the OpenAI dimensions field maps to outputDimensionality. The API is text-only: pre-tokenized token-id arrays and image content parts are rejected with LM-1001 before any upstream call. Usage follows ADR 003: usageMetadata.promptTokenCount is reported when the upstream returns it, otherwise the gateway derives a local estimate marked estimated.
• Embed batch limit: 100 inputs per upstream call (Gemini’s documented batchEmbedContents ceiling); the gateway splits larger requests.
• base_url: optional override.

[[providers]]
name = "google"
kind = "google"
api_key_env = "GEMINI_API_KEY"

[[providers.models]]
id = "gemini-2.0-flash"
upstream_id = "gemini-2.0-flash"
capabilities = ["chat"]

[[providers.models]]
id = "gemini-embedding-001"
capabilities = ["embed"]

vertex_ai

• kind: vertex_ai · capabilities: chat, embed (Gemini and text-embedding-* models on Google Cloud Vertex AI). Distinct from google, which is the public Gemini Developer API: Vertex uses regional endpoints and GCP OAuth instead of a static API key.
• Auth: api_key_env names an env var holding the full service-account key JSON (the contents of the key file downloaded from GCP, not a path and not an API key). LUMEN signs an RS256 JWT assertion with the account’s private key, exchanges it at the account’s token_uri for a short-lived OAuth2 access token (scope cloud-platform), and sends it as a Bearer header. Tokens are cached in memory and refreshed 60 s before expiry, so the exchange stays off the per-request hot path. The private key is redacted from all Debug output and never appears in logs or errors.
• base_url: required - it carries the GCP region (e.g. us-central1), not a URL. The endpoint is derived from it: https://{region}-aiplatform.googleapis.com/v1/projects/{project}/locations/{region}/publishers/google/models/{model}:generateContent (and :streamGenerateContent?alt=sse when streaming).
• Project id: taken from the service-account JSON’s project_id.
• Translation: chat is identical to google (same GenerateContent wire schema), including streaming. Like Gemini, only inline base64 image data is accepted; remote image URLs are rejected with LM-2004.
• Embeddings: Vertex does NOT expose batchEmbedContents; embeddings go through the prediction API on the same regional, project-scoped path: .../publishers/google/models/{model}:predict with instances[].content. The OpenAI dimensions field maps to parameters.outputDimensionality. Text-only: token-id arrays and image parts are rejected with LM-1001. Usage follows ADR 003: per-input statistics.token_count values are summed and reported as upstream usage; when absent the gateway derives a local estimate marked estimated.
• Embed batch limit: 1 input per upstream call. gemini-embedding-001 accepts a single instance per :predict request (other text-embedding-* models take more, but the limit is per-model, so the universally safe value is used); the gateway fans larger requests out over concurrent calls.

[[providers]]
name = "vertex"
kind = "vertex_ai"
# The env var holds the service-account key file's JSON contents:
#   export VERTEX_SA_JSON="$(cat service-account.json)"
api_key_env = "VERTEX_SA_JSON"
base_url = "us-central1"   # GCP region

[[providers.models]]
id = "gemini-flash-vertex"
upstream_id = "gemini-2.0-flash"
capabilities = ["chat"]

[[providers.models]]
id = "gemini-embedding-vertex"
upstream_id = "gemini-embedding-001"
capabilities = ["embed"]

bedrock

• kind: bedrock · capabilities: chat, embed.
• Auth: AWS Signature Version 4 (SigV4), not a bearer key. Credentials are read from the standard AWS environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and optionally AWS_SESSION_TOKEN for temporary credentials) on every request, so values updated in the process environment (or a config hot reload) take effect without a restart. api_key_env is optional and, if set, overrides only the secret access key. The secret and session token are never logged or shown in Debug.
• Credential scope (v1): only static keys and pre-issued STS session tokens. There is no AWS credential-provider chain (no IMDS/instance roles, SSO, profiles or credential_process); an expired session token keeps failing with 403 until the environment supplies a fresh one.
• API (chat): the Bedrock Converse API (POST /model/{modelId}/converse and /converse-stream), which gives one uniform schema across the Anthropic, Meta Llama, Amazon Titan/Nova, Mistral and Cohere model families. The legacy per-model InvokeModel chat schemas are intentionally not implemented (Converse covers the same chat models).
• API (embeddings): Bedrock has no Converse equivalent for embeddings, so the embed path uses per-model InvokeModel (POST /model/{modelId}/invoke), routed by model id (issue #95):
  • Amazon Titan (amazon.titan-embed-text-v2:0 and predecessors): embeds ONE text per call ({ "inputText": ... }); the gateway loops one signed request per input and reassembles the batch in order. Titan v2 honors dimensions (mapped to Titan’s dimensions/normalize); older Titan models do not, so dimensions is dropped (or rejected under strict).
  • Cohere Embed on Bedrock (cohere.embed-english-v3, cohere.embed-multilingual-v3): embeds a batch in one call ({ "texts": [...], "input_type": ... }); input_type defaults to search_document and honors an override. Cohere does not accept dimensions.
  • Batch limit: reported as 1 (the conservative floor across the two families, whose real per-call limits are 1 for Titan and 96 for Cohere), so the router splits every batch into single-input sub-batches it runs concurrently. Pre-tokenized (token-id) and image inputs are rejected with an honest 400 before any upstream call (both families are text-only).
  • Usage (ADR 003): Titan’s inputTextTokenCount and Cohere’s x-amzn-bedrock-input-token-count response header are reported as upstream usage; when absent the request edge derives the local estimate (never a silent zero).
• Region / base_url: set base_url to the runtime endpoint for your region, https://bedrock-runtime.{region}.amazonaws.com; the region is parsed back out of it for the SigV4 signing scope. VPC/PrivateLink endpoint hosts (bedrock-runtime.{region}.vpce.amazonaws.com, including a vpce-…-prefixed DNS name) are recognised too. For any other custom endpoint the region comes from AWS_REGION / AWS_DEFAULT_REGION; if no source yields a region, startup fails with a clear error rather than silently signing for a wrong region.
• Translation: OpenAI ⇄ Converse is bidirectional, including system prompts, inferenceConfig (max tokens, temperature, top-p, stop sequences), tools, and streaming. Streaming arrives as AWS event-stream binary frames, decoded and translated to OpenAI chunks. Usage (inputTokens / outputTokens) is mapped per ADR 003. Converse has no equivalent for response_format, seed, logprobs, parallel_tool_calls, frequency_penalty or presence_penalty
  • see chat extras and the strict flag.
• Images: only inline data: URIs are supported (Converse takes raw image bytes); a remote image URL is rejected (LM-2004) since Bedrock cannot fetch one.

[[providers]]
name = "bedrock"
kind = "bedrock"
base_url = "https://bedrock-runtime.us-east-1.amazonaws.com"
# api_key_env = "AWS_SECRET_ACCESS_KEY"   # optional secret override

[[providers.models]]
id = "bedrock-claude-sonnet-4-5"
upstream_id = "us.anthropic.claude-sonnet-4-5-20250929-v1:0"
capabilities = ["chat"]
modalities = ["text", "image"]

[[providers.models]]
id = "bedrock-titan-embed"
upstream_id = "amazon.titan-embed-text-v2:0"
capabilities = ["embed"]

[[providers.models]]
id = "bedrock-cohere-embed"
upstream_id = "cohere.embed-english-v3"
capabilities = ["embed"]

cohere

• kind: cohere · capabilities: chat, embed, rerank. A single model can serve any combination.
• Auth: api_key_env (e.g. COHERE_API_KEY), bearer token.
• Chat: Command R / R+ via POST /v2/chat, including streaming. The wire shape is OpenAI-adjacent (roles live directly in messages, no top-level system hoist like Anthropic; tool_calls are already OpenAI-shaped), so translation is closer to identity than Anthropic’s. tool_choice collapses to Cohere’s REQUIRED/NONE (forcing one specific named tool has no v2 equivalent and falls back to auto). response_format, seed, frequency_penalty and presence_penalty map onto Cohere’s native fields; logprobs and parallel_tool_calls do not - see chat extras. Usage prefers usage.tokens (actual counts) over usage.billed_units (what’s charged); a response reporting neither leaves the gateway’s local estimator to fill in an honestly-flagged count (ADR 003).
• Vision (issue #73): a user message carrying image parts is translated to Cohere v2 content blocks (text / image_url, OpenAI-shaped); a text-only message keeps the plain-string form, and non-user roles always flatten to text (Cohere only admits image content on user messages). Declare modalities = ["text", "image"] on a vision model (Command-A-Vision) to opt in. Both inline data: URIs and remote http(s) URLs are forwarded (Cohere fetches remote URLs itself, so LM-2004 does not apply); the optional detail hint (low/high/auto) passes through untouched. Provider-native references (anthropic-file:, gs://, Gemini Files API URIs) are rejected pre-flight with LM-2008.
• Embed batch limit: 96.
• Cost: rerank is billed in search units (cost_per_1k_searches).
• input_type override: Cohere’s embed v2 API requires an input_type and the gateway cannot know query-vs-document intent, so it defaults to search_document (the indexing case). Set input_type as an extra field on the /v1/embeddings request body to override it per request, e.g. {"model": "embed-multilingual", "input": "...", "input_type": "search_query"}. Allowed values: search_document, search_query, classification, clustering. An unrecognized value is rejected with LM-1001 before any upstream call. The field is consumed at the gateway: only the Cohere translation reads it, and it is never forwarded in the outgoing body of any other provider (a strict OpenAI-compatible upstream such as vLLM could reject unknown fields).

[[providers]]
name = "cohere"
kind = "cohere"
api_key_env = "COHERE_API_KEY"

[[providers.models]]
id = "command-r-plus"
upstream_id = "command-r-plus-08-2024"
capabilities = ["chat"]

[[providers.models]]
id = "rerank-english"
upstream_id = "rerank-v3.5"
capabilities = ["rerank"]
cost_per_1k_searches = 2.0

[[providers.models]]
id = "embed-multilingual"
upstream_id = "embed-v4.0"
capabilities = ["embed", "rerank"]

jina

• kind: jina · capabilities: embed, rerank (hosted).
• Auth: api_key_env (e.g. JINA_API_KEY), bearer token.
• Embed batch limit: 2048.

[[providers]]
name = "jina"
kind = "jina"
api_key_env = "JINA_API_KEY"

[[providers.models]]
id = "jina-rerank"
upstream_id = "jina-reranker-v2-base-multilingual"
capabilities = ["rerank"]

voyage

• kind: voyage · capabilities: embed, rerank (hosted).
• Auth: api_key_env (e.g. VOYAGE_API_KEY), bearer token.
• Embed batch limit: 128.

[[providers]]
name = "voyage"
kind = "voyage"
api_key_env = "VOYAGE_API_KEY"

[[providers.models]]
id = "voyage-rerank"
upstream_id = "rerank-2"
capabilities = ["rerank"]

mixedbread

• kind: mixedbread · capabilities: rerank (hosted, mxbai-rerank-*).
• Auth: api_key_env (e.g. MXBAI_API_KEY), bearer token.
• base_url: optional; defaults to https://api.mixedbread.com/v1.
• Schema note: Mixedbread’s endpoint is POST /v1/reranking (note the path: reranking, not rerank) and renames the request fields (input instead of documents, top_k instead of top_n) with results nested under data; the gateway translates transparently.
• Usage: billed in tokens, so the gateway reports an estimated token count (ADR 003) rather than upstream search units.

[[providers]]
name = "mixedbread"
kind = "mixedbread"
api_key_env = "MXBAI_API_KEY"

[[providers.models]]
id = "mxbai-rerank"
upstream_id = "mixedbread-ai/mxbai-rerank-large-v1"
capabilities = ["rerank"]

pinecone

• kind: pinecone · capabilities: rerank (hosted inference).
• Auth: api_key_env (e.g. PINECONE_API_KEY), sent as the Api-Key header (not a bearer token), alongside a pinned X-Pinecone-API-Version header the inference API requires.
• base_url: optional; defaults to https://api.pinecone.io.
• Schema note: documents are sent as { "text": ... } objects; only the default text rank field is used (rank_fields selection is out of scope for v1).
• Usage: Pinecone reports usage.rerank_units, carried through verbatim as the response’s search_units (not estimated).

[[providers]]
name = "pinecone"
kind = "pinecone"
api_key_env = "PINECONE_API_KEY"

[[providers.models]]
id = "pinecone-rerank"
upstream_id = "pinecone-rerank-v0"
capabilities = ["rerank"]

typesafe

• kind: typesafe · capabilities: decisions (TypeSafe’s Jev typed-decision models, and any other TypeSafe-format vendor, served on POST /v1/decisions; see Decisions and ADR 017). Reranking goes through a virtual model remap (Decision models as rerankers).
• Auth: api_key_env (e.g. TYPESAFE_API_KEY), sent as a bearer token. Never logged, redacted from Debug.
• base_url: optional; defaults to https://api.typesafe.ai. An override is the API root. Without decisions_path a trailing /v1 is tolerated and the gateway posts to {base}/v1/systemone.
• decisions_path (optional, typesafe only): the upstream path appended to base_url verbatim. Must start with /. Default /v1/systemone. This and the next field make typesafe the generic TypeSafe-format kind.
• forward_unknown_fields (optional, typesafe only, default true): forward unknown top-level request fields upstream, as TypeSafe accepts them. Set false for a vendor that rejects them (Perplexity answers 400). Setting either field on another kind is refused at config load.
• Schema note: the request and response are TypeSafe’s own, forwarded verbatim for a TypeSafe-format client (only model is rewritten to the upstream_id); state, question and answer JSON keeps its key order byte-for-byte. The question contract is validated at the edge (LM-2011 / LM-1001) before any upstream call. An OpenAI-format client is translated to this wire format. A noul question without instructions is not sent to a typesafe model (Jev requires them): the target is skipped.
• Models: jev-latest (stable alias), jev-preview (preview alias) and versioned ids such as jev-1.13.0, mapped through upstream_id.
• Errors: TypeSafe 401 / 422 surface as LM-3003 (502, not retried, upstream body not forwarded); 429 is LM-3001 honouring Retry-After; 529 Overloaded is a retryable 5xx, so retries apply, and a virtual model’s provider_error fallback trigger matches it (not rate_limited, which is an upstream 429).
• Cost: Jev bills input tokens only, $0.042 per 1M (TypeSafe pricing, 2026-09): set cost_per_1m_input = 0.042 and no output price.
• Usage: upstream usage.input_tokens / output_tokens when reported; otherwise an estimated input count over state plus every question (ADR 003), output 0.
• Upstream limits (TypeSafe, 2026-09): 64k tokens of context per request (32k for state plus the longest question), 250k tokens/s and 1200 RPM, dynamic. Not enforced by the gateway.

[[providers]]
name = "typesafe"
kind = "typesafe"
api_key_env = "TYPESAFE_API_KEY"

[[providers.models]]
id = "jev"
upstream_id = "jev-latest"
capabilities = ["decisions"]
cost_per_1m_input = 0.042

# A pinned version for reproducible answers.
[[providers.models]]
id = "typesafe/jev-1.13.0"
upstream_id = "jev-1.13.0"
capabilities = ["decisions"]
cost_per_1m_input = 0.042

# Clients call `jev-1.13.0`; it falls back to the stable alias when the pinned
# version is overloaded (529) or its circuit is open.
[[virtual_models]]
id = "jev-1.13.0"
capability = "decisions"
strategy = "fallback"
targets = [{ model = "typesafe/jev-1.13.0" }, { model = "jev" }]

# Jev as a reranker: plain /v1/rerank, one predicate question per document. A
# `typesafe` model cannot declare `rerank` itself; a virtual model's `remap`
# does the conversion (see docs/virtual-models.md).
[[virtual_models]]
id = "jev-rerank"
capability = "rerank"
strategy = "single"

[[virtual_models.targets]]
model = "jev"
remap = { strategy = "predicate" }

Other TypeSafe-format vendors

Each recipe below is a typesafe provider with its own base_url and decisions_path. The TypeSafe format is shared, the path is not. LUMEN has not been tested against these vendors’ live APIs: check each vendor’s documentation for the model id and whether it tolerates unknown fields.

# Liquid d1. Unverified: set forward_unknown_fields = true if Liquid
# accepts unknown fields.
[[providers]]
name = "liquid"
kind = "typesafe"
base_url = "https://api.liquid.ai"
decisions_path = "/decisions/v1/systemone"
forward_unknown_fields = false
api_key_env = "LIQUID_API_KEY"
[[providers.models]]
id = "liquid-d1"
upstream_id = "d1"
capabilities = ["decisions"]

# Inception Mercury Decide.
[[providers]]
name = "inception"
kind = "typesafe"
base_url = "https://api.inceptionlabs.ai"
decisions_path = "/v1/decisions"
api_key_env = "INCEPTION_API_KEY"
[[providers.models]]
id = "mercury-decide"
upstream_id = "mercury-decide"
capabilities = ["decisions"]

# Self-hosted Kev (kev.serve) on its default path /v1/systemone.
[[providers]]
name = "kev"
kind = "typesafe"
base_url = "http://localhost:8000"
[[providers.models]]
id = "kev"
upstream_id = "kev"
capabilities = ["decisions"]

Upstage Solar Decide serves /v1/systemone, but its base URL is unconfirmed: set base_url to the host Upstage documents for your account and keep the default path. The upstream_id values above for Inception and Kev (mercury-decide, kev) are placeholders: use the model id your vendor documents. For Inception, Kev and Upstage the forward_unknown_fields default (true) is untested.

Decisions by provider kind

Five kinds serve decisions. All use bearer auth except ollama (keyless; a bearer key is sent if one is configured). Limits are checked before any upstream call; a target that cannot take the request is skipped (see Decisions).

Unknown top-level request fields are forwarded by typesafe (unless forward_unknown_fields = false) and stripped by the other kinds; openai sends only safety_identifier. There is no question-count cap at the gateway edge: the per-target limits above apply when a target is chosen. Prices are the published ones as of 2026-10 and only feed cost_per_1m_input: set it on each model, the gateway never fills prices in. A model accepts images only if it declares modalities = ["text", "image"] (the default is ["text"]).

perplexity and cloudflare dispatch per model, so one provider block can hold chat models and decision models side by side. Status mapping is the usual one (429 is LM-3001 honouring Retry-After, 5xx is retryable LM-3003, upstream bodies never reach the client); see Error codes. An Ollama server that is not running is LM-3004 and falls back like any unavailable upstream.

nvidia (NIM)

• kind: nvidia · capabilities: rerank (NVIDIA NIM ranking).
• Auth: keyless by default (self-hosted NIMs run without a key); supply api_key_env (e.g. NVIDIA_API_KEY) for the hosted API, sent as a bearer token.
• base_url: required - the NIM root (e.g. http://localhost:8000 or the NVIDIA-hosted ranking endpoint root). The gateway posts to {base}/v1/ranking.
• Schema note: the request nests query: { text } and passages: [{ text }]; there is no top_n on the wire, so the gateway requests the full ranking and truncates to top_n afterwards (as for TEI).
• Score semantics: NIM returns a raw logit, passed through unchanged as relevance_score. Scores are unbounded (can be negative) and are only comparable within a single response; higher is more relevant. No sigmoid is applied.
• Usage: NIM reports no token usage, so the gateway reports an estimated token count (ADR 003).

[[providers]]
name = "nvidia-nim"
kind = "nvidia"
base_url = "http://localhost:8000"
# api_key_env = "NVIDIA_API_KEY"   # only for the hosted API

[[providers.models]]
id = "nvidia-rerank"
upstream_id = "nvidia/llama-3.2-nv-rerankqa-1b-v2"
capabilities = ["rerank"]

together (rerank)

The together kind (see the OpenAI-compatible section for chat/embed) also serves rerank (LlamaRank) natively through Together’s Cohere-shaped /rerank endpoint. One [[providers]] entry with kind = "together" serves all three capabilities against the same base_url and bearer key. Rerank is billed in tokens, so the gateway reports an estimated token count (ADR 003).

[[providers]]
name = "together"
kind = "together"
api_key_env = "TOGETHER_API_KEY"

[[providers.models]]
id = "llama-rank"
upstream_id = "Salesforce/Llama-Rank-V1"
capabilities = ["rerank"]

tei (self-hosted)

• kind: tei · capabilities: embed, rerank.
• Auth: keyless.
• base_url: required - points at your Text Embeddings Inference server. TEI serves one model per process, so upstream_id is ignored by the upstream but kept for your own clarity.
• Embed batch limit: 32.

[[providers]]
name = "tei-local"
kind = "tei"
base_url = "http://localhost:8081"

[[providers.models]]
id = "bge-reranker"
upstream_id = "BAAI/bge-reranker-large"
capabilities = ["rerank"]

ollama (self-hosted)

• kind: ollama · capabilities: chat, embed, decisions (Ollama 0.35+, on POST {base_url}/v1/systemone; choice and score take 2 to 26 options or levels, and only clef and clef-flash accept images; see Local decision models with Ollama).
• Auth: keyless.
• base_url: required - points at your Ollama server root (no /v1). Embeddings use Ollama’s native POST /api/embed; chat goes through Ollama’s OpenAI-compatible endpoint, which lives under /v1 on the same root - the gateway appends the /v1 itself, so keep base_url as the bare server root either way.
• Chat: served by the shared OpenAI-compatible path - streaming (SSE passthrough), cancellation, and token accounting (upstream usage when Ollama reports it, otherwise a local count marked estimated, ADR 003) all work exactly as for the openai kind.
• api_key asymmetry: if you set an api_key_env on this kind (e.g. an Ollama server behind an authenticated reverse proxy), the chat path sends it as a bearer token but the native embed path currently sends no Authorization header at all - keep the embed route unauthenticated at the proxy, or front only /v1 with auth.
• Embed batch limit: 512.
• Tip: a local model may take a while to load into VRAM on its first call - relax first_token_timeout_ms / total_timeout_ms on the provider block (see config.example.toml). A self-hosted box on a slow link can also override connect_timeout_ms; note that doing so gives this provider its own (unpooled) HTTP client (ADR 005, 2026-07-15 amendment), whereas the first-token and total overrides do not. All three fall back to their global defaults when unset.

[[providers]]
name = "ollama-local"
kind = "ollama"
base_url = "http://localhost:11434"
first_token_timeout_ms = 60000
total_timeout_ms = 120000
connect_timeout_ms = 10000  # optional: own client, relaxed connect deadline

[[providers.models]]
id = "local-llama"
upstream_id = "llama3.2"
capabilities = ["chat"]

[[providers.models]]
id = "nomic-embed"
upstream_id = "nomic-embed-text"
capabilities = ["embed"]

[[providers.models]]
id = "nimble"
upstream_id = "nimble"
capabilities = ["decisions"]

azure

• kind: azure · capabilities: chat, embed. Reuses the OpenAI JSON schema verbatim; only the URL, auth, and routing differ from openai.
• Auth: api_key_env, sent as the api-key header (never a bearer token).
• base_url: required - your Azure resource endpoint, e.g. https://<resource>.openai.azure.com (no shared public default, every resource is operator-specific).
• api_version: optional - pins the Azure API version sent as the api-version query parameter on every request (issue #65). For back-compat the older form still works: append ?api-version=YYYY-MM-DD to base_url. Precedence: the explicit api_version field wins over a base_url query string, which wins over LUMEN’s pinned built-in default (see the azure module doc comment for the exact value). Any query parameters on base_url other than api-version are ignored when building request URLs. api_version is azure-only: setting it on any other kind is rejected at boot.
• Deployment routing: Azure routes by URL path (/openai/deployments/{deployment}/...), not by the model field in the body. Set each model’s upstream_id to the Azure deployment name - the same upstream_id mechanism every other kind uses for aliasing already carries it through.
• Embed batch limit: 2048 (same array-size ceiling as the OpenAI embedding models Azure hosts).

[[providers]]
name = "azure-openai"
kind = "azure"
api_key_env = "AZURE_OPENAI_API_KEY"
base_url = "https://my-resource.openai.azure.com"
api_version = "2024-10-21"

[[providers.models]]
id = "gpt-4o"
upstream_id = "my-gpt4o-deployment"   # the Azure deployment name
capabilities = ["chat"]

[[providers.models]]
id = "azure-embed"
upstream_id = "my-embedding-deployment"
capabilities = ["embed"]

OpenAI-compatible hosts

groq, together, fireworks, deepseek, openrouter, perplexity, xai, deepinfra and huggingface all work the same way: set the kind, point api_key_env at the host’s token, and (optionally) override base_url. The built-in default base URL is used otherwise.

[[providers]]
name = "groq"
kind = "groq"
api_key_env = "GROQ_API_KEY"
[[providers.models]]
id = "fast"
upstream_id = "llama-3.3-70b-versatile"
capabilities = ["chat"]

openrouter

Chat and embeddings, both through OpenRouter’s OpenAI-format API (/chat/completions and /embeddings). upstream_id is OpenRouter’s vendor/model id; the embedding catalog is listed at https://openrouter.ai/api/v1/embeddings/models. Token counts come from the upstream usage (OpenRouter’s extra usage.cost fields are ignored).

[[providers]]
name = "openrouter"
kind = "openrouter"
api_key_env = "OPENROUTER_API_KEY"
[[providers.models]]
id = "titan-embed"
upstream_id = "amazon/titan-embed-text-v2"
capabilities = ["embed"]

huggingface

The OpenAI-compatible Inference router (https://router.huggingface.co/v1), distinct from the self-hosted tei kind. api_key_env holds a Hugging Face token; upstream_id is a routed model id (often owner/model:provider).

[[providers]]
name = "hf"
kind = "huggingface"
api_key_env = "HF_TOKEN"
[[providers.models]]
id = "qwen"
upstream_id = "Qwen/Qwen2.5-72B-Instruct"
capabilities = ["chat"]

cloudflare

Cloudflare Workers AI. Chat and embeddings go through its OpenAI-compatible endpoint; reranking (bge-reranker-* models) goes through Workers AI’s own native POST /ai/run/{model} endpoint instead, since it is not part of the OpenAI-compatible surface - one [[providers]] entry serves all three capabilities against the same base_url. base_url is required because it embeds your account id; api_key_env holds a Cloudflare API token.

The native rerank request is { query, contexts: [{ text }, ...], top_k } (top_n is sent as top_k); the response is Cloudflare’s standard { result: { response: [{ id, score }, ...] }, success, errors } envelope, with id mapped back onto the original document index. Workers AI reports no token usage for this model; LUMEN derives a local estimate per ADR 003.

Decisions use the Clef models (clef, clef-flash) through POST {account root}/ai/run/@cf/cloudflare/{upstream_id} on the same base_url, with at most 64 questions and 4 images per request. HTTP 200 with success: false is LM-3003.

[[providers]]
name = "cf"
kind = "cloudflare"
api_key_env = "CLOUDFLARE_API_TOKEN"
base_url = "https://api.cloudflare.com/client/v4/accounts/YOUR_ACCOUNT_ID/ai/v1"
[[providers.models]]
id = "cf-llama"
upstream_id = "@cf/meta/llama-3.1-8b-instruct"
capabilities = ["chat"]
[[providers.models]]
id = "cf-rerank"
upstream_id = "@cf/baai/bge-reranker-base"
capabilities = ["rerank"]
[[providers.models]]
id = "clef"
upstream_id = "clef"
capabilities = ["decisions"]
modalities = ["text", "image"]
cost_per_1m_input = 0.24

perplexity decisions

The perplexity kind also serves Perplexity’s decision models (pplx-decider-v1.1-27b, pplx-decider-v1-27b) on POST {base_url}/v1/decisions, next to its chat models in the same provider block. Unknown request fields are stripped (Perplexity answers 400 to them), at most 128 questions, images go in state. A 504 (no answer within about 60 s) is a retryable LM-3003; a 413 (body over 32 MiB) is a non-retryable LM-3003.

[[providers]]
name = "perplexity"
kind = "perplexity"
api_key_env = "PERPLEXITY_API_KEY"
[[providers.models]]
id = "sonar"
capabilities = ["chat"]
[[providers.models]]
id = "pplx-decider"
upstream_id = "pplx-decider-v1.1-27b"
capabilities = ["decisions"]
modalities = ["text", "image"]
cost_per_1m_input = 0.02

vllm

Any self-hosted OpenAI-compatible server (vLLM, llama.cpp --api, LM Studio, SGLang, LocalAI). base_url required, API key optional. For Ollama, prefer the native ollama kind (chat + embed, see its section above); its OpenAI-compatible endpoint (http://localhost:11434/v1) also works under this kind, but you lose the native embed path and the /api/version health probe.

[[providers]]
name = "local"
kind = "vllm"
base_url = "http://localhost:8000/v1"
[[providers.models]]
id = "local-llama"
upstream_id = "meta-llama/Llama-3.1-8B-Instruct"
capabilities = ["chat", "embed"]

The developer role

OpenAI’s developer role is its newer name for system (Meilisearch chat sends its system prompt that way). Only the openai kind receives it verbatim; everywhere else it is handled as system, in message order with any system messages (ADR 016):

OpenAI chat extras on translated providers

OpenAI-compatible kinds forward every unmodeled request field verbatim, so response_format, seed, logprobs, top_logprobs, logit_bias, parallel_tool_calls, frequency_penalty and presence_penalty simply work there. The translated chat kinds rebuild the upstream request field by field, so each of these is either mapped onto a native equivalent or explicitly unsupported (issues #72, #91) - never silently lost:

¹ {"type": "json_object"} becomes generationConfig.responseMimeType: "application/json"; {"type": "json_schema"} additionally carries json_schema.schema as generationConfig.responseSchema, with JSON Schema keywords Gemini’s OpenAPI-subset schema rejects (additionalProperties, $schema) stripped recursively. Note that additionalProperties: false is therefore dropped, not enforced, on Gemini: the model may still emit extra keys (each strip is logged at debug level).

² Cohere v2 has no separate json_schema type: OpenAI’s {"type": "json_schema", "json_schema": {"schema": ...}} collapses onto Cohere’s {"type": "json_object", "json_schema": <schema>}; json_object and text pass through as-is.

³ Cohere v2 does accept a logprobs flag upstream, but its response shape is not translated back to OpenAI’s (and top_logprobs rides that same response shape), so the gateway treats both as unsupported rather than returning a malformed response.

⁴ parallel_tool_calls: false becomes Anthropic’s tool_choice.disable_parallel_tool_use: true (defaulting the choice to auto when the request carries tools but no explicit tool_choice); true is the default on both sides and needs no wire field.

What happens to an unsupported field depends on the provider’s strict flag (the same switch Ollama uses for embeddings dimensions, issue #25):

• strict = false (default): the field is dropped and a debug-level log line names the provider and field.
• strict = true: the request is rejected before any upstream call with an honest 400 (LM-1001) naming the field and provider - never a misleading 5xx.

[[providers]]
name = "anthropic"
kind = "anthropic"
api_key_env = "ANTHROPIC_API_KEY"
strict = true   # reject response_format/seed/logprobs instead of dropping

An unrecognised response_format shape (an unknown type value) is dropped with a debug log on the mapping providers, matching how unknown tool_choice shapes are handled: dropped, not guessed.

Vision (image input)

POST /v1/chat/completions accepts OpenAI’s content-parts message shape, so a user message can carry text and image parts in one array:

{
  "model": "gpt-4o",
  "messages": [{
    "role": "user",
    "content": [
      { "type": "text", "text": "What is this?" },
      { "type": "image_url", "image_url": { "url": "data:image/png;base64,iVBORw0KG..." } }
    ]
  }]
}

image_url.url is either a data:<media-type>;base64,<payload> URI (inline bytes) or a remote http(s) URL.

Per-model opt-in. A model only accepts image parts once its config declares the image modality (default is ["text"]):

[[providers.models]]
id = "gpt-4o"
capabilities = ["chat"]
modalities = ["text", "image"]   # opts this model into vision

GET /v1/models reflects this back as "modalities": ["text","image"] per model. Sending an image part to a model whose modalities lack "image" is rejected with LM-2003 (400, see docs/errors.md) before any upstream call.

Which kinds support it:

Never-fetch rule. LUMEN never dereferences a user-supplied image URL itself - doing so would be an SSRF vector (the gateway could be aimed at internal addresses) and would violate the streaming/latency pillar. A remote http(s) image_url is only ever forwarded to a provider that fetches it itself (OpenAI, Anthropic); Gemini’s inline_data field takes only inline bytes, so a remote URL routed to Gemini is rejected with LM-2004 (400) instead of the gateway silently fetching it on the caller’s behalf.

The LM-2004 pre-flight check inspects the primary provider of the model’s fallback chain. In the uncommon case where the primary accepts remote URLs (e.g. OpenAI) but a Gemini model is configured as a fallback, a request with a remote image URL passes pre-flight and, only if the primary then fails over to Gemini, surfaces as an upstream LM-3002 (502) - the gateway still never fetches the URL. Configure inline data: URIs when a Gemini fallback is in play.

Provider-native image sources (issue #12). Two provider-native reference forms are recognised in the image_url.url field, for callers whose images are already uploaded to the provider:

A provider-native reference routed to a model whose primary provider is not the reference’s own provider is rejected pre-flight with LM-2008 (400) - an honest client error instead of a confusing upstream failure. A Gemini Files API URI is also an https:// URL, but it is exempt from the LM-2004 remote-URL check: it is not a URL the provider would have to fetch, and its routing verdict belongs to the LM-2008 check.

gs:// caveat. The gateway forwards a gs:// URI to Gemini verbatim, but the Gemini Developer API (generativelanguage.googleapis.com, the default base_url of the google kind) documents fileData.fileUri for its own Files API URIs; Cloud Storage gs:// URIs are a Vertex AI capability. Against the default endpoint a gs:// reference is passed through and will be rejected by the upstream (surfacing as an upstream error naming google). It is still parsed and forwarded because the reference form is Gemini-native (mismatch routing stays an honest LM-2008), base_url may point at a Vertex-compatible gateway, and the upstream - never the gateway - is the authority on which URI forms it accepts. Upload via the Gemini Files API and pass the returned URI when targeting the Developer API.

For the mime type of a fileData part: it is included only when it can be confidently inferred from the URI’s file extension (.png, .jpg, …); otherwise it is omitted rather than guessed. Files API URIs carry no extension, and Gemini already knows the mime type recorded at upload time.

Accounting. Upstream-reported usage is authoritative and already folds in image tokens. When an upstream reports no usage at all, the local estimation fallback counts each image content part with a flat per-image heuristic (85 tokens at "detail": "low", 765 tokens otherwise) rather than counting it as zero, and the response is still flagged "estimated": true - see the ADR 003 addendum.

Fallbacks across providers

A virtual model with the fallback strategy backs one model with others when its provider exhausts retries or its circuit is open (or on any other trigger you list in fallback_on). Each target must exist and serve the virtual model’s capability (validated at boot and on every reload), which lets you survive a single-vendor outage by spanning providers:

[[providers.models]]
id = "openai/gpt-4o"
upstream_id = "gpt-4o"
capabilities = ["chat"]

[[virtual_models]]
id = "gpt-4o"
capability = "chat"
strategy = "fallback"
targets = [{ model = "openai/gpt-4o" }, { model = "claude-sonnet-4-5" }]   # different vendor, same capability

See docs/adr/005-resilience-execution.md for the resolution and circuit-breaker details, and config.example.toml for a fully worked multi-provider setup including a three-vendor rerank fallback chain.

Team provider keys (BYOK)

On the hosted gateway a budget group (one Lab account) can bring its own API keys for a provider (ADR 018). Only kinds that take a plain API key on the vendor URL are eligible: openai, anthropic, mistral, cohere, jina, voyage, groq, together, fireworks, deepseek, openrouter, perplexity, xai, deepinfra. Azure, Bedrock, Google/Vertex and keyless kinds are not.

• A group holds up to 10 keys per provider, managed under /admin/groups/{id}/byok (keys are sealed at rest and only label, last4 and status are ever returned).
• Keys are used round-robin. A 429 benches a key for max(Retry-After, 30s); a 401/403 marks it failing for 10 minutes.
• When no team key is usable, the request goes to the platform’s key only if the provider’s fallback is on (default off); otherwise it fails with LM-3021 (429) without calling upstream. With fallback off, later links of a fallback chain that have no team pool also refuse, so a chain never reaches the platform’s keys.
• Team-served requests carry no token cost; the gateway bills bandwidth instead (request plus response bytes, only when an upstream call was sent).
• A link served from a group’s team keys has its own circuit breaker, keyed by (provider, model, group): platform-key failures (say a run of 429s on the platform key) never stop a team served on its own keys, and a team’s failing keys never open the breaker for platform traffic. The lumen_circuit_state gauge reports platform breakers only.
• When an earlier link of a chain failed upstream and a later link is refused because the fallback is off, the client gets the earlier error, not LM-3021.

Error codes

Every error LUMEN returns to a client carries a stable LM-XXXX code, an HTTP status, and a coarse type. The response body is always:

{ "error": { "code": "LM-1001", "message": "…", "type": "invalid_request" } }

The type is one of invalid_request, upstream_error, internal, or client_cancelled. The gateway always distinguishes these situations and never disguises one as another - in particular, an internal malfunction is never reported as a misleading 401 (a lesson from OpenRouter outages), a malformed upstream response is a 502, never a gateway 500, and a client-initiated cancel (client_cancelled) is never counted as an internal malfunction either (see LM-6xxx below).

Codes are stable: once assigned, a code keeps its meaning across releases. The code prefix groups by cause: 1xxx request, 2xxx routing, 3xxx upstream, 4xxx auth/budget, 5xxx internal, 6xxx client-cancellation.

Request errors - LM-1xxx · type: invalid_request

Routing & capability-request errors - LM-2xxx · type: invalid_request

Upstream errors - LM-3xxx · type: upstream_error

These always name the provider that failed. Retriable ones may be transparently retried on a fallback before surfacing.

For LM-3001, LM-3020 (and LM-4002/LM-4003), a Retry-After value may be advertised. The three timeouts (LM-3011 first-token, LM-3012 connect, LM-3013 total) are distinct codes purely for debugging - see §6.4 and docs/adr/005-resilience-execution.md.

How resilience shapes these codes

The 3xxx codes are what a client sees only after the resilience machinery has given up. Before surfacing, a retryable failure (LM-3001 429, LM-3003 5xx, LM-3005/LM-3012 timeouts) is retried with exponential backoff, then the request fails over to the next target of the virtual model (its fallback_on triggers decide which failures move on, see Virtual models). The mapping between a failure and the code that eventually surfaces:

• LM-3020 (503) - the primary’s circuit is open and no fallback remained. Skipping an open circuit is instant (no upstream call), and the response carries a Retry-After equal to the cooldown remainder. A link served from a group’s team keys (BYOK) has its own breaker, apart from the platform key’s, so one never opens the other.
• LM-3021 (429) - the caller’s group brought its own keys for the provider and none is usable right now (rate-limited keys are benched for max(Retry-After, 30s), keys the provider rejects with 401/403 for 10 minutes). Decided by the gateway, so it is never retried and never counts against the circuit breaker. A fallback chain may move on under its rate_limited trigger, but a later link only runs if the group also serves it with its own keys: with the fallback off, no link reaches the platform’s keys. When an earlier link of the chain already failed upstream (say the team key answered 503), a later link refused this way surfaces that earlier error instead, so the refusal never hides the real failure. The message tells the team to add a key or turn on the fallback; it never names a key.
• LM-3004 (503) - every link in the fallback chain was tried and failed (retries exhausted or circuits open all the way down).
• LM-3013 (504) - the total per-request deadline elapsed while retrying or failing over; it bounds all attempts together, so a slow chain fails here rather than hanging.
• LM-3011 / LM-3012 (504) - first-token and connect timeouts; each is a retryable failure on its own before it surfaces.

A hard upstream client error (a 4xx bad request) is never retried or failed over - a different provider would reject it too - and surfaces immediately. Whichever model ultimately served a successful request is reported in the x-lumen-model-used response header.

Auth / budget errors - LM-4xxx · type: invalid_request

Codes pinned by the spec. Enforcement happens in memory, before any upstream call - a rejected request never leaks spend to a provider.

Internal errors - LM-5xxx · type: internal

Internal errors return an opaque "internal error" message to the client; the underlying detail is written only to the server logs, never the response.

Client-cancellation - LM-6xxx · type: client_cancelled

499 is the conventional “client closed request” status (nginx). The client is normally already gone by the time this would be returned, so the status exists for logs and metrics, not for anything a client reads. It is deliberately kept out of both type: internal and the 5xx status class: a client hanging up is not a gateway malfunction, and must never inflate the internal-error metrics or alerts a real one would (issue #11, see docs/adr/006-client-cancellation-error-code.md).

Two paths produce it. A cancellation surfacing mid-stream is emitted as a terminal SSE error frame carrying this envelope. A client that simply disconnects mid-stream never sees a frame at all, but the request’s accounting record (usage_log.status and the lumen_request_duration_seconds{status="499"} sample) is settled at 499 instead of being miscounted as a 200 success. A non-streaming disconnect drops the request before any outcome is recorded and produces no sample.

Performance baseline

LUMEN’s first pillar is performance: < 1 ms added p99 off-network, ~15 MB idle RAM, streaming that doesn’t re-serialize. This document records how those promises are measured, the numbers obtained, and how to reproduce every figure. Where a target is not fully measured in a given environment, the gap is stated honestly rather than papered over.

Methodology

Two layers, because “added latency” has two very different scales:

1. In-process overhead (cargo bench) - the CPU work the gateway adds per request, with no sockets involved: the resilience executor wrapping a provider call (circuit-breaker admit → retry loop → per-attempt timeout) plus the JSON (de)serialization the OpenAI surface does. This is the honest measure of “added latency off-network”: it excludes the upstream and the network entirely. Criterion, warmed, reports median with a 95 % CI.

2. End-to-end head-to-head (bench/, docker-compose + k6) - LUMEN and LiteLLM both proxying the same zero-latency mock upstream, driven by k6. Added latency = gateway percentile − direct-to-mock percentile. This is the number a user feels; it includes one extra localhost hop. Provided as a reproducible harness (see bench/README.md). Reported at two marks per target: total request time (http_req_duration) and time to first byte (http_req_waiting: request fully written → first response byte).

3. Streaming time to first bit (cargo bench -p server --bench stream_ttfb) - the one latency k6 cannot see: how much later a client receives the first SSE chunk of a stream: true response because the gateway sits in the middle. Real sockets, full LUMEN stack (axum → router → OpenAI-kind provider), instant mock upstream; the same request is timed direct-to-upstream and via-gateway, and the difference between the two distributions is the gateway’s added streaming TTFB. The companion integration test (tests/chat.rs::first_stream_chunk_reaches_the_client_before_the_upstream_finishes) proves the first frame is forwarded while the upstream is verifiably still mid-stream (gated tail, no timing races), so this bench measures eager forwarding, not buffer-then-flush.

Environment of the recorded run

Numbers are hardware-specific; re-run the commands on your target to get yours.

Results - in-process overhead (measured here)

Total added CPU per non-streaming chat request ≈ 3.2 µs (executor + parse + serialize). Streaming passthrough adds even less per chunk: it forwards upstream Bytes verbatim with no per-chunk serde (ADR 004), so the per-chunk cost is a bounded copy plus the [DONE]/heartbeat scan, not a deserialize.

SystemOne cost scales with the body: a typical request matches chat, while a maximum-size one (2026-09 measurement, noisy host) spends roughly 100 µs in the unavoidable JSON scan and the rest in per-question raw copies and the upstream re-serialization. It stays inside the 1 ms budget; a byte-splice passthrough that would roughly halve the large case is in docs/backlog.md. The large case needs CARGO_PROFILE_RELEASE_STRIP=false to build on macOS 27, where strip = true corrupts proc-macro dylibs (pre-existing, unrelated).

Decisions (ADR 017)

Same machine and toolchain as above (Apple Silicon arm64, macOS, rustc 1.97.0, release profile). Command: cargo bench -p server --bench gateway_overhead -- decisions (add CARGO_PROFILE_RELEASE_STRIP=false on macOS 27). Passthrough is parse + token estimate + one attempt’s clone and encode to the same vendor; cross-vendor is parse + encode to the other vendor’s wire format. “Large” is a big body, not a gateway limit (there is no question-count cap at the edge): 300 questions and 128 KB of state for TypeSafe (same shape as the old SystemOne large case), and 128 questions (chosen for the case) and 128 KB of input for OpenAI.

The final fixes (pre-sized output buffers, the name-only wire_ids fast path, a borrowed structured state in the OpenAI encoder, the question type borrowed at the TypeSafe edge) cut the TypeSafe large passthrough by 9 % (307 to 280 µs) and the OpenAI-to-TypeSafe large translation by 12 % (170 to 150 µs). The TypeSafe-to-OpenAI large translation is flat (414 to 417 µs, within run-to-run noise: a same-session run of the previous commit measured 415.6 µs); its cost is the 300 per-question instruction strings the OpenAI wire needs, not buffer growth. Every case is below the 1 ms budget; the worst is 417 µs. The TypeSafe large passthrough (280 µs) now matches the old systemone_request_pipeline_large (~280 µs), and the small one (3.23 µs) is within 2 % of the old systemone_request_pipeline_small (3.3 µs).

Streaming time to first bit (measured here)

Recorded with cargo bench -p server --bench stream_ttfb in the same environment as above: the span from dispatching a stream: true chat request to reading the first bytes of the SSE body, over real loopback sockets, against an instant mock upstream.

Added streaming TTFB ≈ 97 µs median (~0.1 ms): the extra wait before a client sees the first streamed token because LUMEN sits in the middle. That buys one full extra HTTP hop (accept, parse, route, provider request build, upstream connect-pooled call, headers + first-frame forward), measured end to end, and still lands an order of magnitude inside the < 1 ms pillar. The companion integration test (methodology point 3 above) guarantees the number means what it says: the first frame is forwarded while the upstream is still mid-stream, never buffered until end-of-stream.

Budget accounting per request (measured here)

Recorded 2026-10 with cargo bench -p auth --bench admit_settle (rustc 1.97.0, same machine): one admit plus its settle on a single thread, the in-memory budget accounting every authenticated request pays (ADR 009, and the ADR 015 settled-cost counter that billing reads). Lock-free atomics, no I/O; the database is never touched on the request path.

Well under a microsecond either way: budget enforcement and billing accounting are noise next to the ~3.2 µs of request CPU above. Like the SystemOne bench, it needs CARGO_PROFILE_RELEASE_STRIP=false to build on macOS 27.

The parallel bench guards against false sharing between tenants. Keys are loaded back to back at boot, so before KeyEntry and GroupEntry were aligned to the cache line (128 bytes on x86_64 and aarch64, 64 elsewhere) two neighbouring keys’ hot atomics, or one key’s tail and the next key’s Arc refcount, shared a line: the same bench measured 100.1 ns (96.8 – 103.4 ns), 2.5x the single-thread admit_settle_key of that run (39.3 ns) for keys that never contend logically. Aligned, it is 57.0 ns (-47 %), and the single-thread benches are unchanged within noise. The remaining gap to one thread (about 1.5x) was not investigated; it is not sharing between keys (the entries no longer share a line), and the M1 Pro runs two of the eight threads on efficiency cores. An interleaved A/B (unpadded and padded bench binaries run alternately, 4 rounds, on a heavily loaded machine, load average 54 to 59) agreed: the parallel bench was faster padded in every round (median about 127 ns to 77 ns), and the best single-thread admit_settle_key round was the same either way (38.5 ns unpadded, 38.4 ns padded). The price is padding, paid once per key at load: a key’s allocation grows from 184 to 384 bytes (about 2 MB per 10,000 keys), a group’s to 256 bytes.

Idle memory & binary size (measured here)

The Docker image was built from Dockerfile and smoke-tested: docker run -v config.toml -e OPENAI_API_KEY … answers /health 200 and /v1/models. At 10.6 MB it is well under the 30 MB image budget (§7.2). The amd64 image is built in CI via buildx (release.yml).

Results - loaded head-to-head vs LiteLLM (recorded baseline)

Recorded by bench/run.sh (see bench/README.md); full raw output is committed at bench/results/20260715T231135Z/.

Time to first byte for the same run (http_req_waiting; derived from the committed raw *.summary.json of that run - the metric was always recorded, its report table was added later, so the run’s own report.md predates it):

TTFB tracks total duration almost exactly in this scenario (non-streaming, tiny mock body: once the first byte is out, the rest follows within microseconds), so it carries the same caveat and the same conclusion: LUMEN delays the start of the response by ~2.6 ms at p50 on this noisy host, LiteLLM by ~317 ms.

RAM under load (docker stats, sampled mid-run): lumen ~7.6 MB, litellm ~1.03 GB.

Environment this specific run was recorded in: Darwin arm64, Docker 29.4.0, k6 v2.0.0, LUMEN at commit 51fc809, LiteLLM ghcr.io/berriai/litellm:v1.92.0@sha256:9ef6f45bc0104940571765e610c52a1d761b5ec85efcd193795281086ee61277, mockserver 5.15.0@sha256:0f9ef78c94894ac3e70135d156193b25e23872575d58e2228344964273b4af6b.

Caveat, stated honestly: this run was recorded on a shared development host (not dedicated benchmarking hardware), which is visible in the noisy direct-baseline numbers above (a 0 ms mock should not itself show ~836 ms p99; the mockserver JVM saturates a core by itself, so the direct phase measures host contention as much as transport). Treat the absolute numbers as illustrative, not authoritative. The relative comparison - lumen vs litellm, same mock, same host, same run - is the meaningful part: LUMEN added ~2.5 ms at p50 over direct (9.44 vs 6.91 ms) and its tail stayed below the noisy direct baseline, while LiteLLM’s p50 grew by ~47× (323.75 ms) and its p99 reached 6.5 s. LUMEN sustained ~25× LiteLLM’s throughput at roughly 1/140th the RAM. Re-run bench/run.sh on dedicated hardware for numbers to make capacity decisions on; see “Updating the pinned versions” in bench/README.md for how to refresh this baseline (new pinned image, new results/ directory, update the link above).

Targets

Honest summary: the off-network overhead is measured and is microseconds, comfortably inside the pillar-1 budget. The full loaded head-to-head vs LiteLLM is now a committed, reproducible baseline (pinned versions, one command, recorded result linked above) rather than just a runnable harness; the relative LUMEN-vs-LiteLLM comparison from it is solid, while the absolute and direct-vs-gateway numbers should be treated as this particular (noisy, shared) host’s numbers, not a hardware-independent claim.