Providers

LUMEN ships twenty-seven built-in provider kinds - sixteen native integrations (their own request/response translation, including deployment-routed azure, SigV4-signed bedrock and decisions-only typesafe) plus eleven OpenAI-compatible hosts that reuse the OpenAI path with a per-kind base URL. Each [[providers]] block in your config selects one with a kind string and gives it a unique name (your own label). Each [[providers.models]] block under it exposes a model to clients:

[[providers]]
name = "my-openai"        # your label; must be unique
kind = "openai"           # selects the built-in implementation
api_key_env = "OPENAI_API_KEY"   # NAME of the env var holding the key
# base_url = "https://…"  # optional override (required for self-hosted kinds)

[[providers.models]]
id = "gpt-4o"             # the id clients send (owned entirely by you)
upstream_id = "gpt-4o-2024-08-06"   # what LUMEN sends upstream (defaults to `id`)
capabilities = ["chat"]   # any of "chat", "embed", "rerank", "decisions"
release_date = "2024-08-06"   # optional; lets clients sort GET /v1/models by release

Rules that apply to every provider:

• API keys are never in the config. api_key_env names an environment variable; LUMEN reads it only when a request actually routes to that provider. A hosted provider whose env var is unset fails only at use, not at boot - a partial set of keys is fine.
• release_date is metadata only. An ISO date ("2024-08-06", or the unquoted TOML date 2024-08-06), validated at load. GET /v1/models returns it as release_date and as the OpenAI integer created (Unix seconds, midnight UTC); both are omitted for an undated model. It never affects routing.
• Model ids are globally unique across all providers. A collision aborts startup and names both offending providers. Several ids may map to one upstream_id (versioned aliasing).
• capabilities must match the kind. A model can only declare capabilities its provider kind implements (table below); this is validated at boot.
• base_url is an optional override for hosted kinds, and required for the self-hosted kinds (tei, ollama), which are keyless.
• Batching: an embed request with more inputs than the provider’s batch limit is split into sub-batches, run with bounded concurrency, and reassembled in the original order. The limits below are built in.
• Multimodal embeddings (M9): declare modalities = ["text", "image"] on a model to accept image content parts on /v1/embeddings. input items may be strings or arrays of parts ({"type":"text",...} / {"type":"image_url",...}). Images are passed as data: URIs, or - with [image_fetch] enabled - as remote http(s) URLs the gateway fetches under SSRF/resource guards and inlines. Image input to a model without "image" is rejected with LM-2003; a remote URL with fetching disabled is LM-2005. Cohere (embed-v4) and Voyage embed a combined text+image vector per item; Jina embeds one modality per item (a mixed item is sent as its image, its caption text is not combined). See the multimodal-embeddings design spec for the full guard list.

The typesafe kind serves decisions only; reranking a decision model goes through a virtual model remap (see Decision models as rerankers). perplexity and cloudflare (OpenAI-compatible table below) also serve decisions. The full decisions matrix is below. The together kind (in the OpenAI-compatible table below) additionally serves rerank (LlamaRank) natively; see its section for the model config.

OpenAI-compatible hosts (chat + embed via the OpenAI path). The Embed column reflects what each host actually serves upstream: groq, deepseek, perplexity and xai expose no /embeddings endpoint (perplexity and cloudflare do serve decisions, see Decisions by provider kind), so a model declaring embed on those kinds is rejected at config load (it could only ever 404 at request time) - unless the provider sets a custom base_url, which is taken to mean an operator-run proxy that may serve embeddings:

Self-hosted or catalog-dependent kinds (vllm, huggingface, cloudflare) stay permissive: the operator controls what their endpoint serves. If one of the embed-less hosts above later ships an embeddings API, point a kind = "openai" provider at it with a base_url override, or file an issue to update the capability table.

All embed-serving OpenAI-compatible kinds use a 2048-input embed batch limit. Anything that speaks the OpenAI wire format but isn’t listed can still be used via kind = "openai" with a base_url override.

cloudflare additionally serves rerank (not shown in the table above, which covers only the chat/embed OpenAI-compatible path): its BAAI bge-reranker-* models are served through Workers AI’s native /ai/run/{model} endpoint rather than an OpenAI-compatible one. See ### cloudflare below.

openai

• kind: openai · capabilities: chat, embed, decisions
• Auth: api_key_env (e.g. OPENAI_API_KEY), sent as a bearer token.
• base_url: optional; defaults to OpenAI’s public API. Set it to point at any OpenAI-compatible endpoint.
• Embed batch limit: 2048 inputs per upstream call.
• Decisions: gpt-6-luna on POST {base_url}/decisions (public beta). A choice needs at least 2 options and a request at most 128 images. Set cost_per_1m_input = 0.10 and modalities = ["text", "image"]. See Decisions.

[[providers]]
name = "openai"
kind = "openai"
api_key_env = "OPENAI_API_KEY"

[[providers.models]]
id = "gpt-4o"
upstream_id = "gpt-4o-2024-08-06"
capabilities = ["chat"]

[[providers.models]]
id = "text-embedding-3-small"
capabilities = ["embed"]

[[providers.models]]
id = "gpt-6-luna"
capabilities = ["decisions"]
modalities = ["text", "image"]
cost_per_1m_input = 0.10

mistral

• kind: mistral · capabilities: chat, embed (OpenAI-compatible).
• Auth: api_key_env (e.g. MISTRAL_API_KEY), bearer token.
• base_url: optional override.
• Embed batch limit: 512.

[[providers]]
name = "mistral"
kind = "mistral"
api_key_env = "MISTRAL_API_KEY"

[[providers.models]]
id = "mistral-small"
upstream_id = "mistral-small-latest"
capabilities = ["chat"]

anthropic

• kind: anthropic · capabilities: chat only.
• Auth: api_key_env (e.g. ANTHROPIC_API_KEY). LUMEN authenticates with the x-api-key / anthropic-version headers, not a bearer token.
• Translation: OpenAI ⇄ Anthropic is bidirectional, including tools and streaming events, so clients keep using the OpenAI wire format. parallel_tool_calls: false maps to tool_choice.disable_parallel_tool_use; response_format, seed, logprobs, frequency_penalty and presence_penalty have no Messages API equivalent - see chat extras and the strict flag.
• base_url: optional override.

[[providers]]
name = "anthropic"
kind = "anthropic"
api_key_env = "ANTHROPIC_API_KEY"

[[providers.models]]
id = "claude-sonnet-4-5"
upstream_id = "claude-sonnet-4-5-20250929"
capabilities = ["chat"]

google

• kind: google · capabilities: chat, embed (Gemini).
• Auth: api_key_env (e.g. GEMINI_API_KEY). The key rides the x-goog-api-key header, never the URL.
• Translation: OpenAI ⇄ Gemini, including streaming (streamGenerateContent). response_format JSON mode maps to generationConfig.responseMimeType / responseSchema, seed to generationConfig.seed, and frequency_penalty / presence_penalty to generationConfig.frequencyPenalty / presencePenalty; logprobs and parallel_tool_calls have no mapping - see chat extras.
• Embeddings: served through models/{model}:batchEmbedContents (gemini-embedding-001, text-embedding-004, …). One inner request per input; the OpenAI dimensions field maps to outputDimensionality. The API is text-only: pre-tokenized token-id arrays and image content parts are rejected with LM-1001 before any upstream call. Usage follows ADR 003: usageMetadata.promptTokenCount is reported when the upstream returns it, otherwise the gateway derives a local estimate marked estimated.
• Embed batch limit: 100 inputs per upstream call (Gemini’s documented batchEmbedContents ceiling); the gateway splits larger requests.
• base_url: optional override.

[[providers]]
name = "google"
kind = "google"
api_key_env = "GEMINI_API_KEY"

[[providers.models]]
id = "gemini-2.0-flash"
upstream_id = "gemini-2.0-flash"
capabilities = ["chat"]

[[providers.models]]
id = "gemini-embedding-001"
capabilities = ["embed"]

vertex_ai

• kind: vertex_ai · capabilities: chat, embed (Gemini and text-embedding-* models on Google Cloud Vertex AI). Distinct from google, which is the public Gemini Developer API: Vertex uses regional endpoints and GCP OAuth instead of a static API key.
• Auth: api_key_env names an env var holding the full service-account key JSON (the contents of the key file downloaded from GCP, not a path and not an API key). LUMEN signs an RS256 JWT assertion with the account’s private key, exchanges it at the account’s token_uri for a short-lived OAuth2 access token (scope cloud-platform), and sends it as a Bearer header. Tokens are cached in memory and refreshed 60 s before expiry, so the exchange stays off the per-request hot path. The private key is redacted from all Debug output and never appears in logs or errors.
• base_url: required - it carries the GCP region (e.g. us-central1), not a URL. The endpoint is derived from it: https://{region}-aiplatform.googleapis.com/v1/projects/{project}/locations/{region}/publishers/google/models/{model}:generateContent (and :streamGenerateContent?alt=sse when streaming).
• Project id: taken from the service-account JSON’s project_id.
• Translation: chat is identical to google (same GenerateContent wire schema), including streaming. Like Gemini, only inline base64 image data is accepted; remote image URLs are rejected with LM-2004.
• Embeddings: Vertex does NOT expose batchEmbedContents; embeddings go through the prediction API on the same regional, project-scoped path: .../publishers/google/models/{model}:predict with instances[].content. The OpenAI dimensions field maps to parameters.outputDimensionality. Text-only: token-id arrays and image parts are rejected with LM-1001. Usage follows ADR 003: per-input statistics.token_count values are summed and reported as upstream usage; when absent the gateway derives a local estimate marked estimated.
• Embed batch limit: 1 input per upstream call. gemini-embedding-001 accepts a single instance per :predict request (other text-embedding-* models take more, but the limit is per-model, so the universally safe value is used); the gateway fans larger requests out over concurrent calls.

[[providers]]
name = "vertex"
kind = "vertex_ai"
# The env var holds the service-account key file's JSON contents:
#   export VERTEX_SA_JSON="$(cat service-account.json)"
api_key_env = "VERTEX_SA_JSON"
base_url = "us-central1"   # GCP region

[[providers.models]]
id = "gemini-flash-vertex"
upstream_id = "gemini-2.0-flash"
capabilities = ["chat"]

[[providers.models]]
id = "gemini-embedding-vertex"
upstream_id = "gemini-embedding-001"
capabilities = ["embed"]

bedrock

• kind: bedrock · capabilities: chat, embed.
• Auth: AWS Signature Version 4 (SigV4), not a bearer key. Credentials are read from the standard AWS environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and optionally AWS_SESSION_TOKEN for temporary credentials) on every request, so values updated in the process environment (or a config hot reload) take effect without a restart. api_key_env is optional and, if set, overrides only the secret access key. The secret and session token are never logged or shown in Debug.
• Credential scope (v1): only static keys and pre-issued STS session tokens. There is no AWS credential-provider chain (no IMDS/instance roles, SSO, profiles or credential_process); an expired session token keeps failing with 403 until the environment supplies a fresh one.
• API (chat): the Bedrock Converse API (POST /model/{modelId}/converse and /converse-stream), which gives one uniform schema across the Anthropic, Meta Llama, Amazon Titan/Nova, Mistral and Cohere model families. The legacy per-model InvokeModel chat schemas are intentionally not implemented (Converse covers the same chat models).
• API (embeddings): Bedrock has no Converse equivalent for embeddings, so the embed path uses per-model InvokeModel (POST /model/{modelId}/invoke), routed by model id (issue #95):
  • Amazon Titan (amazon.titan-embed-text-v2:0 and predecessors): embeds ONE text per call ({ "inputText": ... }); the gateway loops one signed request per input and reassembles the batch in order. Titan v2 honors dimensions (mapped to Titan’s dimensions/normalize); older Titan models do not, so dimensions is dropped (or rejected under strict).
  • Cohere Embed on Bedrock (cohere.embed-english-v3, cohere.embed-multilingual-v3): embeds a batch in one call ({ "texts": [...], "input_type": ... }); input_type defaults to search_document and honors an override. Cohere does not accept dimensions.
  • Batch limit: reported as 1 (the conservative floor across the two families, whose real per-call limits are 1 for Titan and 96 for Cohere), so the router splits every batch into single-input sub-batches it runs concurrently. Pre-tokenized (token-id) and image inputs are rejected with an honest 400 before any upstream call (both families are text-only).
  • Usage (ADR 003): Titan’s inputTextTokenCount and Cohere’s x-amzn-bedrock-input-token-count response header are reported as upstream usage; when absent the request edge derives the local estimate (never a silent zero).
• Region / base_url: set base_url to the runtime endpoint for your region, https://bedrock-runtime.{region}.amazonaws.com; the region is parsed back out of it for the SigV4 signing scope. VPC/PrivateLink endpoint hosts (bedrock-runtime.{region}.vpce.amazonaws.com, including a vpce-…-prefixed DNS name) are recognised too. For any other custom endpoint the region comes from AWS_REGION / AWS_DEFAULT_REGION; if no source yields a region, startup fails with a clear error rather than silently signing for a wrong region.
• Translation: OpenAI ⇄ Converse is bidirectional, including system prompts, inferenceConfig (max tokens, temperature, top-p, stop sequences), tools, and streaming. Streaming arrives as AWS event-stream binary frames, decoded and translated to OpenAI chunks. Usage (inputTokens / outputTokens) is mapped per ADR 003. Converse has no equivalent for response_format, seed, logprobs, parallel_tool_calls, frequency_penalty or presence_penalty
  • see chat extras and the strict flag.
• Images: only inline data: URIs are supported (Converse takes raw image bytes); a remote image URL is rejected (LM-2004) since Bedrock cannot fetch one.

[[providers]]
name = "bedrock"
kind = "bedrock"
base_url = "https://bedrock-runtime.us-east-1.amazonaws.com"
# api_key_env = "AWS_SECRET_ACCESS_KEY"   # optional secret override

[[providers.models]]
id = "bedrock-claude-sonnet-4-5"
upstream_id = "us.anthropic.claude-sonnet-4-5-20250929-v1:0"
capabilities = ["chat"]
modalities = ["text", "image"]

[[providers.models]]
id = "bedrock-titan-embed"
upstream_id = "amazon.titan-embed-text-v2:0"
capabilities = ["embed"]

[[providers.models]]
id = "bedrock-cohere-embed"
upstream_id = "cohere.embed-english-v3"
capabilities = ["embed"]

cohere

• kind: cohere · capabilities: chat, embed, rerank. A single model can serve any combination.
• Auth: api_key_env (e.g. COHERE_API_KEY), bearer token.
• Chat: Command R / R+ via POST /v2/chat, including streaming. The wire shape is OpenAI-adjacent (roles live directly in messages, no top-level system hoist like Anthropic; tool_calls are already OpenAI-shaped), so translation is closer to identity than Anthropic’s. tool_choice collapses to Cohere’s REQUIRED/NONE (forcing one specific named tool has no v2 equivalent and falls back to auto). response_format, seed, frequency_penalty and presence_penalty map onto Cohere’s native fields; logprobs and parallel_tool_calls do not - see chat extras. Usage prefers usage.tokens (actual counts) over usage.billed_units (what’s charged); a response reporting neither leaves the gateway’s local estimator to fill in an honestly-flagged count (ADR 003).
• Vision (issue #73): a user message carrying image parts is translated to Cohere v2 content blocks (text / image_url, OpenAI-shaped); a text-only message keeps the plain-string form, and non-user roles always flatten to text (Cohere only admits image content on user messages). Declare modalities = ["text", "image"] on a vision model (Command-A-Vision) to opt in. Both inline data: URIs and remote http(s) URLs are forwarded (Cohere fetches remote URLs itself, so LM-2004 does not apply); the optional detail hint (low/high/auto) passes through untouched. Provider-native references (anthropic-file:, gs://, Gemini Files API URIs) are rejected pre-flight with LM-2008.
• Embed batch limit: 96.
• Cost: rerank is billed in search units (cost_per_1k_searches).
• input_type override: Cohere’s embed v2 API requires an input_type and the gateway cannot know query-vs-document intent, so it defaults to search_document (the indexing case). Set input_type as an extra field on the /v1/embeddings request body to override it per request, e.g. {"model": "embed-multilingual", "input": "...", "input_type": "search_query"}. Allowed values: search_document, search_query, classification, clustering. An unrecognized value is rejected with LM-1001 before any upstream call. The field is consumed at the gateway: only the Cohere translation reads it, and it is never forwarded in the outgoing body of any other provider (a strict OpenAI-compatible upstream such as vLLM could reject unknown fields).

[[providers]]
name = "cohere"
kind = "cohere"
api_key_env = "COHERE_API_KEY"

[[providers.models]]
id = "command-r-plus"
upstream_id = "command-r-plus-08-2024"
capabilities = ["chat"]

[[providers.models]]
id = "rerank-english"
upstream_id = "rerank-v3.5"
capabilities = ["rerank"]
cost_per_1k_searches = 2.0

[[providers.models]]
id = "embed-multilingual"
upstream_id = "embed-v4.0"
capabilities = ["embed", "rerank"]

jina

• kind: jina · capabilities: embed, rerank (hosted).
• Auth: api_key_env (e.g. JINA_API_KEY), bearer token.
• Embed batch limit: 2048.

[[providers]]
name = "jina"
kind = "jina"
api_key_env = "JINA_API_KEY"

[[providers.models]]
id = "jina-rerank"
upstream_id = "jina-reranker-v2-base-multilingual"
capabilities = ["rerank"]

voyage

• kind: voyage · capabilities: embed, rerank (hosted).
• Auth: api_key_env (e.g. VOYAGE_API_KEY), bearer token.
• Embed batch limit: 128.

[[providers]]
name = "voyage"
kind = "voyage"
api_key_env = "VOYAGE_API_KEY"

[[providers.models]]
id = "voyage-rerank"
upstream_id = "rerank-2"
capabilities = ["rerank"]

mixedbread

• kind: mixedbread · capabilities: rerank (hosted, mxbai-rerank-*).
• Auth: api_key_env (e.g. MXBAI_API_KEY), bearer token.
• base_url: optional; defaults to https://api.mixedbread.com/v1.
• Schema note: Mixedbread’s endpoint is POST /v1/reranking (note the path: reranking, not rerank) and renames the request fields (input instead of documents, top_k instead of top_n) with results nested under data; the gateway translates transparently.
• Usage: billed in tokens, so the gateway reports an estimated token count (ADR 003) rather than upstream search units.

[[providers]]
name = "mixedbread"
kind = "mixedbread"
api_key_env = "MXBAI_API_KEY"

[[providers.models]]
id = "mxbai-rerank"
upstream_id = "mixedbread-ai/mxbai-rerank-large-v1"
capabilities = ["rerank"]

pinecone

• kind: pinecone · capabilities: rerank (hosted inference).
• Auth: api_key_env (e.g. PINECONE_API_KEY), sent as the Api-Key header (not a bearer token), alongside a pinned X-Pinecone-API-Version header the inference API requires.
• base_url: optional; defaults to https://api.pinecone.io.
• Schema note: documents are sent as { "text": ... } objects; only the default text rank field is used (rank_fields selection is out of scope for v1).
• Usage: Pinecone reports usage.rerank_units, carried through verbatim as the response’s search_units (not estimated).

[[providers]]
name = "pinecone"
kind = "pinecone"
api_key_env = "PINECONE_API_KEY"

[[providers.models]]
id = "pinecone-rerank"
upstream_id = "pinecone-rerank-v0"
capabilities = ["rerank"]

typesafe

• kind: typesafe · capabilities: decisions (TypeSafe’s Jev typed-decision models, and any other TypeSafe-format vendor, served on POST /v1/decisions; see Decisions and ADR 017). Reranking goes through a virtual model remap (Decision models as rerankers).
• Auth: api_key_env (e.g. TYPESAFE_API_KEY), sent as a bearer token. Never logged, redacted from Debug.
• base_url: optional; defaults to https://api.typesafe.ai. An override is the API root. Without decisions_path a trailing /v1 is tolerated and the gateway posts to {base}/v1/systemone.
• decisions_path (optional, typesafe only): the upstream path appended to base_url verbatim. Must start with /. Default /v1/systemone. This and the next field make typesafe the generic TypeSafe-format kind.
• forward_unknown_fields (optional, typesafe only, default true): forward unknown top-level request fields upstream, as TypeSafe accepts them. Set false for a vendor that rejects them (Perplexity answers 400). Setting either field on another kind is refused at config load.
• Schema note: the request and response are TypeSafe’s own, forwarded verbatim for a TypeSafe-format client (only model is rewritten to the upstream_id); state, question and answer JSON keeps its key order byte-for-byte. The question contract is validated at the edge (LM-2011 / LM-1001) before any upstream call. An OpenAI-format client is translated to this wire format. A noul question without instructions is not sent to a typesafe model (Jev requires them): the target is skipped.
• Models: jev-latest (stable alias), jev-preview (preview alias) and versioned ids such as jev-1.13.0, mapped through upstream_id.
• Errors: TypeSafe 401 / 422 surface as LM-3003 (502, not retried, upstream body not forwarded); 429 is LM-3001 honouring Retry-After; 529 Overloaded is a retryable 5xx, so retries apply, and a virtual model’s provider_error fallback trigger matches it (not rate_limited, which is an upstream 429).
• Cost: Jev bills input tokens only, $0.042 per 1M (TypeSafe pricing, 2026-09): set cost_per_1m_input = 0.042 and no output price.
• Usage: upstream usage.input_tokens / output_tokens when reported; otherwise an estimated input count over state plus every question (ADR 003), output 0.
• Upstream limits (TypeSafe, 2026-09): 64k tokens of context per request (32k for state plus the longest question), 250k tokens/s and 1200 RPM, dynamic. Not enforced by the gateway.

[[providers]]
name = "typesafe"
kind = "typesafe"
api_key_env = "TYPESAFE_API_KEY"

[[providers.models]]
id = "jev"
upstream_id = "jev-latest"
capabilities = ["decisions"]
cost_per_1m_input = 0.042

# A pinned version for reproducible answers.
[[providers.models]]
id = "typesafe/jev-1.13.0"
upstream_id = "jev-1.13.0"
capabilities = ["decisions"]
cost_per_1m_input = 0.042

# Clients call `jev-1.13.0`; it falls back to the stable alias when the pinned
# version is overloaded (529) or its circuit is open.
[[virtual_models]]
id = "jev-1.13.0"
capability = "decisions"
strategy = "fallback"
targets = [{ model = "typesafe/jev-1.13.0" }, { model = "jev" }]

# Jev as a reranker: plain /v1/rerank, one predicate question per document. A
# `typesafe` model cannot declare `rerank` itself; a virtual model's `remap`
# does the conversion (see docs/virtual-models.md).
[[virtual_models]]
id = "jev-rerank"
capability = "rerank"
strategy = "single"

[[virtual_models.targets]]
model = "jev"
remap = { strategy = "predicate" }

Other TypeSafe-format vendors

Each recipe below is a typesafe provider with its own base_url and decisions_path. The TypeSafe format is shared, the path is not. LUMEN has not been tested against these vendors’ live APIs: check each vendor’s documentation for the model id and whether it tolerates unknown fields.

# Liquid d1. Unverified: set forward_unknown_fields = true if Liquid
# accepts unknown fields.
[[providers]]
name = "liquid"
kind = "typesafe"
base_url = "https://api.liquid.ai"
decisions_path = "/decisions/v1/systemone"
forward_unknown_fields = false
api_key_env = "LIQUID_API_KEY"
[[providers.models]]
id = "liquid-d1"
upstream_id = "d1"
capabilities = ["decisions"]

# Inception Mercury Decide.
[[providers]]
name = "inception"
kind = "typesafe"
base_url = "https://api.inceptionlabs.ai"
decisions_path = "/v1/decisions"
api_key_env = "INCEPTION_API_KEY"
[[providers.models]]
id = "mercury-decide"
upstream_id = "mercury-decide"
capabilities = ["decisions"]

# Self-hosted Kev (kev.serve) on its default path /v1/systemone.
[[providers]]
name = "kev"
kind = "typesafe"
base_url = "http://localhost:8000"
[[providers.models]]
id = "kev"
upstream_id = "kev"
capabilities = ["decisions"]

Upstage Solar Decide serves /v1/systemone, but its base URL is unconfirmed: set base_url to the host Upstage documents for your account and keep the default path. The upstream_id values above for Inception and Kev (mercury-decide, kev) are placeholders: use the model id your vendor documents. For Inception, Kev and Upstage the forward_unknown_fields default (true) is untested.

Decisions by provider kind

Five kinds serve decisions. All use bearer auth except ollama (keyless; a bearer key is sent if one is configured). Limits are checked before any upstream call; a target that cannot take the request is skipped (see Decisions).

Unknown top-level request fields are forwarded by typesafe (unless forward_unknown_fields = false) and stripped by the other kinds; openai sends only safety_identifier. There is no question-count cap at the gateway edge: the per-target limits above apply when a target is chosen. Prices are the published ones as of 2026-10 and only feed cost_per_1m_input: set it on each model, the gateway never fills prices in. A model accepts images only if it declares modalities = ["text", "image"] (the default is ["text"]).

perplexity and cloudflare dispatch per model, so one provider block can hold chat models and decision models side by side. Status mapping is the usual one (429 is LM-3001 honouring Retry-After, 5xx is retryable LM-3003, upstream bodies never reach the client); see Error codes. An Ollama server that is not running is LM-3004 and falls back like any unavailable upstream.

nvidia (NIM)

• kind: nvidia · capabilities: rerank (NVIDIA NIM ranking).
• Auth: keyless by default (self-hosted NIMs run without a key); supply api_key_env (e.g. NVIDIA_API_KEY) for the hosted API, sent as a bearer token.
• base_url: required - the NIM root (e.g. http://localhost:8000 or the NVIDIA-hosted ranking endpoint root). The gateway posts to {base}/v1/ranking.
• Schema note: the request nests query: { text } and passages: [{ text }]; there is no top_n on the wire, so the gateway requests the full ranking and truncates to top_n afterwards (as for TEI).
• Score semantics: NIM returns a raw logit, passed through unchanged as relevance_score. Scores are unbounded (can be negative) and are only comparable within a single response; higher is more relevant. No sigmoid is applied.
• Usage: NIM reports no token usage, so the gateway reports an estimated token count (ADR 003).

[[providers]]
name = "nvidia-nim"
kind = "nvidia"
base_url = "http://localhost:8000"
# api_key_env = "NVIDIA_API_KEY"   # only for the hosted API

[[providers.models]]
id = "nvidia-rerank"
upstream_id = "nvidia/llama-3.2-nv-rerankqa-1b-v2"
capabilities = ["rerank"]

together (rerank)

The together kind (see the OpenAI-compatible section for chat/embed) also serves rerank (LlamaRank) natively through Together’s Cohere-shaped /rerank endpoint. One [[providers]] entry with kind = "together" serves all three capabilities against the same base_url and bearer key. Rerank is billed in tokens, so the gateway reports an estimated token count (ADR 003).

[[providers]]
name = "together"
kind = "together"
api_key_env = "TOGETHER_API_KEY"

[[providers.models]]
id = "llama-rank"
upstream_id = "Salesforce/Llama-Rank-V1"
capabilities = ["rerank"]

tei (self-hosted)

• kind: tei · capabilities: embed, rerank.
• Auth: keyless.
• base_url: required - points at your Text Embeddings Inference server. TEI serves one model per process, so upstream_id is ignored by the upstream but kept for your own clarity.
• Embed batch limit: 32.

[[providers]]
name = "tei-local"
kind = "tei"
base_url = "http://localhost:8081"

[[providers.models]]
id = "bge-reranker"
upstream_id = "BAAI/bge-reranker-large"
capabilities = ["rerank"]

ollama (self-hosted)

• kind: ollama · capabilities: chat, embed, decisions (Ollama 0.35+, on POST {base_url}/v1/systemone; choice and score take 2 to 26 options or levels, and only clef and clef-flash accept images; see Local decision models with Ollama).
• Auth: keyless.
• base_url: required - points at your Ollama server root (no /v1). Embeddings use Ollama’s native POST /api/embed; chat goes through Ollama’s OpenAI-compatible endpoint, which lives under /v1 on the same root - the gateway appends the /v1 itself, so keep base_url as the bare server root either way.
• Chat: served by the shared OpenAI-compatible path - streaming (SSE passthrough), cancellation, and token accounting (upstream usage when Ollama reports it, otherwise a local count marked estimated, ADR 003) all work exactly as for the openai kind.
• api_key asymmetry: if you set an api_key_env on this kind (e.g. an Ollama server behind an authenticated reverse proxy), the chat path sends it as a bearer token but the native embed path currently sends no Authorization header at all - keep the embed route unauthenticated at the proxy, or front only /v1 with auth.
• Embed batch limit: 512.
• Tip: a local model may take a while to load into VRAM on its first call - relax first_token_timeout_ms / total_timeout_ms on the provider block (see config.example.toml). A self-hosted box on a slow link can also override connect_timeout_ms; note that doing so gives this provider its own (unpooled) HTTP client (ADR 005, 2026-07-15 amendment), whereas the first-token and total overrides do not. All three fall back to their global defaults when unset.

[[providers]]
name = "ollama-local"
kind = "ollama"
base_url = "http://localhost:11434"
first_token_timeout_ms = 60000
total_timeout_ms = 120000
connect_timeout_ms = 10000  # optional: own client, relaxed connect deadline

[[providers.models]]
id = "local-llama"
upstream_id = "llama3.2"
capabilities = ["chat"]

[[providers.models]]
id = "nomic-embed"
upstream_id = "nomic-embed-text"
capabilities = ["embed"]

[[providers.models]]
id = "nimble"
upstream_id = "nimble"
capabilities = ["decisions"]

azure

• kind: azure · capabilities: chat, embed. Reuses the OpenAI JSON schema verbatim; only the URL, auth, and routing differ from openai.
• Auth: api_key_env, sent as the api-key header (never a bearer token).
• base_url: required - your Azure resource endpoint, e.g. https://<resource>.openai.azure.com (no shared public default, every resource is operator-specific).
• api_version: optional - pins the Azure API version sent as the api-version query parameter on every request (issue #65). For back-compat the older form still works: append ?api-version=YYYY-MM-DD to base_url. Precedence: the explicit api_version field wins over a base_url query string, which wins over LUMEN’s pinned built-in default (see the azure module doc comment for the exact value). Any query parameters on base_url other than api-version are ignored when building request URLs. api_version is azure-only: setting it on any other kind is rejected at boot.
• Deployment routing: Azure routes by URL path (/openai/deployments/{deployment}/...), not by the model field in the body. Set each model’s upstream_id to the Azure deployment name - the same upstream_id mechanism every other kind uses for aliasing already carries it through.
• Embed batch limit: 2048 (same array-size ceiling as the OpenAI embedding models Azure hosts).

[[providers]]
name = "azure-openai"
kind = "azure"
api_key_env = "AZURE_OPENAI_API_KEY"
base_url = "https://my-resource.openai.azure.com"
api_version = "2024-10-21"

[[providers.models]]
id = "gpt-4o"
upstream_id = "my-gpt4o-deployment"   # the Azure deployment name
capabilities = ["chat"]

[[providers.models]]
id = "azure-embed"
upstream_id = "my-embedding-deployment"
capabilities = ["embed"]

OpenAI-compatible hosts

groq, together, fireworks, deepseek, openrouter, perplexity, xai, deepinfra and huggingface all work the same way: set the kind, point api_key_env at the host’s token, and (optionally) override base_url. The built-in default base URL is used otherwise.

[[providers]]
name = "groq"
kind = "groq"
api_key_env = "GROQ_API_KEY"
[[providers.models]]
id = "fast"
upstream_id = "llama-3.3-70b-versatile"
capabilities = ["chat"]

openrouter

Chat and embeddings, both through OpenRouter’s OpenAI-format API (/chat/completions and /embeddings). upstream_id is OpenRouter’s vendor/model id; the embedding catalog is listed at https://openrouter.ai/api/v1/embeddings/models. Token counts come from the upstream usage (OpenRouter’s extra usage.cost fields are ignored).

[[providers]]
name = "openrouter"
kind = "openrouter"
api_key_env = "OPENROUTER_API_KEY"
[[providers.models]]
id = "titan-embed"
upstream_id = "amazon/titan-embed-text-v2"
capabilities = ["embed"]

huggingface

The OpenAI-compatible Inference router (https://router.huggingface.co/v1), distinct from the self-hosted tei kind. api_key_env holds a Hugging Face token; upstream_id is a routed model id (often owner/model:provider).

[[providers]]
name = "hf"
kind = "huggingface"
api_key_env = "HF_TOKEN"
[[providers.models]]
id = "qwen"
upstream_id = "Qwen/Qwen2.5-72B-Instruct"
capabilities = ["chat"]

cloudflare

Cloudflare Workers AI. Chat and embeddings go through its OpenAI-compatible endpoint; reranking (bge-reranker-* models) goes through Workers AI’s own native POST /ai/run/{model} endpoint instead, since it is not part of the OpenAI-compatible surface - one [[providers]] entry serves all three capabilities against the same base_url. base_url is required because it embeds your account id; api_key_env holds a Cloudflare API token.

The native rerank request is { query, contexts: [{ text }, ...], top_k } (top_n is sent as top_k); the response is Cloudflare’s standard { result: { response: [{ id, score }, ...] }, success, errors } envelope, with id mapped back onto the original document index. Workers AI reports no token usage for this model; LUMEN derives a local estimate per ADR 003.

Decisions use the Clef models (clef, clef-flash) through POST {account root}/ai/run/@cf/cloudflare/{upstream_id} on the same base_url, with at most 64 questions and 4 images per request. HTTP 200 with success: false is LM-3003.

[[providers]]
name = "cf"
kind = "cloudflare"
api_key_env = "CLOUDFLARE_API_TOKEN"
base_url = "https://api.cloudflare.com/client/v4/accounts/YOUR_ACCOUNT_ID/ai/v1"
[[providers.models]]
id = "cf-llama"
upstream_id = "@cf/meta/llama-3.1-8b-instruct"
capabilities = ["chat"]
[[providers.models]]
id = "cf-rerank"
upstream_id = "@cf/baai/bge-reranker-base"
capabilities = ["rerank"]
[[providers.models]]
id = "clef"
upstream_id = "clef"
capabilities = ["decisions"]
modalities = ["text", "image"]
cost_per_1m_input = 0.24

perplexity decisions

The perplexity kind also serves Perplexity’s decision models (pplx-decider-v1.1-27b, pplx-decider-v1-27b) on POST {base_url}/v1/decisions, next to its chat models in the same provider block. Unknown request fields are stripped (Perplexity answers 400 to them), at most 128 questions, images go in state. A 504 (no answer within about 60 s) is a retryable LM-3003; a 413 (body over 32 MiB) is a non-retryable LM-3003.

[[providers]]
name = "perplexity"
kind = "perplexity"
api_key_env = "PERPLEXITY_API_KEY"
[[providers.models]]
id = "sonar"
capabilities = ["chat"]
[[providers.models]]
id = "pplx-decider"
upstream_id = "pplx-decider-v1.1-27b"
capabilities = ["decisions"]
modalities = ["text", "image"]
cost_per_1m_input = 0.02

vllm

Any self-hosted OpenAI-compatible server (vLLM, llama.cpp --api, LM Studio, SGLang, LocalAI). base_url required, API key optional. For Ollama, prefer the native ollama kind (chat + embed, see its section above); its OpenAI-compatible endpoint (http://localhost:11434/v1) also works under this kind, but you lose the native embed path and the /api/version health probe.

[[providers]]
name = "local"
kind = "vllm"
base_url = "http://localhost:8000/v1"
[[providers.models]]
id = "local-llama"
upstream_id = "meta-llama/Llama-3.1-8B-Instruct"
capabilities = ["chat", "embed"]

The developer role

OpenAI’s developer role is its newer name for system (Meilisearch chat sends its system prompt that way). Only the openai kind receives it verbatim; everywhere else it is handled as system, in message order with any system messages (ADR 016):

OpenAI chat extras on translated providers

OpenAI-compatible kinds forward every unmodeled request field verbatim, so response_format, seed, logprobs, top_logprobs, logit_bias, parallel_tool_calls, frequency_penalty and presence_penalty simply work there. The translated chat kinds rebuild the upstream request field by field, so each of these is either mapped onto a native equivalent or explicitly unsupported (issues #72, #91) - never silently lost:

¹ {"type": "json_object"} becomes generationConfig.responseMimeType: "application/json"; {"type": "json_schema"} additionally carries json_schema.schema as generationConfig.responseSchema, with JSON Schema keywords Gemini’s OpenAPI-subset schema rejects (additionalProperties, $schema) stripped recursively. Note that additionalProperties: false is therefore dropped, not enforced, on Gemini: the model may still emit extra keys (each strip is logged at debug level).

² Cohere v2 has no separate json_schema type: OpenAI’s {"type": "json_schema", "json_schema": {"schema": ...}} collapses onto Cohere’s {"type": "json_object", "json_schema": <schema>}; json_object and text pass through as-is.

³ Cohere v2 does accept a logprobs flag upstream, but its response shape is not translated back to OpenAI’s (and top_logprobs rides that same response shape), so the gateway treats both as unsupported rather than returning a malformed response.

⁴ parallel_tool_calls: false becomes Anthropic’s tool_choice.disable_parallel_tool_use: true (defaulting the choice to auto when the request carries tools but no explicit tool_choice); true is the default on both sides and needs no wire field.

What happens to an unsupported field depends on the provider’s strict flag (the same switch Ollama uses for embeddings dimensions, issue #25):

• strict = false (default): the field is dropped and a debug-level log line names the provider and field.
• strict = true: the request is rejected before any upstream call with an honest 400 (LM-1001) naming the field and provider - never a misleading 5xx.

[[providers]]
name = "anthropic"
kind = "anthropic"
api_key_env = "ANTHROPIC_API_KEY"
strict = true   # reject response_format/seed/logprobs instead of dropping

An unrecognised response_format shape (an unknown type value) is dropped with a debug log on the mapping providers, matching how unknown tool_choice shapes are handled: dropped, not guessed.

Vision (image input)

POST /v1/chat/completions accepts OpenAI’s content-parts message shape, so a user message can carry text and image parts in one array:

{
  "model": "gpt-4o",
  "messages": [{
    "role": "user",
    "content": [
      { "type": "text", "text": "What is this?" },
      { "type": "image_url", "image_url": { "url": "data:image/png;base64,iVBORw0KG..." } }
    ]
  }]
}

image_url.url is either a data:<media-type>;base64,<payload> URI (inline bytes) or a remote http(s) URL.

Per-model opt-in. A model only accepts image parts once its config declares the image modality (default is ["text"]):

[[providers.models]]
id = "gpt-4o"
capabilities = ["chat"]
modalities = ["text", "image"]   # opts this model into vision

GET /v1/models reflects this back as "modalities": ["text","image"] per model. Sending an image part to a model whose modalities lack "image" is rejected with LM-2003 (400, see docs/errors.md) before any upstream call.

Which kinds support it:

Never-fetch rule. LUMEN never dereferences a user-supplied image URL itself - doing so would be an SSRF vector (the gateway could be aimed at internal addresses) and would violate the streaming/latency pillar. A remote http(s) image_url is only ever forwarded to a provider that fetches it itself (OpenAI, Anthropic); Gemini’s inline_data field takes only inline bytes, so a remote URL routed to Gemini is rejected with LM-2004 (400) instead of the gateway silently fetching it on the caller’s behalf.

The LM-2004 pre-flight check inspects the primary provider of the model’s fallback chain. In the uncommon case where the primary accepts remote URLs (e.g. OpenAI) but a Gemini model is configured as a fallback, a request with a remote image URL passes pre-flight and, only if the primary then fails over to Gemini, surfaces as an upstream LM-3002 (502) - the gateway still never fetches the URL. Configure inline data: URIs when a Gemini fallback is in play.

Provider-native image sources (issue #12). Two provider-native reference forms are recognised in the image_url.url field, for callers whose images are already uploaded to the provider:

A provider-native reference routed to a model whose primary provider is not the reference’s own provider is rejected pre-flight with LM-2008 (400) - an honest client error instead of a confusing upstream failure. A Gemini Files API URI is also an https:// URL, but it is exempt from the LM-2004 remote-URL check: it is not a URL the provider would have to fetch, and its routing verdict belongs to the LM-2008 check.

gs:// caveat. The gateway forwards a gs:// URI to Gemini verbatim, but the Gemini Developer API (generativelanguage.googleapis.com, the default base_url of the google kind) documents fileData.fileUri for its own Files API URIs; Cloud Storage gs:// URIs are a Vertex AI capability. Against the default endpoint a gs:// reference is passed through and will be rejected by the upstream (surfacing as an upstream error naming google). It is still parsed and forwarded because the reference form is Gemini-native (mismatch routing stays an honest LM-2008), base_url may point at a Vertex-compatible gateway, and the upstream - never the gateway - is the authority on which URI forms it accepts. Upload via the Gemini Files API and pass the returned URI when targeting the Developer API.

For the mime type of a fileData part: it is included only when it can be confidently inferred from the URI’s file extension (.png, .jpg, …); otherwise it is omitted rather than guessed. Files API URIs carry no extension, and Gemini already knows the mime type recorded at upload time.

Accounting. Upstream-reported usage is authoritative and already folds in image tokens. When an upstream reports no usage at all, the local estimation fallback counts each image content part with a flat per-image heuristic (85 tokens at "detail": "low", 765 tokens otherwise) rather than counting it as zero, and the response is still flagged "estimated": true - see the ADR 003 addendum.

Fallbacks across providers

A virtual model with the fallback strategy backs one model with others when its provider exhausts retries or its circuit is open (or on any other trigger you list in fallback_on). Each target must exist and serve the virtual model’s capability (validated at boot and on every reload), which lets you survive a single-vendor outage by spanning providers:

[[providers.models]]
id = "openai/gpt-4o"
upstream_id = "gpt-4o"
capabilities = ["chat"]

[[virtual_models]]
id = "gpt-4o"
capability = "chat"
strategy = "fallback"
targets = [{ model = "openai/gpt-4o" }, { model = "claude-sonnet-4-5" }]   # different vendor, same capability

See docs/adr/005-resilience-execution.md for the resolution and circuit-breaker details, and config.example.toml for a fully worked multi-provider setup including a three-vendor rerank fallback chain.

Team provider keys (BYOK)

On the hosted gateway a budget group (one Lab account) can bring its own API keys for a provider (ADR 018). Only kinds that take a plain API key on the vendor URL are eligible: openai, anthropic, mistral, cohere, jina, voyage, groq, together, fireworks, deepseek, openrouter, perplexity, xai, deepinfra. Azure, Bedrock, Google/Vertex and keyless kinds are not.

• A group holds up to 10 keys per provider, managed under /admin/groups/{id}/byok (keys are sealed at rest and only label, last4 and status are ever returned).
• Keys are used round-robin. A 429 benches a key for max(Retry-After, 30s); a 401/403 marks it failing for 10 minutes.
• When no team key is usable, the request goes to the platform’s key only if the provider’s fallback is on (default off); otherwise it fails with LM-3021 (429) without calling upstream. With fallback off, later links of a fallback chain that have no team pool also refuse, so a chain never reaches the platform’s keys.
• Team-served requests carry no token cost; the gateway bills bandwidth instead (request plus response bytes, only when an upstream call was sent).
• A link served from a group’s team keys has its own circuit breaker, keyed by (provider, model, group): platform-key failures (say a run of 429s on the platform key) never stop a team served on its own keys, and a team’s failing keys never open the breaker for platform traffic. The lumen_circuit_state gauge reports platform breakers only.
• When an earlier link of a chain failed upstream and a later link is refused because the fallback is off, the client gets the earlier error, not LM-3021.