Providers

LUMEN ships twenty-six built-in provider kinds - fifteen native integrations (their own request/response translation, including deployment-routed azure and SigV4-signed bedrock) plus eleven OpenAI-compatible hosts that reuse the OpenAI path with a per-kind base URL. Each [[providers]] block in your config selects one with a kind string and gives it a unique name (your own label). Each [[providers.models]] block under it exposes a model to clients:

[[providers]]
name = "my-openai"        # your label; must be unique
kind = "openai"           # selects the built-in implementation
api_key_env = "OPENAI_API_KEY"   # NAME of the env var holding the key
# base_url = "https://…"  # optional override (required for self-hosted kinds)

[[providers.models]]
id = "gpt-4o"             # the id clients send (owned entirely by you)
upstream_id = "gpt-4o-2024-08-06"   # what LUMEN sends upstream (defaults to `id`)
capabilities = ["chat"]   # any of "chat", "embed", "rerank"

Rules that apply to every provider:

• API keys are never in the config. api_key_env names an environment variable; LUMEN reads it only when a request actually routes to that provider. A hosted provider whose env var is unset fails only at use, not at boot - a partial set of keys is fine.
• Model ids are globally unique across all providers. A collision aborts startup and names both offending providers. Several ids may map to one upstream_id (versioned aliasing).
• capabilities must match the kind. A model can only declare capabilities its provider kind implements (table below); this is validated at boot.
• base_url is an optional override for hosted kinds, and required for the self-hosted kinds (tei, ollama), which are keyless.
• Batching: an embed request with more inputs than the provider’s batch limit is split into sub-batches, run with bounded concurrency, and reassembled in the original order. The limits below are built in.
• Multimodal embeddings (M9): declare modalities = ["text", "image"] on a model to accept image content parts on /v1/embeddings. input items may be strings or arrays of parts ({"type":"text",...} / {"type":"image_url",...}). Images are passed as data: URIs, or - with [image_fetch] enabled - as remote http(s) URLs the gateway fetches under SSRF/resource guards and inlines. Image input to a model without "image" is rejected with LM-2003; a remote URL with fetching disabled is LM-2005. Cohere (embed-v4) and Voyage embed a combined text+image vector per item; Jina embeds one modality per item (a mixed item is sent as its image, its caption text is not combined). See the multimodal-embeddings design spec for the full guard list.

The together kind (in the OpenAI-compatible table below) additionally serves rerank (LlamaRank) natively; see its section for the model config.

OpenAI-compatible hosts (chat + embed via the OpenAI path). The Embed column reflects what each host actually serves upstream: groq, deepseek, openrouter, perplexity and xai expose no /embeddings endpoint, so a model declaring embed on those kinds is rejected at config load (it could only ever 404 at request time) - unless the provider sets a custom base_url, which is taken to mean an operator-run proxy that may serve embeddings:

Self-hosted or catalog-dependent kinds (vllm, huggingface, cloudflare) stay permissive: the operator controls what their endpoint serves. If one of the embed-less hosts above later ships an embeddings API, point a kind = "openai" provider at it with a base_url override, or file an issue to update the capability table.

All embed-serving OpenAI-compatible kinds use a 2048-input embed batch limit. Anything that speaks the OpenAI wire format but isn’t listed can still be used via kind = "openai" with a base_url override.

cloudflare additionally serves rerank (not shown in the table above, which covers only the chat/embed OpenAI-compatible path): its BAAI bge-reranker-* models are served through Workers AI’s native /ai/run/{model} endpoint rather than an OpenAI-compatible one. See ### cloudflare below.

openai

• kind: openai · capabilities: chat, embed
• Auth: api_key_env (e.g. OPENAI_API_KEY), sent as a bearer token.
• base_url: optional; defaults to OpenAI’s public API. Set it to point at any OpenAI-compatible endpoint.
• Embed batch limit: 2048 inputs per upstream call.

[[providers]]
name = "openai"
kind = "openai"
api_key_env = "OPENAI_API_KEY"

[[providers.models]]
id = "gpt-4o"
upstream_id = "gpt-4o-2024-08-06"
capabilities = ["chat"]

[[providers.models]]
id = "text-embedding-3-small"
capabilities = ["embed"]

mistral

• kind: mistral · capabilities: chat, embed (OpenAI-compatible).
• Auth: api_key_env (e.g. MISTRAL_API_KEY), bearer token.
• base_url: optional override.
• Embed batch limit: 512.

[[providers]]
name = "mistral"
kind = "mistral"
api_key_env = "MISTRAL_API_KEY"

[[providers.models]]
id = "mistral-small"
upstream_id = "mistral-small-latest"
capabilities = ["chat"]

anthropic

• kind: anthropic · capabilities: chat only.
• Auth: api_key_env (e.g. ANTHROPIC_API_KEY). LUMEN authenticates with the x-api-key / anthropic-version headers, not a bearer token.
• Translation: OpenAI ⇄ Anthropic is bidirectional, including tools and streaming events, so clients keep using the OpenAI wire format. parallel_tool_calls: false maps to tool_choice.disable_parallel_tool_use; response_format, seed, logprobs, frequency_penalty and presence_penalty have no Messages API equivalent - see chat extras and the strict flag.
• base_url: optional override.

[[providers]]
name = "anthropic"
kind = "anthropic"
api_key_env = "ANTHROPIC_API_KEY"

[[providers.models]]
id = "claude-sonnet-4-5"
upstream_id = "claude-sonnet-4-5-20250929"
capabilities = ["chat"]

google

• kind: google · capabilities: chat, embed (Gemini).
• Auth: api_key_env (e.g. GEMINI_API_KEY). The key rides the x-goog-api-key header, never the URL.
• Translation: OpenAI ⇄ Gemini, including streaming (streamGenerateContent). response_format JSON mode maps to generationConfig.responseMimeType / responseSchema, seed to generationConfig.seed, and frequency_penalty / presence_penalty to generationConfig.frequencyPenalty / presencePenalty; logprobs and parallel_tool_calls have no mapping - see chat extras.
• Embeddings: served through models/{model}:batchEmbedContents (gemini-embedding-001, text-embedding-004, …). One inner request per input; the OpenAI dimensions field maps to outputDimensionality. The API is text-only: pre-tokenized token-id arrays and image content parts are rejected with LM-1001 before any upstream call. Usage follows ADR 003: usageMetadata.promptTokenCount is reported when the upstream returns it, otherwise the gateway derives a local estimate marked estimated.
• Embed batch limit: 100 inputs per upstream call (Gemini’s documented batchEmbedContents ceiling); the gateway splits larger requests.
• base_url: optional override.

[[providers]]
name = "google"
kind = "google"
api_key_env = "GEMINI_API_KEY"

[[providers.models]]
id = "gemini-2.0-flash"
upstream_id = "gemini-2.0-flash"
capabilities = ["chat"]

[[providers.models]]
id = "gemini-embedding-001"
capabilities = ["embed"]

vertex_ai

• kind: vertex_ai · capabilities: chat, embed (Gemini and text-embedding-* models on Google Cloud Vertex AI). Distinct from google, which is the public Gemini Developer API: Vertex uses regional endpoints and GCP OAuth instead of a static API key.
• Auth: api_key_env names an env var holding the full service-account key JSON (the contents of the key file downloaded from GCP, not a path and not an API key). LUMEN signs an RS256 JWT assertion with the account’s private key, exchanges it at the account’s token_uri for a short-lived OAuth2 access token (scope cloud-platform), and sends it as a Bearer header. Tokens are cached in memory and refreshed 60 s before expiry, so the exchange stays off the per-request hot path. The private key is redacted from all Debug output and never appears in logs or errors.
• base_url: required - it carries the GCP region (e.g. us-central1), not a URL. The endpoint is derived from it: https://{region}-aiplatform.googleapis.com/v1/projects/{project}/locations/{region}/publishers/google/models/{model}:generateContent (and :streamGenerateContent?alt=sse when streaming).
• Project id: taken from the service-account JSON’s project_id.
• Translation: chat is identical to google (same GenerateContent wire schema), including streaming. Like Gemini, only inline base64 image data is accepted; remote image URLs are rejected with LM-2004.
• Embeddings: Vertex does NOT expose batchEmbedContents; embeddings go through the prediction API on the same regional, project-scoped path: .../publishers/google/models/{model}:predict with instances[].content. The OpenAI dimensions field maps to parameters.outputDimensionality. Text-only: token-id arrays and image parts are rejected with LM-1001. Usage follows ADR 003: per-input statistics.token_count values are summed and reported as upstream usage; when absent the gateway derives a local estimate marked estimated.
• Embed batch limit: 1 input per upstream call. gemini-embedding-001 accepts a single instance per :predict request (other text-embedding-* models take more, but the limit is per-model, so the universally safe value is used); the gateway fans larger requests out over concurrent calls.

[[providers]]
name = "vertex"
kind = "vertex_ai"
# The env var holds the service-account key file's JSON contents:
#   export VERTEX_SA_JSON="$(cat service-account.json)"
api_key_env = "VERTEX_SA_JSON"
base_url = "us-central1"   # GCP region

[[providers.models]]
id = "gemini-flash-vertex"
upstream_id = "gemini-2.0-flash"
capabilities = ["chat"]

[[providers.models]]
id = "gemini-embedding-vertex"
upstream_id = "gemini-embedding-001"
capabilities = ["embed"]

bedrock

• kind: bedrock · capabilities: chat, embed.
• Auth: AWS Signature Version 4 (SigV4), not a bearer key. Credentials are read from the standard AWS environment variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and optionally AWS_SESSION_TOKEN for temporary credentials) on every request, so values updated in the process environment (or a config hot reload) take effect without a restart. api_key_env is optional and, if set, overrides only the secret access key. The secret and session token are never logged or shown in Debug.
• Credential scope (v1): only static keys and pre-issued STS session tokens. There is no AWS credential-provider chain (no IMDS/instance roles, SSO, profiles or credential_process); an expired session token keeps failing with 403 until the environment supplies a fresh one.
• API (chat): the Bedrock Converse API (POST /model/{modelId}/converse and /converse-stream), which gives one uniform schema across the Anthropic, Meta Llama, Amazon Titan/Nova, Mistral and Cohere model families. The legacy per-model InvokeModel chat schemas are intentionally not implemented (Converse covers the same chat models).
• API (embeddings): Bedrock has no Converse equivalent for embeddings, so the embed path uses per-model InvokeModel (POST /model/{modelId}/invoke), routed by model id (issue #95):
  • Amazon Titan (amazon.titan-embed-text-v2:0 and predecessors): embeds ONE text per call ({ "inputText": ... }); the gateway loops one signed request per input and reassembles the batch in order. Titan v2 honors dimensions (mapped to Titan’s dimensions/normalize); older Titan models do not, so dimensions is dropped (or rejected under strict).
  • Cohere Embed on Bedrock (cohere.embed-english-v3, cohere.embed-multilingual-v3): embeds a batch in one call ({ "texts": [...], "input_type": ... }); input_type defaults to search_document and honors an override. Cohere does not accept dimensions.
  • Batch limit: reported as 1 (the conservative floor across the two families, whose real per-call limits are 1 for Titan and 96 for Cohere), so the router splits every batch into single-input sub-batches it runs concurrently. Pre-tokenized (token-id) and image inputs are rejected with an honest 400 before any upstream call (both families are text-only).
  • Usage (ADR 003): Titan’s inputTextTokenCount and Cohere’s x-amzn-bedrock-input-token-count response header are reported as upstream usage; when absent the request edge derives the local estimate (never a silent zero).
• Region / base_url: set base_url to the runtime endpoint for your region, https://bedrock-runtime.{region}.amazonaws.com; the region is parsed back out of it for the SigV4 signing scope. VPC/PrivateLink endpoint hosts (bedrock-runtime.{region}.vpce.amazonaws.com, including a vpce-…-prefixed DNS name) are recognised too. For any other custom endpoint the region comes from AWS_REGION / AWS_DEFAULT_REGION; if no source yields a region, startup fails with a clear error rather than silently signing for a wrong region.
• Translation: OpenAI ⇄ Converse is bidirectional, including system prompts, inferenceConfig (max tokens, temperature, top-p, stop sequences), tools, and streaming. Streaming arrives as AWS event-stream binary frames, decoded and translated to OpenAI chunks. Usage (inputTokens / outputTokens) is mapped per ADR 003. Converse has no equivalent for response_format, seed, logprobs, parallel_tool_calls, frequency_penalty or presence_penalty
  • see chat extras and the strict flag.
• Images: only inline data: URIs are supported (Converse takes raw image bytes); a remote image URL is rejected (LM-2004) since Bedrock cannot fetch one.

[[providers]]
name = "bedrock"
kind = "bedrock"
base_url = "https://bedrock-runtime.us-east-1.amazonaws.com"
# api_key_env = "AWS_SECRET_ACCESS_KEY"   # optional secret override

[[providers.models]]
id = "bedrock-claude-sonnet-4-5"
upstream_id = "us.anthropic.claude-sonnet-4-5-20250929-v1:0"
capabilities = ["chat"]
modalities = ["text", "image"]

[[providers.models]]
id = "bedrock-titan-embed"
upstream_id = "amazon.titan-embed-text-v2:0"
capabilities = ["embed"]

[[providers.models]]
id = "bedrock-cohere-embed"
upstream_id = "cohere.embed-english-v3"
capabilities = ["embed"]

cohere

• kind: cohere · capabilities: chat, embed, rerank. A single model can serve any combination.
• Auth: api_key_env (e.g. COHERE_API_KEY), bearer token.
• Chat: Command R / R+ via POST /v2/chat, including streaming. The wire shape is OpenAI-adjacent (roles live directly in messages, no top-level system hoist like Anthropic; tool_calls are already OpenAI-shaped), so translation is closer to identity than Anthropic’s. tool_choice collapses to Cohere’s REQUIRED/NONE (forcing one specific named tool has no v2 equivalent and falls back to auto). response_format, seed, frequency_penalty and presence_penalty map onto Cohere’s native fields; logprobs and parallel_tool_calls do not - see chat extras. Usage prefers usage.tokens (actual counts) over usage.billed_units (what’s charged); a response reporting neither leaves the gateway’s local estimator to fill in an honestly-flagged count (ADR 003).
• Vision (issue #73): a user message carrying image parts is translated to Cohere v2 content blocks (text / image_url, OpenAI-shaped); a text-only message keeps the plain-string form, and non-user roles always flatten to text (Cohere only admits image content on user messages). Declare modalities = ["text", "image"] on a vision model (Command-A-Vision) to opt in. Both inline data: URIs and remote http(s) URLs are forwarded (Cohere fetches remote URLs itself, so LM-2004 does not apply); the optional detail hint (low/high/auto) passes through untouched. Provider-native references (anthropic-file:, gs://, Gemini Files API URIs) are rejected pre-flight with LM-2008.
• Embed batch limit: 96.
• Cost: rerank is billed in search units (cost_per_1k_searches).
• input_type override: Cohere’s embed v2 API requires an input_type and the gateway cannot know query-vs-document intent, so it defaults to search_document (the indexing case). Set input_type as an extra field on the /v1/embeddings request body to override it per request, e.g. {"model": "embed-multilingual", "input": "...", "input_type": "search_query"}. Allowed values: search_document, search_query, classification, clustering. An unrecognized value is rejected with LM-1001 before any upstream call. The field is consumed at the gateway: only the Cohere translation reads it, and it is never forwarded in the outgoing body of any other provider (a strict OpenAI-compatible upstream such as vLLM could reject unknown fields).

[[providers]]
name = "cohere"
kind = "cohere"
api_key_env = "COHERE_API_KEY"

[[providers.models]]
id = "command-r-plus"
upstream_id = "command-r-plus-08-2024"
capabilities = ["chat"]

[[providers.models]]
id = "rerank-english"
upstream_id = "rerank-v3.5"
capabilities = ["rerank"]
cost_per_1k_searches = 2.0

[[providers.models]]
id = "embed-multilingual"
upstream_id = "embed-v4.0"
capabilities = ["embed", "rerank"]

jina

• kind: jina · capabilities: embed, rerank (hosted).
• Auth: api_key_env (e.g. JINA_API_KEY), bearer token.
• Embed batch limit: 2048.

[[providers]]
name = "jina"
kind = "jina"
api_key_env = "JINA_API_KEY"

[[providers.models]]
id = "jina-rerank"
upstream_id = "jina-reranker-v2-base-multilingual"
capabilities = ["rerank"]

voyage

• kind: voyage · capabilities: embed, rerank (hosted).
• Auth: api_key_env (e.g. VOYAGE_API_KEY), bearer token.
• Embed batch limit: 128.

[[providers]]
name = "voyage"
kind = "voyage"
api_key_env = "VOYAGE_API_KEY"

[[providers.models]]
id = "voyage-rerank"
upstream_id = "rerank-2"
capabilities = ["rerank"]

mixedbread

• kind: mixedbread · capabilities: rerank (hosted, mxbai-rerank-*).
• Auth: api_key_env (e.g. MXBAI_API_KEY), bearer token.
• base_url: optional; defaults to https://api.mixedbread.com/v1.
• Schema note: Mixedbread’s endpoint is POST /v1/reranking (note the path: reranking, not rerank) and renames the request fields (input instead of documents, top_k instead of top_n) with results nested under data; the gateway translates transparently.
• Usage: billed in tokens, so the gateway reports an estimated token count (ADR 003) rather than upstream search units.

[[providers]]
name = "mixedbread"
kind = "mixedbread"
api_key_env = "MXBAI_API_KEY"

[[providers.models]]
id = "mxbai-rerank"
upstream_id = "mixedbread-ai/mxbai-rerank-large-v1"
capabilities = ["rerank"]

pinecone

• kind: pinecone · capabilities: rerank (hosted inference).
• Auth: api_key_env (e.g. PINECONE_API_KEY), sent as the Api-Key header (not a bearer token), alongside a pinned X-Pinecone-API-Version header the inference API requires.
• base_url: optional; defaults to https://api.pinecone.io.
• Schema note: documents are sent as { "text": ... } objects; only the default text rank field is used (rank_fields selection is out of scope for v1).
• Usage: Pinecone reports usage.rerank_units, carried through verbatim as the response’s search_units (not estimated).

[[providers]]
name = "pinecone"
kind = "pinecone"
api_key_env = "PINECONE_API_KEY"

[[providers.models]]
id = "pinecone-rerank"
upstream_id = "pinecone-rerank-v0"
capabilities = ["rerank"]

nvidia (NIM)

• kind: nvidia · capabilities: rerank (NVIDIA NIM ranking).
• Auth: keyless by default (self-hosted NIMs run without a key); supply api_key_env (e.g. NVIDIA_API_KEY) for the hosted API, sent as a bearer token.
• base_url: required - the NIM root (e.g. http://localhost:8000 or the NVIDIA-hosted ranking endpoint root). The gateway posts to {base}/v1/ranking.
• Schema note: the request nests query: { text } and passages: [{ text }]; there is no top_n on the wire, so the gateway requests the full ranking and truncates to top_n afterwards (as for TEI).
• Score semantics: NIM returns a raw logit, passed through unchanged as relevance_score. Scores are unbounded (can be negative) and are only comparable within a single response; higher is more relevant. No sigmoid is applied.
• Usage: NIM reports no token usage, so the gateway reports an estimated token count (ADR 003).

[[providers]]
name = "nvidia-nim"
kind = "nvidia"
base_url = "http://localhost:8000"
# api_key_env = "NVIDIA_API_KEY"   # only for the hosted API

[[providers.models]]
id = "nvidia-rerank"
upstream_id = "nvidia/llama-3.2-nv-rerankqa-1b-v2"
capabilities = ["rerank"]

together (rerank)

The together kind (see the OpenAI-compatible section for chat/embed) also serves rerank (LlamaRank) natively through Together’s Cohere-shaped /rerank endpoint. One [[providers]] entry with kind = "together" serves all three capabilities against the same base_url and bearer key. Rerank is billed in tokens, so the gateway reports an estimated token count (ADR 003).

[[providers]]
name = "together"
kind = "together"
api_key_env = "TOGETHER_API_KEY"

[[providers.models]]
id = "llama-rank"
upstream_id = "Salesforce/Llama-Rank-V1"
capabilities = ["rerank"]

tei (self-hosted)

• kind: tei · capabilities: embed, rerank.
• Auth: keyless.
• base_url: required - points at your Text Embeddings Inference server. TEI serves one model per process, so upstream_id is ignored by the upstream but kept for your own clarity.
• Embed batch limit: 32.

[[providers]]
name = "tei-local"
kind = "tei"
base_url = "http://localhost:8081"

[[providers.models]]
id = "bge-reranker"
upstream_id = "BAAI/bge-reranker-large"
capabilities = ["rerank"]

ollama (self-hosted)

• kind: ollama · capabilities: chat, embed.
• Auth: keyless.
• base_url: required - points at your Ollama server root (no /v1). Embeddings use Ollama’s native POST /api/embed; chat goes through Ollama’s OpenAI-compatible endpoint, which lives under /v1 on the same root - the gateway appends the /v1 itself, so keep base_url as the bare server root either way.
• Chat: served by the shared OpenAI-compatible path - streaming (SSE passthrough), cancellation, and token accounting (upstream usage when Ollama reports it, otherwise a local count marked estimated, ADR 003) all work exactly as for the openai kind.
• api_key asymmetry: if you set an api_key_env on this kind (e.g. an Ollama server behind an authenticated reverse proxy), the chat path sends it as a bearer token but the native embed path currently sends no Authorization header at all - keep the embed route unauthenticated at the proxy, or front only /v1 with auth.
• Embed batch limit: 512.
• Tip: a local model may take a while to load into VRAM on its first call - relax first_token_timeout_ms / total_timeout_ms on the provider block (see config.example.toml). A self-hosted box on a slow link can also override connect_timeout_ms; note that doing so gives this provider its own (unpooled) HTTP client (ADR 005, 2026-07-15 amendment), whereas the first-token and total overrides do not. All three fall back to their global defaults when unset.

[[providers]]
name = "ollama-local"
kind = "ollama"
base_url = "http://localhost:11434"
first_token_timeout_ms = 60000
total_timeout_ms = 120000
connect_timeout_ms = 10000  # optional: own client, relaxed connect deadline

[[providers.models]]
id = "local-llama"
upstream_id = "llama3.2"
capabilities = ["chat"]

[[providers.models]]
id = "nomic-embed"
upstream_id = "nomic-embed-text"
capabilities = ["embed"]

azure

• kind: azure · capabilities: chat, embed. Reuses the OpenAI JSON schema verbatim; only the URL, auth, and routing differ from openai.
• Auth: api_key_env, sent as the api-key header (never a bearer token).
• base_url: required - your Azure resource endpoint, e.g. https://<resource>.openai.azure.com (no shared public default, every resource is operator-specific).
• api_version: optional - pins the Azure API version sent as the api-version query parameter on every request (issue #65). For back-compat the older form still works: append ?api-version=YYYY-MM-DD to base_url. Precedence: the explicit api_version field wins over a base_url query string, which wins over LUMEN’s pinned built-in default (see the azure module doc comment for the exact value). Any query parameters on base_url other than api-version are ignored when building request URLs. api_version is azure-only: setting it on any other kind is rejected at boot.
• Deployment routing: Azure routes by URL path (/openai/deployments/{deployment}/...), not by the model field in the body. Set each model’s upstream_id to the Azure deployment name - the same upstream_id mechanism every other kind uses for aliasing already carries it through.
• Embed batch limit: 2048 (same array-size ceiling as the OpenAI embedding models Azure hosts).

[[providers]]
name = "azure-openai"
kind = "azure"
api_key_env = "AZURE_OPENAI_API_KEY"
base_url = "https://my-resource.openai.azure.com"
api_version = "2024-10-21"

[[providers.models]]
id = "gpt-4o"
upstream_id = "my-gpt4o-deployment"   # the Azure deployment name
capabilities = ["chat"]

[[providers.models]]
id = "azure-embed"
upstream_id = "my-embedding-deployment"
capabilities = ["embed"]

OpenAI-compatible hosts

groq, together, fireworks, deepseek, openrouter, perplexity, xai, deepinfra and huggingface all work the same way: set the kind, point api_key_env at the host’s token, and (optionally) override base_url. The built-in default base URL is used otherwise.

[[providers]]
name = "groq"
kind = "groq"
api_key_env = "GROQ_API_KEY"
[[providers.models]]
id = "fast"
upstream_id = "llama-3.3-70b-versatile"
capabilities = ["chat"]

huggingface

The OpenAI-compatible Inference router (https://router.huggingface.co/v1), distinct from the self-hosted tei kind. api_key_env holds a Hugging Face token; upstream_id is a routed model id (often owner/model:provider).

[[providers]]
name = "hf"
kind = "huggingface"
api_key_env = "HF_TOKEN"
[[providers.models]]
id = "qwen"
upstream_id = "Qwen/Qwen2.5-72B-Instruct"
capabilities = ["chat"]

cloudflare

Cloudflare Workers AI. Chat and embeddings go through its OpenAI-compatible endpoint; reranking (bge-reranker-* models) goes through Workers AI’s own native POST /ai/run/{model} endpoint instead, since it is not part of the OpenAI-compatible surface - one [[providers]] entry serves all three capabilities against the same base_url. base_url is required because it embeds your account id; api_key_env holds a Cloudflare API token.

The native rerank request is { query, contexts: [{ text }, ...], top_k } (top_n is sent as top_k); the response is Cloudflare’s standard { result: { response: [{ id, score }, ...] }, success, errors } envelope, with id mapped back onto the original document index. Workers AI reports no token usage for this model; LUMEN derives a local estimate per ADR 003.

[[providers]]
name = "cf"
kind = "cloudflare"
api_key_env = "CLOUDFLARE_API_TOKEN"
base_url = "https://api.cloudflare.com/client/v4/accounts/YOUR_ACCOUNT_ID/ai/v1"
[[providers.models]]
id = "cf-llama"
upstream_id = "@cf/meta/llama-3.1-8b-instruct"
capabilities = ["chat"]
[[providers.models]]
id = "cf-rerank"
upstream_id = "@cf/baai/bge-reranker-base"
capabilities = ["rerank"]

vllm

Any self-hosted OpenAI-compatible server (vLLM, llama.cpp --api, LM Studio, SGLang, LocalAI). base_url required, API key optional. For Ollama, prefer the native ollama kind (chat + embed, see its section above); its OpenAI-compatible endpoint (http://localhost:11434/v1) also works under this kind, but you lose the native embed path and the /api/version health probe.

[[providers]]
name = "local"
kind = "vllm"
base_url = "http://localhost:8000/v1"
[[providers.models]]
id = "local-llama"
upstream_id = "meta-llama/Llama-3.1-8B-Instruct"
capabilities = ["chat", "embed"]

OpenAI chat extras on translated providers

OpenAI-compatible kinds forward every unmodeled request field verbatim, so response_format, seed, logprobs, top_logprobs, logit_bias, parallel_tool_calls, frequency_penalty and presence_penalty simply work there. The translated chat kinds rebuild the upstream request field by field, so each of these is either mapped onto a native equivalent or explicitly unsupported (issues #72, #91) - never silently lost:

¹ {"type": "json_object"} becomes generationConfig.responseMimeType: "application/json"; {"type": "json_schema"} additionally carries json_schema.schema as generationConfig.responseSchema, with JSON Schema keywords Gemini’s OpenAPI-subset schema rejects (additionalProperties, $schema) stripped recursively. Note that additionalProperties: false is therefore dropped, not enforced, on Gemini: the model may still emit extra keys (each strip is logged at debug level).

² Cohere v2 has no separate json_schema type: OpenAI’s {"type": "json_schema", "json_schema": {"schema": ...}} collapses onto Cohere’s {"type": "json_object", "json_schema": <schema>}; json_object and text pass through as-is.

³ Cohere v2 does accept a logprobs flag upstream, but its response shape is not translated back to OpenAI’s (and top_logprobs rides that same response shape), so the gateway treats both as unsupported rather than returning a malformed response.

⁴ parallel_tool_calls: false becomes Anthropic’s tool_choice.disable_parallel_tool_use: true (defaulting the choice to auto when the request carries tools but no explicit tool_choice); true is the default on both sides and needs no wire field.

What happens to an unsupported field depends on the provider’s strict flag (the same switch Ollama uses for embeddings dimensions, issue #25):

• strict = false (default): the field is dropped and a debug-level log line names the provider and field.
• strict = true: the request is rejected before any upstream call with an honest 400 (LM-1001) naming the field and provider - never a misleading 5xx.

[[providers]]
name = "anthropic"
kind = "anthropic"
api_key_env = "ANTHROPIC_API_KEY"
strict = true   # reject response_format/seed/logprobs instead of dropping

An unrecognised response_format shape (an unknown type value) is dropped with a debug log on the mapping providers, matching how unknown tool_choice shapes are handled: dropped, not guessed.

Vision (image input)

POST /v1/chat/completions accepts OpenAI’s content-parts message shape, so a user message can carry text and image parts in one array:

{
  "model": "gpt-4o",
  "messages": [{
    "role": "user",
    "content": [
      { "type": "text", "text": "What is this?" },
      { "type": "image_url", "image_url": { "url": "data:image/png;base64,iVBORw0KG..." } }
    ]
  }]
}

image_url.url is either a data:<media-type>;base64,<payload> URI (inline bytes) or a remote http(s) URL.

Per-model opt-in. A model only accepts image parts once its config declares the image modality (default is ["text"]):

[[providers.models]]
id = "gpt-4o"
capabilities = ["chat"]
modalities = ["text", "image"]   # opts this model into vision

GET /v1/models reflects this back as "modalities": ["text","image"] per model. Sending an image part to a model whose modalities lack "image" is rejected with LM-2003 (400, see docs/errors.md) before any upstream call.

Which kinds support it:

Never-fetch rule. LUMEN never dereferences a user-supplied image URL itself - doing so would be an SSRF vector (the gateway could be aimed at internal addresses) and would violate the streaming/latency pillar. A remote http(s) image_url is only ever forwarded to a provider that fetches it itself (OpenAI, Anthropic); Gemini’s inline_data field takes only inline bytes, so a remote URL routed to Gemini is rejected with LM-2004 (400) instead of the gateway silently fetching it on the caller’s behalf.

The LM-2004 pre-flight check inspects the primary provider of the model’s fallback chain. In the uncommon case where the primary accepts remote URLs (e.g. OpenAI) but a Gemini model is configured as a fallback, a request with a remote image URL passes pre-flight and, only if the primary then fails over to Gemini, surfaces as an upstream LM-3002 (502) - the gateway still never fetches the URL. Configure inline data: URIs when a Gemini fallback is in play.

Provider-native image sources (issue #12). Two provider-native reference forms are recognised in the image_url.url field, for callers whose images are already uploaded to the provider:

A provider-native reference routed to a model whose primary provider is not the reference’s own provider is rejected pre-flight with LM-2008 (400) - an honest client error instead of a confusing upstream failure. A Gemini Files API URI is also an https:// URL, but it is exempt from the LM-2004 remote-URL check: it is not a URL the provider would have to fetch, and its routing verdict belongs to the LM-2008 check.

gs:// caveat. The gateway forwards a gs:// URI to Gemini verbatim, but the Gemini Developer API (generativelanguage.googleapis.com, the default base_url of the google kind) documents fileData.fileUri for its own Files API URIs; Cloud Storage gs:// URIs are a Vertex AI capability. Against the default endpoint a gs:// reference is passed through and will be rejected by the upstream (surfacing as an upstream error naming google). It is still parsed and forwarded because the reference form is Gemini-native (mismatch routing stays an honest LM-2008), base_url may point at a Vertex-compatible gateway, and the upstream - never the gateway - is the authority on which URI forms it accepts. Upload via the Gemini Files API and pass the returned URI when targeting the Developer API.

For the mime type of a fileData part: it is included only when it can be confidently inferred from the URI’s file extension (.png, .jpg, …); otherwise it is omitted rather than guessed. Files API URIs carry no extension, and Gemini already knows the mime type recorded at upload time.

Accounting. Upstream-reported usage is authoritative and already folds in image tokens. When an upstream reports no usage at all, the local estimation fallback counts each image content part with a flat per-image heuristic (85 tokens at "detail": "low", 765 tokens otherwise) rather than counting it as zero, and the response is still flagged "estimated": true - see the ADR 003 addendum.

Fallbacks across providers

Any model can name an ordered list of fallbacks - models that back it when its provider exhausts retries or its circuit is open. Each fallback must exist and serve every capability of the model it backs (validated at boot), which lets you survive a single-vendor outage by spanning providers:

[[providers.models]]
id = "gpt-4o"
capabilities = ["chat"]
fallbacks = ["claude-sonnet-4-5"]     # different vendor, same capability

See docs/adr/005-resilience-execution.md for the resolution and circuit-breaker details, and config.example.toml for a fully worked multi-provider setup including a three-vendor rerank fallback chain.