Resilience tuning

LUMEN survives flaky upstreams without becoming flaky itself. A request goes through, in order: retries, then fallback to the next target of a virtual model, then the circuit breaker deciding whether to even try a given provider. None of this touches the database on the request path. Retries and the circuit breaker are always active with sane defaults; fallback exists only where you declare a virtual model, and background health checks are opt-in. All of it lives under [resilience] - see ADR 005 for the design.

Retries

Applied only to retryable upstream failures: 5xx, connect/read timeouts, and 429. A client 4xx is never retried - a fallback provider would reject it too. Backoff is exponential with equal jitter, and honors an upstream Retry-After header as a floor (a Retry-After: 3 guarantees at least a 3-second wait). While streaming, commitment happens at the first content frame, not at the open: after the upstream opens (2xx + headers), the gateway peeks the first frame before committing. An upstream that opens 200 then errors or closes before delivering any content still retries and falls over, charging the circuit breaker like an open failure, instead of surfacing a terminal SSE error frame - so an immediately-dead stream is still retried even though it opened. Once the first content frame reaches the client the request is committed and a later mid-stream error becomes a clean SSE error frame instead of a retry. See ADR 005 (first-frame-peek amendment) and Streaming.

[resilience]
retry_max_attempts = 3   # total attempts per provider incl. the first (>= 1)
retry_base_ms = 200      # base backoff wait after the first failure
retry_max_ms = 5000      # ceiling on the exponential backoff term

Set retry_max_attempts = 1 to disable retries entirely.

Fallback chains

Fallback is declared on a virtual model, not on the foundation model. A fallback virtual model lists targets to try in order, and fallback_on says which failures move on to the next one (default: a 5xx, a 429, a timeout or an open circuit; opt in to context_length and content_filter). Targets can be other virtual models, so chains compose. Chains are validated at boot and on every reload: every target must exist and serve the same capability, with no cycles, so a runtime resolution miss never happens in practice. Whichever foundation model actually served a request is reported in the x-lumen-model-used response header, and the path taken in x-lumen-route, so a caller (and the usage log) can see when a fallback fired.

[[virtual_models]]
id = "gpt-4o"
capability = "chat"
strategy = "fallback"
fallback_on = ["provider_error", "rate_limited", "timeout", "circuit_open"]
targets = [{ model = "openai/gpt-4o" }, { model = "claude-sonnet-4-5" }]

Circuit breaker

Tracked per (provider, model) pair:

circuit_failure_threshold = 5     # consecutive failures that trip it open
circuit_cooldown_ms = 30000       # time spent open before a half-open probe

After circuit_failure_threshold consecutive provider-fault failures, the circuit opens. While open, that link is skipped instantly - no upstream call - straight to the next fallback, or answered with 503 LM-3020 (plus Retry-After set to the cooldown remainder) if none remains. After circuit_cooldown_ms, a single half-open probe decides whether to close again. State is exported as lumen_circuit_state{provider,model} (0 closed, 1 open, 2 half-open).

The three timeouts

A provider’s own connect_timeout_ms override is picked up on hot reload like the other resilience knobs, since the registry rebuilds its clients from the new config.

[resilience]
connect_timeout_ms = 5000
total_timeout_ms = 600000   # 10 minutes

Health checks

Off by default. When health_check_enabled = true, a background task probes, on health_check_interval_ms, every provider that has an explicit base_url (self-hosted TEI/Ollama, or any explicit override) - providers on a built-in vendor URL are never probed and report unknown, since the gateway hardcodes no vendor endpoints. Results are published at GET /health/providers and the lumen_provider_up{provider} gauge. The list follows the live config: a provider added or removed by a hot reload (including one requested by PUT/DELETE /admin/config/providers/{name}) appears as unknown or disappears once that reload has run, and, when health checks were enabled at boot, a new provider with a base_url is probed from the next interval on (health_check_enabled and health_check_interval_ms themselves are read at boot: changing them takes a restart). This is independent of the gateway’s own liveness: GET /health never depends on provider state and does no I/O.

These probes never send provider credentials, so a revoked key still reads up. To check a key on demand, use POST /admin/providers/{name}/check.

health_check_enabled = false
health_check_interval_ms = 30000

How this shapes the error codes you see

Retries, fallback and the circuit breaker all happen before a 3xxx error ever reaches a client - by the time one surfaces, the resilience machinery has already given up. See Error codes: “How resilience shapes these codes” for the full mapping from a given upstream failure to the code you’ll see.