Metrics & dashboards
GET /metrics exposes every gateway metric in Prometheus text exposition
format. It is unauthenticated by design - restrict it at the network layer
(firewall, reverse proxy, service mesh) rather than expecting the gateway
to gate it. See SECURITY.md.
Metrics reference
| Metric | Labels | Meaning |
|---|---|---|
lumen_tokens_total | capability, model, provider, direction, estimated | Tokens processed, cumulative. direction is input/output; estimated is true/false (ADR 003). |
lumen_tokens_estimated_total | none | Subset of the above that was locally estimated rather than upstream-reported. |
lumen_token_breakdown_total | capability, model, provider, kind | Upstream-reported token breakdown, a subset of lumen_tokens_total: kind is cached, reasoning or cache_write (issue #99). Never estimated. See Token accounting. |
lumen_rerank_search_units_total | model, provider | Rerank search units, upstream-reported when available. |
lumen_media_total | capability, model, provider, media_type | Media items (images, …) processed - a billing dimension alongside tokens (M9). |
lumen_media_bytes_total | capability, model, provider, media_type | Decoded media bytes processed (M9). |
lumen_virtual_model_requests_total | virtual_model, model_used | Requests served through a virtual model (ADR 014), by the virtual id the client sent and the foundation model that served it. Counted for served requests; a direct call to a foundation model is not counted. Virtual ids are operator-defined, so cardinality is bounded. |
lumen_http_request_duration_seconds | method, path, status | Wall time of every HTTP request, including /health and /metrics. path is the matched route template, never the raw URI. Streaming responses count time-to-response-headers. |
lumen_request_duration_seconds | capability, model, provider, status | End-to-end latency of one accounted API call. For streaming chat this covers the full stream, recorded when accounting closes. |
lumen_circuit_state | provider, model | Circuit-breaker state: 0 closed, 1 open, 2 half-open. |
lumen_provider_up | provider | Background health-probe result: 1 up, 0 down (absent = unknown / not probed). |
lumen_usage_log_dropped_total | none | Usage-log entries dropped because the logging channel was full. |
lumen_metadata_rejected_total | none | x-lumen-metadata headers dropped as malformed or out of bounds. |
lumen_config_reloads_total | none | Successful configuration hot reloads. |
lumen_config_reload_failures_total | none | Configuration reloads rejected as invalid; the previous config kept serving. |
lumen_webhook_queued_total | none | Webhook events accepted into the outbound queue (ADR 011), across every configured event kind - key lifecycle included, not only the budget ones. The whole lumen_webhook_* family appears the first time a webhook is enabled, not at boot. |
lumen_webhook_sent_total | none | Webhook events the receiver acknowledged with a 2xx. |
lumen_webhook_dropped_total | none | Webhook events dropped because the queue was full. |
lumen_webhook_retries_total | none | Webhook delivery attempts that failed and were retried with backoff. |
lumen_webhook_dead_total | none | Webhook events abandoned after exhausting their retry budget, or permanently rejected by the receiver. |
lumen_webhook_delivery_seconds | none | Wall time of a single webhook delivery attempt. |
lumen_usage_events_pending | none | Billing usage events in the outbox and not yet acknowledged (ADR 015). Only registered when [usage_events] is configured. See Lab integration. |
lumen_usage_events_oldest_pending_seconds | none | Age of the oldest unacknowledged billing usage event, in seconds (0 when none). |
lumen_usage_events_delivered_total | none | Billing usage events the control plane acknowledged in accepted. |
lumen_usage_events_failed_total | reason | Billing usage events whose delivery attempt failed, one increment per event; they stay pending and are retried; see lumen_usage_events_dropped_total for the one exception. not_accepted counts only the events that will be retried: an event dropped on that attempt counts in lumen_usage_events_dropped_total instead. reason is connect, timeout, auth, status, malformed, not_accepted or store. |
lumen_usage_events_dropped_total | none | Billing usage events the control plane kept out of accepted for 24 h, counted from its first 2xx answer that left the event out, dropped with an error log naming the event id (platform contract v2). A skip after a long outage starts the clock; it never drops on its own. The row stays in usage_outbox with dropped_ms set until the 7-day purge. An unreachable or failing control plane never drops an event. |
lumen_tokens_total, lumen_rerank_search_units_total, lumen_media_total
and lumen_media_bytes_total also gain one extra label per key listed in
telemetry.metadata_labels - see
Usage log & multi-tenant metadata for the allowlist
mechanics and a multi-tenant query example.
Per-request cost is not a Prometheus series: it is a usage_log-only
figure, computed from the cost_per_1m_* / cost_per_1k_searches prices in
config. See Token accounting & cost.
What to alert on
Several failure modes are deliberately silent on the request path: the gateway sheds work instead of failing requests, and the only trace is a counter. “The drop is visible in your dashboards” is only true if something watches these:
| Alert condition | Meaning |
|---|---|
increase(lumen_usage_log_dropped_total[5m]) > 0 | Usage-log rows are being shed under pressure: token accounting is incomplete for those requests. Raise usage_channel_capacity or investigate DB write latency. |
increase(lumen_config_reload_failures_total[15m]) > 0 | A config reload was rejected; the old config keeps serving. The deploy that “went out” did not. |
increase(lumen_webhook_dropped_total[5m]) > 0 | The webhook queue is full and signals are being shed: a billing backend relying on budget.threshold for auto-recharge is running blind, and key-lifecycle events are being lost too. Raise channel_capacity or fix a slow receiver. |
increase(lumen_webhook_dead_total[15m]) > 0 | Events are being abandoned - the receiver is down past the retry budget, or rejecting deliveries outright (check for a signature mismatch). Reconcile through GET /admin/usage/export until it recovers. |
lumen_usage_events_oldest_pending_seconds > 900 for 5m | Billing usage events are not reaching the control plane (unreachable, rejecting, or skipping them). Serving is unaffected, but every bill is delayed; check lumen_usage_events_failed_total by reason. See Lab integration. |
increase(lumen_usage_events_dropped_total[1h]) > 0 | A reachable Lab kept leaving a usage event out of accepted for 24 h and the gateway dropped it: that usage was never billed. Reconcile it by hand before the 7-day purge, see the runbook in Lab integration. |
lumen_circuit_state == 1 for 2m | A provider/model circuit is open: calls are short-circuited to fallbacks (or failing). |
lumen_provider_up == 0 for 5m | A background health probe cannot reach a provider (only exported for probed providers). |
5xx share of lumen_request_duration_seconds_count > 5% | Upstream or gateway failures; 499 (client cancelled) is deliberately outside the 5xx class and does not count. |
The monitoring rig ships these as ready-made Prometheus rules in
monitoring/prometheus/alerts.yml -
copy them into your own Prometheus and tune the windows. Routing them to a
pager needs an Alertmanager, which the rig intentionally leaves out.
See it live: the monitoring rig
monitoring/ is a one-command Docker Compose stack that runs the gateway
against real providers with Prometheus scraping it and a pre-provisioned
Grafana dashboard covering every metric above: token rates by
provider/model/capability/direction, rerank search units, media accounting,
circuit-breaker state and the internal counters. ./smoke.py exercises
chat, streaming, embeddings and rerank per provider and asserts a non-zero
token count on each; ./traffic.py generates sustained randomized traffic,
tagged with multi-tenant metadata, so the dashboard has something to show.
It is the fastest way to see all of this live - see monitoring/README.md.