Performance baseline

LUMEN’s first pillar is performance: < 1 ms added p99 off-network, ~15 MB idle RAM, streaming that doesn’t re-serialize. This document records how those promises are measured, the numbers obtained, and how to reproduce every figure. Where a target is not fully measured in a given environment, the gap is stated honestly rather than papered over.

Methodology

Two layers, because “added latency” has two very different scales:

1. In-process overhead (cargo bench) - the CPU work the gateway adds per request, with no sockets involved: the resilience executor wrapping a provider call (circuit-breaker admit → retry loop → per-attempt timeout) plus the JSON (de)serialization the OpenAI surface does. This is the honest measure of “added latency off-network”: it excludes the upstream and the network entirely. Criterion, warmed, reports median with a 95 % CI.

2. End-to-end head-to-head (bench/, docker-compose + k6) - LUMEN and LiteLLM both proxying the same zero-latency mock upstream, driven by k6. Added latency = gateway percentile − direct-to-mock percentile. This is the number a user feels; it includes one extra localhost hop. Provided as a reproducible harness (see bench/README.md). Reported at two marks per target: total request time (http_req_duration) and time to first byte (http_req_waiting: request fully written → first response byte).

3. Streaming time to first bit (cargo bench -p server --bench stream_ttfb) - the one latency k6 cannot see: how much later a client receives the first SSE chunk of a stream: true response because the gateway sits in the middle. Real sockets, full LUMEN stack (axum → router → OpenAI-kind provider), instant mock upstream; the same request is timed direct-to-upstream and via-gateway, and the difference between the two distributions is the gateway’s added streaming TTFB. The companion integration test (tests/chat.rs::first_stream_chunk_reaches_the_client_before_the_upstream_finishes) proves the first frame is forwarded while the upstream is verifiably still mid-stream (gated tail, no timing races), so this bench measures eager forwarding, not buffer-then-flush.

Environment of the recorded run

Numbers are hardware-specific; re-run the commands on your target to get yours.

Results - in-process overhead (measured here)

Total added CPU per non-streaming chat request ≈ 3.2 µs (executor + parse + serialize). Streaming passthrough adds even less per chunk: it forwards upstream Bytes verbatim with no per-chunk serde (ADR 004), so the per-chunk cost is a bounded copy plus the [DONE]/heartbeat scan, not a deserialize.

SystemOne cost scales with the body: a typical request matches chat, while a maximum-size one (2026-09 measurement, noisy host) spends roughly 100 µs in the unavoidable JSON scan and the rest in per-question raw copies and the upstream re-serialization. It stays inside the 1 ms budget; a byte-splice passthrough that would roughly halve the large case is in docs/backlog.md. The large case needs CARGO_PROFILE_RELEASE_STRIP=false to build on macOS 27, where strip = true corrupts proc-macro dylibs (pre-existing, unrelated).

Decisions (ADR 017)

Same machine and toolchain as above (Apple Silicon arm64, macOS, rustc 1.97.0, release profile). Command: cargo bench -p server --bench gateway_overhead -- decisions (add CARGO_PROFILE_RELEASE_STRIP=false on macOS 27). Passthrough is parse + token estimate + one attempt’s clone and encode to the same vendor; cross-vendor is parse + encode to the other vendor’s wire format. “Large” is a big body, not a gateway limit (there is no question-count cap at the edge): 300 questions and 128 KB of state for TypeSafe (same shape as the old SystemOne large case), and 128 questions (chosen for the case) and 128 KB of input for OpenAI.

The final fixes (pre-sized output buffers, the name-only wire_ids fast path, a borrowed structured state in the OpenAI encoder, the question type borrowed at the TypeSafe edge) cut the TypeSafe large passthrough by 9 % (307 to 280 µs) and the OpenAI-to-TypeSafe large translation by 12 % (170 to 150 µs). The TypeSafe-to-OpenAI large translation is flat (414 to 417 µs, within run-to-run noise: a same-session run of the previous commit measured 415.6 µs); its cost is the 300 per-question instruction strings the OpenAI wire needs, not buffer growth. Every case is below the 1 ms budget; the worst is 417 µs. The TypeSafe large passthrough (280 µs) now matches the old systemone_request_pipeline_large (~280 µs), and the small one (3.23 µs) is within 2 % of the old systemone_request_pipeline_small (3.3 µs).

Streaming time to first bit (measured here)

Recorded with cargo bench -p server --bench stream_ttfb in the same environment as above: the span from dispatching a stream: true chat request to reading the first bytes of the SSE body, over real loopback sockets, against an instant mock upstream.

Added streaming TTFB ≈ 97 µs median (~0.1 ms): the extra wait before a client sees the first streamed token because LUMEN sits in the middle. That buys one full extra HTTP hop (accept, parse, route, provider request build, upstream connect-pooled call, headers + first-frame forward), measured end to end, and still lands an order of magnitude inside the < 1 ms pillar. The companion integration test (methodology point 3 above) guarantees the number means what it says: the first frame is forwarded while the upstream is still mid-stream, never buffered until end-of-stream.

Budget accounting per request (measured here)

Recorded 2026-10 with cargo bench -p auth --bench admit_settle (rustc 1.97.0, same machine): one admit plus its settle on a single thread, the in-memory budget accounting every authenticated request pays (ADR 009, and the ADR 015 settled-cost counter that billing reads). Lock-free atomics, no I/O; the database is never touched on the request path.

Well under a microsecond either way: budget enforcement and billing accounting are noise next to the ~3.2 µs of request CPU above. Like the SystemOne bench, it needs CARGO_PROFILE_RELEASE_STRIP=false to build on macOS 27.

The parallel bench guards against false sharing between tenants. Keys are loaded back to back at boot, so before KeyEntry and GroupEntry were aligned to the cache line (128 bytes on x86_64 and aarch64, 64 elsewhere) two neighbouring keys’ hot atomics, or one key’s tail and the next key’s Arc refcount, shared a line: the same bench measured 100.1 ns (96.8 – 103.4 ns), 2.5x the single-thread admit_settle_key of that run (39.3 ns) for keys that never contend logically. Aligned, it is 57.0 ns (-47 %), and the single-thread benches are unchanged within noise. The remaining gap to one thread (about 1.5x) was not investigated; it is not sharing between keys (the entries no longer share a line), and the M1 Pro runs two of the eight threads on efficiency cores. An interleaved A/B (unpadded and padded bench binaries run alternately, 4 rounds, on a heavily loaded machine, load average 54 to 59) agreed: the parallel bench was faster padded in every round (median about 127 ns to 77 ns), and the best single-thread admit_settle_key round was the same either way (38.5 ns unpadded, 38.4 ns padded). The price is padding, paid once per key at load: a key’s allocation grows from 184 to 384 bytes (about 2 MB per 10,000 keys), a group’s to 256 bytes.

Idle memory & binary size (measured here)

The Docker image was built from Dockerfile and smoke-tested: docker run -v config.toml -e OPENAI_API_KEY … answers /health 200 and /v1/models. At 10.6 MB it is well under the 30 MB image budget (§7.2). The amd64 image is built in CI via buildx (release.yml).

Results - loaded head-to-head vs LiteLLM (recorded baseline)

Recorded by bench/run.sh (see bench/README.md); full raw output is committed at bench/results/20260715T231135Z/.

Time to first byte for the same run (http_req_waiting; derived from the committed raw *.summary.json of that run - the metric was always recorded, its report table was added later, so the run’s own report.md predates it):

TTFB tracks total duration almost exactly in this scenario (non-streaming, tiny mock body: once the first byte is out, the rest follows within microseconds), so it carries the same caveat and the same conclusion: LUMEN delays the start of the response by ~2.6 ms at p50 on this noisy host, LiteLLM by ~317 ms.

RAM under load (docker stats, sampled mid-run): lumen ~7.6 MB, litellm ~1.03 GB.

Environment this specific run was recorded in: Darwin arm64, Docker 29.4.0, k6 v2.0.0, LUMEN at commit 51fc809, LiteLLM ghcr.io/berriai/litellm:v1.92.0@sha256:9ef6f45bc0104940571765e610c52a1d761b5ec85efcd193795281086ee61277, mockserver 5.15.0@sha256:0f9ef78c94894ac3e70135d156193b25e23872575d58e2228344964273b4af6b.

Caveat, stated honestly: this run was recorded on a shared development host (not dedicated benchmarking hardware), which is visible in the noisy direct-baseline numbers above (a 0 ms mock should not itself show ~836 ms p99; the mockserver JVM saturates a core by itself, so the direct phase measures host contention as much as transport). Treat the absolute numbers as illustrative, not authoritative. The relative comparison - lumen vs litellm, same mock, same host, same run - is the meaningful part: LUMEN added ~2.5 ms at p50 over direct (9.44 vs 6.91 ms) and its tail stayed below the noisy direct baseline, while LiteLLM’s p50 grew by ~47× (323.75 ms) and its p99 reached 6.5 s. LUMEN sustained ~25× LiteLLM’s throughput at roughly 1/140th the RAM. Re-run bench/run.sh on dedicated hardware for numbers to make capacity decisions on; see “Updating the pinned versions” in bench/README.md for how to refresh this baseline (new pinned image, new results/ directory, update the link above).

Targets

Honest summary: the off-network overhead is measured and is microseconds, comfortably inside the pillar-1 budget. The full loaded head-to-head vs LiteLLM is now a committed, reproducible baseline (pinned versions, one command, recorded result linked above) rather than just a runnable harness; the relative LUMEN-vs-LiteLLM comparison from it is solid, while the absolute and direct-vs-gateway numbers should be treated as this particular (noisy, shared) host’s numbers, not a hardware-independent claim.

Reproducing

# In-process overhead (no Docker needed):
cargo bench -p server --bench gateway_overhead

# Streaming time to first bit, direct vs via-gateway (no Docker needed):
cargo bench -p server --bench stream_ttfb

# Budget admit + settle per request (no Docker needed):
cargo bench -p auth --bench admit_settle

# Idle RAM + binary size:
cargo build --release -p server --bin lumen
./target/release/lumen --config config.example.toml &   # then: ps -o rss= -p <pid>

# Full head-to-head vs LiteLLM (Docker + k6, one command): see bench/README.md
bench/run.sh