Performance baseline

LUMEN’s first pillar is performance: < 1 ms added p99 off-network, ~15 MB idle RAM, streaming that doesn’t re-serialize. This document records how those promises are measured, the numbers obtained, and how to reproduce every figure. Where a target is not fully measured in a given environment, the gap is stated honestly rather than papered over.

Methodology

Two layers, because “added latency” has two very different scales:

1. In-process overhead (cargo bench) - the CPU work the gateway adds per request, with no sockets involved: the resilience executor wrapping a provider call (circuit-breaker admit → retry loop → per-attempt timeout) plus the JSON (de)serialization the OpenAI surface does. This is the honest measure of “added latency off-network”: it excludes the upstream and the network entirely. Criterion, warmed, reports median with a 95 % CI.

2. End-to-end head-to-head (bench/, docker-compose + k6) - LUMEN and LiteLLM both proxying the same zero-latency mock upstream, driven by k6. Added latency = gateway percentile − direct-to-mock percentile. This is the number a user feels; it includes one extra localhost hop. Provided as a reproducible harness (see bench/README.md). Reported at two marks per target: total request time (http_req_duration) and time to first byte (http_req_waiting: request fully written → first response byte).

3. Streaming time to first bit (cargo bench -p server --bench stream_ttfb) - the one latency k6 cannot see: how much later a client receives the first SSE chunk of a stream: true response because the gateway sits in the middle. Real sockets, full LUMEN stack (axum → router → OpenAI-kind provider), instant mock upstream; the same request is timed direct-to-upstream and via-gateway, and the difference between the two distributions is the gateway’s added streaming TTFB. The companion integration test (tests/chat.rs::first_stream_chunk_reaches_the_client_before_the_upstream_finishes) proves the first frame is forwarded while the upstream is verifiably still mid-stream (gated tail, no timing races), so this bench measures eager forwarding, not buffer-then-flush.

Environment of the recorded run

Numbers are hardware-specific; re-run the commands on your target to get yours.

Results - in-process overhead (measured here)

Total added CPU per non-streaming chat request ≈ 3.2 µs (executor + parse + serialize). Streaming passthrough adds even less per chunk: it forwards upstream Bytes verbatim with no per-chunk serde (ADR 004), so the per-chunk cost is a bounded copy plus the [DONE]/heartbeat scan, not a deserialize.

Streaming time to first bit (measured here)

Recorded with cargo bench -p server --bench stream_ttfb in the same environment as above: the span from dispatching a stream: true chat request to reading the first bytes of the SSE body, over real loopback sockets, against an instant mock upstream.

Added streaming TTFB ≈ 97 µs median (~0.1 ms): the extra wait before a client sees the first streamed token because LUMEN sits in the middle. That buys one full extra HTTP hop (accept, parse, route, provider request build, upstream connect-pooled call, headers + first-frame forward), measured end to end, and still lands an order of magnitude inside the < 1 ms pillar. The companion integration test (methodology point 3 above) guarantees the number means what it says: the first frame is forwarded while the upstream is still mid-stream, never buffered until end-of-stream.

Idle memory & binary size (measured here)

The Docker image was built from Dockerfile and smoke-tested: docker run -v config.toml -e OPENAI_API_KEY … answers /health 200 and /v1/models. At 10.6 MB it is well under the 30 MB image budget (§7.2). The amd64 image is built in CI via buildx (release.yml).

Results - loaded head-to-head vs LiteLLM (recorded baseline)

Recorded by bench/run.sh (see bench/README.md); full raw output is committed at bench/results/20260715T231135Z/.

Time to first byte for the same run (http_req_waiting; derived from the committed raw *.summary.json of that run - the metric was always recorded, its report table was added later, so the run’s own report.md predates it):

TTFB tracks total duration almost exactly in this scenario (non-streaming, tiny mock body: once the first byte is out, the rest follows within microseconds), so it carries the same caveat and the same conclusion: LUMEN delays the start of the response by ~2.6 ms at p50 on this noisy host, LiteLLM by ~317 ms.

RAM under load (docker stats, sampled mid-run): lumen ~7.6 MB, litellm ~1.03 GB.

Environment this specific run was recorded in: Darwin arm64, Docker 29.4.0, k6 v2.0.0, LUMEN at commit 51fc809, LiteLLM ghcr.io/berriai/litellm:v1.92.0@sha256:9ef6f45bc0104940571765e610c52a1d761b5ec85efcd193795281086ee61277, mockserver 5.15.0@sha256:0f9ef78c94894ac3e70135d156193b25e23872575d58e2228344964273b4af6b.

Caveat, stated honestly: this run was recorded on a shared development host (not dedicated benchmarking hardware), which is visible in the noisy direct-baseline numbers above (a 0 ms mock should not itself show ~836 ms p99; the mockserver JVM saturates a core by itself, so the direct phase measures host contention as much as transport). Treat the absolute numbers as illustrative, not authoritative. The relative comparison - lumen vs litellm, same mock, same host, same run - is the meaningful part: LUMEN added ~2.5 ms at p50 over direct (9.44 vs 6.91 ms) and its tail stayed below the noisy direct baseline, while LiteLLM’s p50 grew by ~47× (323.75 ms) and its p99 reached 6.5 s. LUMEN sustained ~25× LiteLLM’s throughput at roughly 1/140th the RAM. Re-run bench/run.sh on dedicated hardware for numbers to make capacity decisions on; see “Updating the pinned versions” in bench/README.md for how to refresh this baseline (new pinned image, new results/ directory, update the link above).

Targets

Honest summary: the off-network overhead is measured and is microseconds, comfortably inside the pillar-1 budget. The full loaded head-to-head vs LiteLLM is now a committed, reproducible baseline (pinned versions, one command, recorded result linked above) rather than just a runnable harness; the relative LUMEN-vs-LiteLLM comparison from it is solid, while the absolute and direct-vs-gateway numbers should be treated as this particular (noisy, shared) host’s numbers, not a hardware-independent claim.

Reproducing

# In-process overhead (no Docker needed):
cargo bench -p server --bench gateway_overhead

# Streaming time to first bit, direct vs via-gateway (no Docker needed):
cargo bench -p server --bench stream_ttfb

# Idle RAM + binary size:
cargo build --release -p server --bin lumen
./target/release/lumen --config config.example.toml &   # then: ps -o rss= -p <pid>

# Full head-to-head vs LiteLLM (Docker + k6, one command): see bench/README.md
bench/run.sh