LUMEN

Lightweight Unified Model ENdpoint

A universal, self-hostable LLM gateway written in Rust. One OpenAI-compatible endpoint in front of many providers - for chat, embeddings and reranking alike. It is designed to be light, fast and sovereign: a single static binary, zero telemetry, and prompts that are never logged by default.

What’s here

• Getting started - installation, quickstart and configuration basics.
• Chat - completions, streaming, vision and tool calling.
• Embeddings - embeddings, batching and multimodal embeddings.
• Reranking - reranking.
• Operations - token accounting & cost, metrics & dashboards, logging, usage log & multi-tenant metadata, keys, quotas & budgets, resilience tuning, deployment and upgrades.
• Examples - ready-made scenario configs.
• Reference - providers, error codes and the performance baseline.
• Architecture decisions - the ADRs that pin the design.
• Project - backlog and contributing.

The five pillars

Every trade-off is decided in this order:

1. Performance - < 1 ms added latency p99, zero-copy streaming, ~15 MB RAM idle.
2. Sovereignty - zero telemetry, prompts never logged by default, single binary.
3. Robustness - propagated cancellation, backpressure, DB off the request path.
4. Multi-capability - chat + embeddings + rerank are first-class citizens.
5. Token observability - every request of every capability produces a token count: upstream usage when reported, otherwise a local estimate flagged estimated. Never a silent zero. See token accounting & cost.

Want to contribute? Start with the contribution guide.