Kelvran

Kelvran is a self-hosted LLM gateway written in Go, for operators who want one endpoint in front of several model providers, with budgets and cost attribution per agent run. It is solo-maintained under the kelvran GitHub organization.

• Two client wire formats — OpenAI Chat Completions and Embeddings, plus Anthropic's Messages API for the Anthropic SDKs and Claude Code.
• Five provider adapters — OpenAI, Anthropic, Gemini, Bedrock and OpenAI-compatible servers such as vLLM and Ollama.
• Three-layer cache — exact match, light normalization and lexical near-duplicate matching, partitioned per virtual key; hits are never billed.
• Non-configurable gates — near-duplicate hits must also pass entity, number, date, negation, freshness and model checks, and no setting disables them.
• Virtual keys and budgets — Decimal-precision spend (never float64) plus RPM, TPM and tokens-per-day limits, on bbolt for one replica or Redis for several.
• Regex guardrails — scans for email, phone, SSN, IBAN, credit card, IP, secret keys and prompt injection; credential and ID matches return a 400, the rest warn.
• Cost attribution — a GenAI span per chat request carries kelvran.agent_run_id from W3C baggage and kelvran.cost.usd.
• evals CLI — a separate Python tool (not on PyPI, run from source) scoring agent outputs with Wilson intervals and an opt-in two-judge panel.

863 commits on main, 23 gateway releases and 0 stars as of 2026-10-21, OpenSSF Scorecard 8/10 on 2026-10-19; main is ahead of v0.20.0, and the benchmark figures are developer-laptop trend numbers, not production latency.

Architecture

Kelvran is one monorepo with two deployables: gateway/, a single static Go binary with the cache compiled in as a package rather than a network hop, and evals/, a Python CLI. They share only the protobuf contract in api/, generated into both languages and guarded by buf breaking in CI.

Package boundaries are enforced: gateway/.go-arch-lint.yml declares an explicit mayDependOn graph checked in make lint-gateway, and evals mirrors it with import-linter contracts. Identity, budgets, rate limits and the spend ledger each have a stdlib-only core with bbolt or Redis backends; the response cache, prompts and Idempotency-Key replay stay per replica by design. The gateway/compat SDK matrix runs against replaying test upstreams, not live providers. Observability is OpenTelemetry-first, with a protobuf GatewayDecisionEvent log line per request that feeds evals ingest.

The README marks only the Bedrock adapter as having seen live pilot traffic, and the Kubernetes base was applied only to a local kind cluster (2026-09-21), never to a production or EKS cluster. The Redis round-trip reduction and the R19 budget fix are on main but not in v0.20.0, which still has R19: with budget.redis_addr set, a key with budget_usd admits one in-flight request at a time.

How It Works

1. Request in — a client sends a W3C traceparent and optionally baggage: agent_run_id=...; the bearer secret resolves to a virtual key by SHA-256 hash, then its allowlists are checked.
2. Idempotency and prompts — an Idempotency-Key is claimed in a 10-minute in-process window, then server-side prompt templates resolve.
3. Reserve — rate limit and budget are reserved before the upstream call; in Redis mode on main that is two round trips.
4. Cache lookup — L1, L2, then L3; a hit returns at once and is never billed, and concurrent identical misses share one upstream call.
5. Guard and route — pre-call guardrails scan every message role, the router picks a healthy deployment by weighted round-robin, and error-classified fallback_chains retry across models and providers.
6. Finalize — post-call guardrails run (audit-only on streams), the response is cached, and settlement, Decimal cost, the spend ledger entry, the GenAI span and the decision event follow.
7. Evals, offline — evals run scores by exact match, regex or LLM judge, --llm-judge-panel adds a second judge, and evals report --fail-under gates on the Wilson lower bound.