Tombstone

Tombstone is a self-hosted production intelligence layer for feature flags at scale — built to answer the question that every SRE asks at 2am but no flag system answers: which flag caused this incident, and can I roll it back safely right now?

At its core, Tombstone treats flags as causal agents in a live production system, not boolean configuration. It combines an 8-service polyglot backend (Go for performance, Python for ML, TypeScript for the management UI) with a circuit-breaker auto-rollback engine, a causal dependency graph for "What Changed?" incident correlation, and a Merkle-linked audit trail connected to Sigstore Rekor for SOC2-grade immutability.

• Circuit-breaker auto-rollback — 5%+ error rate over 100 requests in 10s auto-disables the flag; no human in the loop
• Blast-radius gating — BLOCKED / HIGH / MEDIUM / LOW tiers; BLOCKED changes require a 10-char justification
• 3-model ensemble anomaly detection — Z-score + Isolation Forest + EWMA with 2/3 vote, eliminating false positives
• Thompson Sampling + LinUCB bandit for ML-driven rollout recommendations
• Causal dependency graph — Redis sorted sets (O(log n) updates), daily rebuild at 02:00 UTC
• Merkle audit chains — SHA-256 coverage of every state transition, Rekor transparency log submission
• WASM evaluation engine (@flagmind/eval) — zero-dependency, runs on Cloudflare Workers

Inspired by Knight Capital's $440M flag incident (2012). 775 commits, 8 services, full Kubernetes operator.

The production resilience work hardened Tombstone against distributed failure modes discovered during real-world pressure testing: resilient inter-service HTTP with failsafe-go retry + jitter, distributed Redis Lua rate limiting (shared across replicas), dependency-aware /readyz probes, idempotency keys on mutation endpoints, Redis Streams dead-letter queue, and multi-replica-safe scheduler via FOR UPDATE SKIP LOCKED.

Architecture

Tombstone runs as 8 polyglot services split by workload. Go handles the request path — flag-api (mutations, scheduling, Rekor submission), gateway (stream hub, dead-letter queue), evaluator (circuit breakers, blast-radius scoring) — suited to latency-sensitive eval. Python's intelligence service owns the ML surface: the anomaly ensemble, the LinUCB/Thompson Sampling rollout bandit, CUPED experiment analysis, and the causal-graph builder. TypeScript covers workspace-dashboard (the management UI, including the client-side blast-radius justification gate) and @tombstone/eval, a zero-dependency WASM engine that hand-ports MurmurHash3 and FNV-32a so it runs identically across five JS runtimes, Cloudflare Workers included.

Each Go service carries its own copy of a resilient HTTP client (retry, backoff, a service-level circuit breaker via failsafe-go) rather than sharing one module — deliberate duplication across 6+ services, matching the repo's existing OpenTelemetry-setup convention. Redis holds per-flag circuit-breaker state, the causal dependency graph, and the gateway's dead-letter queue; Postgres holds the audit log and scheduled-change queue, claimed via FOR UPDATE SKIP LOCKED so replicas never double-fire.

How It Works

The evaluator exposes blast-radius scoring as a standalone query at GET /api/v1/blast-radius: blast.Calculator.Compute() estimates traffic-percentage exposure from the proposed rollout, cross-references the last 30 days of the audit log for flags changed together (dependent-flag correlation), and factors in historical error-rate delta to produce a LOW / MEDIUM / HIGH / BLOCKED score. Scheduled changes wait in Postgres until the scheduler claims them via FOR UPDATE SKIP LOCKED.

Live, the evaluator's Redis-backed circuit breaker watches error rate per environment: 100+ requests in 10 seconds above 5% errors flips the flag off, no human involved. In parallel, the anomaly ensemble — Z-score, Isolation Forest, EWMA — scores metrics at 10s/60s/5m granularities and only fires on a 2-of-3 vote, limiting single-model false positives. Rollout recommendations come from a LinUCB bandit persisting per-arm matrices to Redis, falling back to Thompson Sampling below 50 observations per flag-environment pair.

Every state transition is Merkle-chained (sha256 of id, event type, actor, prev/new state, timestamp) and submitted asynchronously — fail-open — to a Sigstore Rekor log. A daily 02:00 UTC job rebuilds the causal graph in Redis sorted sets, which backs the dependency-graph view (GET /api/v1/graph/dependencies); the "What Changed?" incident timeline is sourced separately, straight from the audit log (GET /api/v1/audit).