Agent Frameworks & Infrastructure
Tombstone
Production intelligence layer for 5,000+ feature flags.
4 min read · 722 words
Tombstone is a self-hosted production intelligence layer for feature flags at scale — built to answer the question that every SRE asks at 2am but no flag system answers: which flag caused this incident, and can I roll it back safely right now?
At its core, Tombstone treats flags as causal agents in a live production system, not boolean configuration. It combines a polyglot backend (Go for performance, Python for ML) and a TypeScript management UI with a circuit-breaker auto-rollback engine, a causal dependency graph plus a "What Changed?" incident timeline, and a Merkle-linked audit trail with opt-in Sigstore Rekor submission.
- Circuit-breaker auto-rollback — an error rate above 5% over at least 100 requests in 10s auto-disables the flag; no human in the loop
- Blast-radius gating — BLOCKED / HIGH / MEDIUM / LOW tiers; BLOCKED changes need a 10-character justification in the dashboard
- 3-model ensemble anomaly detection — Z-score + Isolation Forest + EWMA with 2/3 vote, limiting single-model false positives
- Thompson Sampling + LinUCB bandit for ML-driven rollout recommendations
- Causal dependency graph — Redis sorted sets (O(log n) updates), daily rebuild at 02:00 UTC
- Merkle audit chains — keyed HMAC-SHA256 chain over every audit entry; opt-in Rekor transparency-log submission for flag changes
- Evaluation engine (
@tombstone/eval, 0.1.0, not on npm) — zero-dependency TypeScript, no WebAssembly shipped, built for Cloudflare Workers
Inspired by Knight Capital's $440M flag incident (2012). 1,034 commits on main and 0 stars as of 2026-10-21, latest release v2.0.3; eight services in the tree, seven of them active (gitops-sync is deprecated in favour of the v1alpha1 Kubernetes operator).
The v1.2.0 resilience work (2026-07-05) hardened Tombstone against distributed failure modes: resilient inter-service HTTP with failsafe-go retry + jitter, distributed Redis Lua rate limiting (shared across replicas), dependency-aware /readyz probes, idempotency keys on mutation endpoints, Redis Streams dead-letter queue, and multi-replica-safe scheduler via FOR UPDATE SKIP LOCKED.
Architecture
Tombstone runs as eight polyglot services split by workload (gitops-sync is deprecated in favor of the tombstone-operator). Go handles the request path — flag-api (mutations, scheduling, Rekor submission), gateway (stream hub, dead-letter queue), evaluator (circuit breakers, blast-radius scoring) — suited to latency-sensitive eval. Python's intelligence service owns the ML surface: the anomaly ensemble, the LinUCB/Thompson Sampling rollout bandit, CUPED experiment analysis, and the causal-graph builder. TypeScript covers workspace-dashboard (the management UI, including the client-side blast-radius justification gate) and @tombstone/eval, a zero-dependency engine that hand-ports MurmurHash3 and FNV-32a so it can run in five JS runtimes, Cloudflare Workers included; CI tests it under Node 22 only.
Six of the seven Go services (one of them the deprecated gitops-sync) carry their own copy of a resilient HTTP client (retry, backoff, a service-level circuit breaker via failsafe-go) rather than sharing one module — deliberate duplication, matching the repo's existing OpenTelemetry-setup convention. Redis holds per-flag circuit-breaker state, the causal dependency graph, and the gateway's dead-letter queue; Postgres holds the audit log and scheduled-change queue, claimed via FOR UPDATE SKIP LOCKED so replicas never double-fire.
How It Works
The evaluator exposes blast-radius scoring as a standalone query at GET /api/v1/blast-radius: blast.Calculator.Compute() takes the proposed rollout percentage as the exposure, reads 24 hours of per-flag, per-environment telemetry for volume and error rate, and counts the flags that declare this one as a prerequisite to produce a LOW / MEDIUM / HIGH / BLOCKED score. A rollout of 50% or more is BLOCKED when that error rate exceeds 5% or fewer than 100 evaluations were seen. Scheduled changes wait in Postgres until the scheduler claims them via FOR UPDATE SKIP LOCKED.
Live, the evaluator's Redis-backed circuit breaker watches error rate per environment: 100+ requests in 10 seconds above 5% errors flips the flag off, no human involved. In parallel, the anomaly ensemble — Z-score, Isolation Forest, EWMA — scores metrics at 10s/60s/5m granularities and only fires on a 2-of-3 vote, limiting single-model false positives. Rollout recommendations come from a LinUCB bandit persisting per-arm matrices to Redis, falling back to Thompson Sampling below 50 observations per flag-environment pair.
Every audit entry is Merkle-chained: an HMAC-SHA256 under AUDIT_HMAC_KEY over its fields and the previous hash. Flag changes made through flag-api can also be submitted asynchronously — fail-open, off unless REKOR_ENABLED=true — to a Sigstore Rekor log. A daily 02:00 UTC job rebuilds the causal graph in Redis sorted sets from the audit log, which backs the dependency-graph view (GET /api/v1/graph/dependencies); the "What Changed?" incident timeline is sourced separately, straight from the audit log (GET /api/v1/audit).