RateCap

RateCap is an open-source rate limiter and load shedder for HTTP services, written as a recreation of the four-tier design Stripe described in a 2017 engineering post. It exposes per-key limits, concurrency caps and priority-aware shedding through Go and Python HTTP clients. It is a one-author project, and its fidelity to Stripe rests on that post and a companion Redis-Lua gist, not on Stripe's production code.

• Token bucket (Tier 1) — a per-key bucket refilled and decremented in one atomic Redis Lua call.
• Concurrency cap (Tier 2) — a per-key in-flight limit with optional Redis-backed bounded queueing, fleet-wide since v2.5.0.
• Fleet shedder (Tier 3) — a global concurrency counter that holds back a configurable share of capacity for critical traffic.
• Sidecar shedder (Tier 4) — an in-flight counter with an optional linear shed ramp that rejects before any network call.
• Shadow mode — per tier or global; a would-be rejection is allowed but counted, so enforcement can be previewed.
• Token-cost refunds — a caller reserves an estimated LLM token cost up front and refunds the unused part.
• Operations — Prometheus metrics, one-hop OpenTelemetry tracing, opt-in mTLS, hot-reloaded config and an admin set-limit lever.

At v2.13.1 (2026-09-02): 347 commits on main, 22 releases, 445 Go test functions (a grep count, not a run), 0 stars and one human author; the benchmark figures are a one-time same-host Apple M4 snapshot of pass-through overhead, not capacity.

Architecture

Seven Go modules share one go.work: proto, core, core integration tests, sidecar, Go SDK, CLI and a sample app. The Python SDK sits outside it, uses only the standard library, and is on PyPI as ratecap (0.1.1); the Go SDK is consumed from the repo path only. The core runs a pipeline of limiters behind a StateStore interface whose only implementation is Redis with three Lua scripts.

Applications talk to the sidecar over plain HTTP. It resolves priority from a request header, then a hot-reloaded critical-routes file, then a sheddable default, and it owns shadow-mode coercion and a negative cache of recently denied keys. Sidecar-to-core calls carry a shared secret, with optional mTLS, and core signs Tier 2 tokens so a forged release is refused. Priority claims are unauthenticated, and every key shares one default rate and burst.

Releases ship as GHCR images, a Helm chart on Artifact Hub, and ratecapctl binaries with cosign-signed checksums. CI runs race-detector tests with a 50% per-module coverage floor, Toxiproxy fault injection and a mutation gate. OpenSSF Scorecard scored 7.2/10 on 2026-10-18, with Code-Review 0 and Branch-Protection 4 because a solo workflow has no required reviews. The latest release is v2.13.1 (2026-09-02); develop is eight commits ahead with unreleased fixes. ARCHITECTURE.md still calls the Tier 2 backlog per-instance, which the code and CHANGELOG contradict.

How It Works

1. The app calls Allow or Acquire on the Go or Python SDK, which sends GET /check to the localhost sidecar with priority and route headers.
2. The sidecar skips Tier 4 for critical requests; the rest pass the in-flight shedder, which returns 503 before any network call, probabilistically once past the ramp start.
3. Core's CheckRateLimit runs the tiers in order. Tier 3, driven by one global concurrency counter in Redis rather than CPU or GC telemetry, gives sheddable traffic a reduced cap of cap * (100 - reserved_critical_pct) / 100, and the first non-ALLOW result short-circuits. In shadow mode a would-be rejection becomes an allow.
4. If Tier 2 is full and queueing is on, the request takes a Redis backlog slot and polls until a slot frees, the maximum wait passes, or the context cancels.
5. The sidecar replies 200, 429 or 503 with rate-limit headers and one signed token per reservation; Ticket.Release posts each back, and Ticket.Refund returns unused token cost.
6. On a Redis error Tier 1 fails open and Tiers 2 and 3 fail closed with HTTP 500; a slot whose release never arrives is reaped after max_request_duration_ms.