Order Processing System

A production-grade, event-driven order-processing backend built as polyglot Go + Python/FastAPI microservices that communicate over gRPC internally behind a REST gateway — the most classic-backend, no-LLM system in the portfolio, and exactly what distributed-systems interviews probe.

The hard parts, up front: Saga-orchestrated distributed transactions (no 2PC) · ACID inventory reservations · a Debezium/WAL CDC transactional outbox for exactly-once-ish delivery · idempotency keys + Dead-Letter Queue · etcd leader election for single-writer coordination · metric-driven read routing across PostgreSQL replicas · Bloom-filter short-circuiting · Redis caching — all fronted by an Envoy L7 proxy with OpenTelemetry / Prometheus / Grafana observability.

• Saga distributed transactions + CDC outbox
• etcd leader election · replica read routing
• DLQ + idempotency · Bloom filters · Redis cache
• Full observability via Envoy L7

Senior-grade distributed-systems work without an LLM in the loop.

Architecture

The system splits into two layers: an external REST gateway sitting behind an Envoy L7 proxy, and internal services that only ever speak gRPC to each other. The service layer is polyglot Go + Python/FastAPI.

• Envoy L7 proxy — external entry point, fronting the REST gateway.
• Go and Python/FastAPI microservices — internal, gRPC-to-gRPC.
• PostgreSQL — system of record, with read replicas for scaled query traffic.
• Redis — cache layer in front of hot read paths.
• etcd — coordination substrate for leader election.
• Debezium, tailing the Postgres write-ahead log, feeding the transactional outbox.
• OpenTelemetry / Prometheus / Grafana — tracing, metrics, and dashboards wrapping every hop.

How It Works

1. A request lands at the Envoy L7 proxy and is routed to the REST gateway.
2. The gateway issues a gRPC call into the owning Go or Python service.
3. For inventory, that service reserves stock inside a single ACID database transaction.
4. Everything past that point runs as a Saga: the saga orchestrator sequences the remaining steps across services, compensating backwards if a later step fails, and each step writes its outbox row in the same local transaction as its business write, so the write to the database and the write announcing that write can never disagree.
5. Debezium reads those outbox rows straight off the Postgres WAL rather than through a separate polling or dual-write path, and publishes them downstream.
6. Consumers process each event idempotently, using idempotency keys to collapse duplicate redeliveries; anything that still fails goes to a Dead-Letter Queue instead of retrying forever or silently vanishing.
7. Writes that require single-writer semantics only proceed once a service instance holds the etcd-elected leader lease, so two instances can't race on the same resource.
8. Reads that aren't on the critical path get sent through metric-driven routing across PostgreSQL replicas, short-circuited by a Bloom filter before ever touching a replica, and served out of Redis when cached.