odin
2026 · A durable workflow engine in Python: a three-node cluster where any process can be killed at any line and no run is lost, with live chaos every ten minutes.
odin runs long-lived business processes, like a checkout or a ride, as plain async Python. Every step is written to an event history in PostgreSQL, so any process can be killed at any line: another node takes over its shards within seconds and carries on where it left off, without repeating a finished step's side effects.
It's built from scratch rather than on a workflow library: sharded ownership with lease fencing, a transactional outbox into RabbitMQ, deterministic replay, admission control, and a chaos runner that breaks the cluster on a schedule. The public site shows it all live: start an order, follow each of its steps, and watch it survive the next fault.
- Python
- PostgreSQL
- RabbitMQ
- FastAPI
- Next.js
- TypeScript
- Docker
- GitHub Actions
Highlights
- Workflows are plain async Python functions, replayed deterministically from their history: the worker drives the coroutine itself, with no event loop, so a run survives any worker or engine being killed mid-step.
- Three engine nodes split 64 shards by rendezvous hashing, with 10-second leases; every write is fenced in PostgreSQL (FOR KEY SHARE against the takeover's FOR UPDATE), so a node that lost its lease can never commit.
- History, timers and outgoing tasks commit in one transaction; a transactional outbox publishes the tasks to RabbitMQ quorum queues, and a staleness check before every task drops the duplicates at-least-once delivery brings.
- A chaos runner kills, pauses and partitions nodes, cuts the broker and the database, and injects payment faults, then an invariant checker proves no run was lost, no order charged twice and no stale owner wrote. A 5-minute smoke suite: 298 runs, 0 violations, shards taken over in about 9.5 s.
- A 1-hour soak at one run a second with payment faults injected: 3,626 runs, every one closed by its deadline, 86 retries absorbed by idempotency keys, and no order charged twice.
- Admission control with per-origin quotas, a queueing-delay limit and backlog caps; Prometheus, Grafana and OpenTelemetry traces that follow a run across the outbox; a live Next.js dashboard streamed over SSE by a FastAPI service; 270 unit tests, including Hypothesis property tests of replay, plus integration and end-to-end suites against the real services.