Pattern: outbox pattern
This course covers the outbox pattern in a Rails application that uses Solid Queue. The queue lives in a separate database, so saving a record and enqueuing a job are two commits that cannot be atomic. The lessons explain why no ordering of the two writes is safe. They show an outbox table that stores the message in the same transaction as the data, and a relay that drains it with FOR UPDATE SKIP LOCKED. They also cover at-least-once delivery, duplicates, adoption cost and when to skip the pattern.
What you'll learn
- ✓Why two writes to two databases cannot be made atomic
- ✓Why retries miss the crash window and after_commit only narrows it
- ✓How to design an outbox table written in the same transaction as the data
- ✓How a relay claims rows safely with FOR UPDATE SKIP LOCKED
- ✓At-least-once delivery and the idempotency it requires from consumers
- ✓When periodic reconciliation is a better choice than an outbox
The write that isn't in your transaction
This application enqueues background work in a handful of places: its mailers, notification emails, partner webhooks, and the partner icon fetch. The simplest of them is app/services/admin/partners/save_partner_service.rb, shown here in full:
Read the full lesson free →Curriculum
7 lessonsThe write that isn't in your transaction
Shows how saving a partner and enqueuing its icon job land in two databases with separate commits.
Why two writes cannot be atomic
Explains why no ordering of two writes is safe and why after_commit and retries leave the window open.
The outbox table
Sketches an outbox table and writes each message in the same transaction as the data it describes.
Draining the outbox
Builds a relay that claims rows with FOR UPDATE SKIP LOCKED and explains why crashes cause redelivery.
At-least-once, and what it costs
Covers at-least-once delivery, when duplicates appear, and why every consumer must be safe to run twice.
An outbox in this app: what it would take
Works out the migration, service change and relay this app would need, and why after_commit changes nothing here.
When not to reach for an outbox
Argues that most apps, this one included, should accept the gap or reconcile periodically instead of building an outbox.