Database: change data capture (CDC)
Change data capture turns database writes into a stream of events that other systems can consume. The course starts with the dual write problem and three requirements that pull against each other: miss nothing, preserve order and never break the write path. It compares four techniques: polling a timestamp, triggers, an outbox and log-based capture. Later lessons cover the holes in polling, at-least-once delivery, idempotent consumers, schema drift and backfills, and what one Rails application would choose. It ends with an idempotent consumer challenge.
What you'll learn
- ✓Why publishing an event beside a write is a dual write
- ✓How polling, triggers, an outbox and log-based capture compare in completeness and cost
- ✓The four ways polling a timestamp loses events
- ✓At-least-once delivery and how to build idempotent consumers
- ✓How row-level capture couples consumers to your schema, and how outbox payloads act as versioned contracts
- ✓How to combine a backfill and a change stream without leaving a gap
When something else needs to know
Change data capture is the practice of turning writes to your database into a stream of events other systems can consume. The name describes the mechanism, and the problem it solves is older than the name.
Read the full lesson free →Curriculum
7 lessonsWhen something else needs to know
Why publishing an event beside a database write is a dual write, and the three requirements capture must meet.
Four ways to capture
Compares polling, triggers, an outbox and log-based capture by deletes, extra writes, ordering and setup cost.
The holes in polling
The four ways polling loses events: deletes, shared timestamps, intermediate states and tables without updated_at.
At-least-once and idempotency
Why delivery is at-least-once, and how idempotent consumers, unique constraints and versions handle duplicates and ordering.
The schema that keeps moving
How column renames break row-level consumers, why outbox payloads act as versioned contracts, and how to backfill safely.
What this application would do
Ranks the four techniques for this Rails application and explains why building nothing new fits today.
Putting it together
Recaps the course and sets a challenge: implement apply_changes, an idempotent consumer applying events in version order.