When something else needs to know

Change data capture is the practice of turning writes to your database into a stream of events other systems can consume. The name describes the mechanism, and the problem it solves is older than the name.

The problem

A row changes, and something outside the application needs to know.

A search index needs reindexing. A data warehouse needs the new figure. A cache in another service is now wrong. An email should go out, an audit trail should record it, a partner API should be notified.

The naive answer is to do all of it inline, in the same request that made the change. That ties your write path to every consumer, and whichever consumer is slow, or down, becomes your problem.

Why this is harder than it sounds

The database already knows every change: its write-ahead log records each one. The difficulty is getting that knowledge out reliably, and "reliably" hides three requirements that pull against each other.

Nothing may be missed. A change that doesn't produce an event is a permanent inconsistency, and nobody finds out until the two systems are compared.

Order must be preserved, at least per row. "Created, then deleted" and "deleted, then created" describe different worlds.

And the capture must not break the write path. A capture mechanism that fails a user's save because a downstream consumer is unavailable has made the system worse.

flowchart TB
  W["a row changes"] --> Q["who else needs to know?"]
  Q --> S["search index"]
  Q --> A["analytics warehouse"]
  Q --> N["notifications"]
  Q --> P["partner systems"]
  W --> R["and none of them may break the write"]

The dual write trap

The tempting shape is to write to the database and publish an event, side by side:

course.update!(active: true)
EventBus.publish(:course_activated, course.id)

These are two systems, and no transaction spans them. The update can commit and the publish can fail, or the publish can succeed and the transaction can roll back. Either way the two disagree, permanently, with no error anywhere.

That's the failure the mid-level course "Pattern: outbox pattern" covers, and change data capture is the family of techniques for avoiding it.

The idea they all share is to derive the events from the database's own record of what happened, rather than publishing them alongside it. If the event comes from the same commit as the data, the two can't disagree.

What this course is about

It covers four techniques, not one, with different costs and different holes, and the skill is choosing between them:

Polling a timestamp column. Triggers writing to a change table. An outbox written in the same transaction. Log-based capture reading the write-ahead log.

Lesson 2 compares them, and the rest of the course is about the ways each one loses events.

Key takeaways

• Change data capture turns database writes into a stream other systems can consume.
• Three requirements pull against each other: miss nothing, preserve order, and never break the write path.
• Publishing an event beside the write is a dual write. The two systems can disagree with no error.
• Every technique here derives events from the database's own record, so the event and the data share a commit.
• There are four approaches: polling, triggers, an outbox and log-based capture.