What is Change Data Capture (CDC)? A Complete Guide for Data Engineers
CDC Captures Every Database Change the Instant It Happens
Change Data Capture (CDC) monitors your source database for inserts, updates, and deletes, then streams those changes downstream in real-time. No batch jobs. No full-table scans. No stale dashboards.
If your analytics show yesterday's data, your fraud detection lags transactions by hours, or your microservices poll a shared database for changes, CDC eliminates those problems at the architecture level.
How It Works
CDC monitors a data source for changes and extracts them in a structured format. Instead of re-scanning your entire dataset (the hallmark of batch ETL), CDC focuses exclusively on what changed. That's dramatically more efficient for large datasets.
Think of it as a security camera that only records when something moves, not one that photographs the entire room every hour.
Three Detection Methods
Log-based CDC reads the database's transaction log: PostgreSQL's WAL, MySQL's binlog, Oracle's redo logs, SQL Server's transaction log. The database already writes these logs for durability. CDC piggybacks on that existing work with minimal overhead. Sub-second latency. Captures everything. See database CDC sources for how log-based capture is configured.
Query-based CDC polls the source with queries like WHERE updated_at > @last_watermark. Database-agnostic and simple to implement, but adds query load, misses deletes, and introduces minutes-to-hours of latency. This is the model behind a direct pull source.
Trigger-based CDC fires database triggers on every write, recording changes to a changelog table. Captures complete before/after images, but every write incurs trigger overhead. That's a serious performance tax at scale.
| Approach | Latency | Source Impact | Captures Deletes | Best For |
|---|---|---|---|---|
| Log-Based | Sub-second | Minimal | Yes | Production real-time pipelines |
| Query-Based | Minutes-Hours | Medium | No (without soft deletes) | Simple, non-critical sync |
| Trigger-Based | Milliseconds-Seconds | High | Yes | Custom implementations |
The Architecture
A complete CDC implementation has five components:
- Source connector. Monitors your database and extracts change events. See the full connector reference for supported sources.
- Change event format. Structured payload (JSON/Avro) with operation type, timestamp, old/new values.
- Message broker. Kafka, Pulsar, or similar queue that decouples capture from consumption.
- Sink connectors. Write changes to destinations (warehouses, lakes, search indexes, caches).
- Schema registry. Tracks schema versions as source systems evolve.
Why CDC Beats Batch ETL
Traditional ETL queries your entire source table on a schedule: nightly, hourly, every few hours. If your nightly ETL runs at 2 AM and finishes at 3 AM, your dashboards show yesterday's data until the next run. Customers make decisions on stale information. Fraud detection lags behind transactions. Business teams can't react to emerging trends.
For a full breakdown of the tradeoffs, see CDC vs ETL: What's the Difference and When to Use Each. The short version:
Real-time freshness. Changes appear in destination systems within seconds. Dashboards show what's happening now, not last night.
Dramatic volume reduction. A billion-row table with 10,000 daily changes? Batch ETL processes a billion rows. CDC processes 10,000. Processing cost follows the rate of change, not the size of the table.
Lower source impact. Log-based CDC reads transaction logs the database writes anyway. No extra queries, no full-table scans hammering production.
Native delete capture. CDC captures DELETE operations as naturally as inserts. No soft-delete workarounds.
Multiple consumers. CDC platforms decouple capture from consumption. The same change stream feeds your warehouse, search index, and event-driven services independently.
Why it matters: This doesn't mean batch ETL disappears. Many organizations use both: CDC for real-time operations, ETL for historical analysis. But for any use case where data freshness matters, CDC is the architecture.
Use Cases That Justify CDC
Real-time data warehouse. Load fresh data continuously so analytical queries reflect current state. BI teams get dashboards they can trust.
Event-driven architecture. Every database change is an event. Feed changes into Kafka, trigger microservices, automate workflows based on specific mutations. The outbox pattern is the reliable way to publish those events without dual-write bugs.
Database replication. Maintain synchronized replicas across geographies. If your primary fails, a CDC-synced replica is ready with minimal data loss.
PII masking in transit. Pair CDC with transforms to mask credit cards, SSNs, and medical records as data moves to non-production environments. Compliance built into the pipeline. See how to mask PII in real-time data pipelines for the technique.
Multi-system synchronization. Keep CRM, ERP, warehouse, and analytics platform converged within seconds, not the hours that batch ETL requires. When the source is a SaaS app rather than a database, a SaaS connector fills the same role as log-based CDC.
Audit trails. Immutable, timestamped record of every modification. Who changed what, when. Regulatory compliance made structural.
The CDC Technology Landscape
Debezium. The industry-standard open-source platform. Connectors for PostgreSQL, MySQL, Oracle, SQL Server, MongoDB. Captures to Kafka. Mature, widely adopted, free. Requires Kafka infrastructure and operational expertise.
SchemaBounce. Purpose-built for sub-second CDC with native Python transforms. Kolumn IaC handles schema management automatically. Consumption-based pricing with a free tier. Designed to be easy for data engineers without Kafka complexity.
Fivetran. Commercial market leader with 500+ source connectors. Fully managed, excellent support. Premium pricing. Latency is minutes, not seconds. Transformation capabilities are limited. See SchemaBounce vs Fivetran for a direct comparison.
Airbyte. Open-source data integration with growing CDC capabilities. Good source coverage, low cost, self-hostable. CDC features are less mature than dedicated platforms. See SchemaBounce vs Airbyte.
Hevo Data. Cloud-native with strong UI/UX and consumption-based pricing. Good for teams who want managed CDC without deep technical investment.
For a side-by-side of eight platforms across latency, pricing, and connector breadth, see Best CDC Tools in 2026.
Choosing a CDC Solution
Be honest about latency. "Real-time" means different things. Fraud detection needs sub-second. Daily reporting needs minutes. Log-based CDC is necessary for the former; query-based may suffice for the latter.
Match to your sources. Not all CDC approaches work with all databases. Legacy Oracle systems need specialized support. Diverse stacks (PostgreSQL + MongoDB + SaaS) need broad connector coverage. Check the connector reference before committing to a platform.
Decide on transforms. Some platforms (Debezium) require a separate transformation layer. Others (SchemaBounce) include in-flight transforms. This affects total complexity and latency. Our PostgreSQL CDC tutorial walks through the mechanics if you want to see the raw setup first.
Plan for schema evolution. Real databases evolve. Columns get added, removed, renamed. Poor schema evolution handling becomes a nightmare at scale. Evaluate how each platform handles this before committing.
Calculate total cost. Platform pricing, infrastructure, ops labor, training, and vendor lock-in switching costs. Open-source minimizes licensing but maximizes operational investment.
Best Practices
Monitor replication lag relentlessly. The gap between a change occurring and arriving at the destination is your most important metric. Set alerts. Most teams target sub-5-second lag for operational systems.
Design for idempotency. Network hiccups happen. Your downstream systems must handle the same change applied twice without corrupting data. Dead letter queues catch failures for manual inspection.
Automate schema evolution. Column additions should flow through automatically. Deletions and type changes need compatibility checks. The more automated this is, the fewer outages you'll have.
Test the edges. Verify deletes actually remove downstream records. Test thousands of simultaneous changes. Test schema changes. Stop the pipeline mid-operation, restart, verify no data loss. Automated testing catches what manual review misses.
Preserve the audit trail. CDC's superpower is immutable change history. Store raw change events durably. Include timestamps and metadata. Make it easy to answer "what was this record's value on date X?"
The Bottom Line
CDC shifts data engineering from "process everything on a schedule" to "capture and propagate every change immediately." This enables real-time analytics, event-driven architectures, and fundamentally more responsive data systems.
The choice isn't finding the "best" CDC solution. It's matching the right approach to your latency requirements, team capabilities, and architecture.
Ready to explore CDC? Try SchemaBounce's free tier for sub-second CDC with Python transforms. No credit card required.