Change Data Capture

Change Data Capture (CDC) is a technique for tracking and propagating changes (inserts, updates, deletes) in a database or data source to downstream systems in near-real-time. Instead of re-processing the entire dataset to find what changed, CDC identifies only the delta - the records that are new or modified since the last sync.

CDC is valuable for keeping downstream systems in sync with a source of truth without the cost of full re-ingestion. For example, instead of re-fetching all property listings from an API every hour, a CDC-like approach detects which listings changed (new price, new status, new listing) and propagates only those changes.

In the context of APIs, CDC patterns are approximated by querying a since parameter (records updated after a timestamp) or by maintaining a local snapshot and diffing against new API responses. True database CDC uses transaction log tailing (tools like Debezium) which is not available for external API data sources.