Skip to main content

Real-Time CDC Webhooks: Streaming Database Changes to Any Endpoint

· 6 min read
blog-writer

Change Data Capture gives you a continuous feed of every insert, update, and delete from your database. Webhooks give you a way to push that feed to any HTTP endpoint — a microservice, a Slack notification system, a data lake ingestor, or an LLM pipeline. Combine them and you get low-latency, event-driven integrations without polling or cron jobs.

This tutorial walks through configuring SchemaBounce to route CDC events to webhook sinks, covering filtering, retry policies, payload schema, and dead-letter handling.

Why Webhooks Over Polling​

Polling a database or an API for changes is wasteful by default:

  • Latency: You only catch changes at poll interval — typically 30s to 5 minutes.
  • Load: Every poll is a read query, even when nothing changed.
  • Missed events: If a record is written and then deleted between polls, the change is invisible.

CDC reads from the database replication log (e.g., PostgreSQL's logical replication slot, MySQL's binlog), so every row-level event is captured exactly once, in order, with sub-second latency.

Routing those events to a webhook endpoint means your downstream service reacts within milliseconds of the commit.

How SchemaBounce CDC-to-Webhook Works​

SchemaBounce runs a replication consumer that tails your source database's change stream. Each event is normalized into a structured payload, filtered against your pipeline rules, and delivered to your configured sink.

For webhook sinks, the delivery flow is:

Database WAL / binlog
       ↓
  SchemaBounce CDC consumer
       ↓
  Event normalization (unified schema)
       ↓
  Filter rules (table, operation, column conditions)
       ↓
  Webhook sink → HTTPS POST to your endpoint
       ↓ (on failure)
  Retry with exponential backoff
       ↓ (after max retries)
  Dead-letter queue

Step 1: Configure Your Source​

SchemaBounce supports PostgreSQL (logical replication), MySQL (binlog), and MongoDB (change streams) as CDC sources.

For PostgreSQL, you need logical replication enabled and a replication slot:

-- Enable logical replication (requires superuser, set in postgresql.conf)
-- wal_level = logical

-- Create a replication slot
SELECT pg_create_logical_replication_slot('schemabounce_slot', 'pgoutput');

-- Create a publication for the tables you want to stream
CREATE PUBLICATION schemabounce_pub FOR TABLE orders, customers, inventory;

In SchemaBounce, register the source:

# schemabounce-source.yaml
source:
  id: pg_production
  type: postgres_cdc
  connection: "postgresql://user:[email protected]:5432/mydb"
  replication_slot: schemabounce_slot
  publication: schemabounce_pub

Step 2: Define the Webhook Sink​

A webhook sink is an HTTPS endpoint that receives POST requests. SchemaBounce requires HTTPS — plaintext HTTP is rejected.

# schemabounce-sink.yaml
sink:
  id: order_webhook
  type: webhook
  url: "https://api.example.com/hooks/order-events"
  headers:
    Authorization: "Bearer ${WEBHOOK_SECRET}"
    Content-Type: "application/json"
  batch_size: 1          # deliver events one at a time (default)
  flush_interval_ms: 0   # no buffering — deliver immediately on commit
  retry_policy:
    max_attempts: 5
    backoff: exponential
    initial_delay_ms: 500
    max_delay_ms: 30000
  dlq_enabled: true

Secret injection: Reference environment variables with ${VAR_NAME} — SchemaBounce never logs header values. Store secrets in your vault and inject at runtime.

Step 3: Create the Pipeline Route​

A route connects a source to a sink, with optional filter rules:

# schemabounce-route.yaml
route:
  id: orders_to_webhook
  source_id: pg_production
  sink_id: order_webhook
  filters:
    - table: orders
      operations: [INSERT, UPDATE]   # skip DELETEs
      conditions:
        - column: status
          operator: in
          values: ["paid", "shipped", "cancelled"]

This route will stream only INSERT and UPDATE events from the orders table, and only when status is one of the specified values. Low-signal noise (draft orders, internal updates) is dropped before delivery.

Apply the config:

schemabounce route apply --file schemabounce-route.yaml
# Route 'orders_to_webhook' created. Status: active

Event Payload Schema​

Each webhook delivery is a single JSON object (or an array for batched delivery). The payload follows SchemaBounce's normalized CDC envelope:

{
  "event_id": "evt_01HX4K9N2P3Q7RSTVWXYZ",
  "source_id": "pg_production",
  "table": "orders",
  "operation": "UPDATE",
  "committed_at": "2026-05-23T14:32:01.847Z",
  "lsn": "0/3A1F290",
  "before": {
    "id": 8821,
    "status": "pending",
    "total_cents": 4999
  },
  "after": {
    "id": 8821,
    "status": "paid",
    "total_cents": 4999
  }
}

Key fields:

FieldDescription
event_idGlobally unique, idempotency-safe identifier
operationINSERT, UPDATE, or DELETE
committed_atUTC timestamp of the database commit
lsnPostgreSQL Log Sequence Number (for ordering)
beforeRow state before the change (null for INSERT)
afterRow state after the change (null for DELETE)

Handling Retries and Idempotency​

SchemaBounce retries on any non-2xx response from your endpoint. This means your endpoint will receive duplicate deliveries during transient failures. Design for idempotency:

# Python Flask example — idempotent event handler
from flask import Flask, request, jsonify
import redis

app = Flask(__name__)
r = redis.Redis()

@app.route("/hooks/order-events", methods=["POST"])
def handle_order_event():
    event = request.get_json()
    event_id = event["event_id"]

    # Deduplicate using event_id
    if r.exists(f"processed:{event_id}"):
        return jsonify({"status": "duplicate"}), 200  # 2xx so SchemaBounce stops retrying

    # Process the event
    process_order_change(event)

    # Mark as processed (TTL 24h is sufficient for retry windows)
    r.setex(f"processed:{event_id}", 86400, "1")
    return jsonify({"status": "ok"}), 200

Return a 2xx even for duplicates — returning 4xx or 5xx for a duplicate will trigger another retry.

Dead-Letter Queue​

Events that exhaust all retry attempts are written to the DLQ. SchemaBounce supports DLQ targets:

  • An S3/GCS bucket (JSON lines, partitioned by date)
  • A secondary webhook endpoint
  • A Kafka topic

Configure a DLQ target in your sink:

sink:
  id: order_webhook
  type: webhook
  # ... main config ...
  dlq_enabled: true
  dlq_target:
    type: s3
    bucket: "my-dlq-bucket"
    prefix: "schemabounce/dlq/order_webhook/"
    partition_by: "date"

Inspect and replay DLQ events:

# List DLQ events for a route
schemabounce dlq list --route orders_to_webhook --limit 20

# Replay a specific event
schemabounce dlq replay --event-id evt_01HX4K9N2P3Q7RSTVWXYZ

# Replay all DLQ events for a route (use with caution)
schemabounce dlq replay --route orders_to_webhook --all

Monitoring Delivery Health​

SchemaBounce exposes per-route metrics you can scrape with Prometheus or view in the dashboard:

schemabounce_route_events_total{route="orders_to_webhook", status="delivered"} 148291
schemabounce_route_events_total{route="orders_to_webhook", status="retried"}   312
schemabounce_route_events_total{route="orders_to_webhook", status="dlq"}       4
schemabounce_route_delivery_latency_p99{route="orders_to_webhook"}             87ms

A healthy route has a dlq count near zero and p99 latency under 500ms. If retried is climbing without dlq increasing, your endpoint is recovering on retry — acceptable but worth investigating.

Common Patterns​

Fan-out: one source to multiple webhooks​

routes:
  - id: orders_to_fulfillment
    source_id: pg_production
    sink_id: fulfillment_webhook
    filters:
      - table: orders
        operations: [INSERT]

  - id: orders_to_analytics
    source_id: pg_production
    sink_id: analytics_webhook
    filters:
      - table: orders
        operations: [INSERT, UPDATE, DELETE]

Filtered enrichment: only changed fields​

Add changed_fields_only: true to your route to include only the columns that actually changed in the after object:

route:
  id: orders_to_webhook
  source_id: pg_production
  sink_id: order_webhook
  options:
    changed_fields_only: true

This reduces payload size significantly for wide tables.

Wrapping Up​

CDC-to-webhook pipelines give you exactly-once semantics (with idempotent consumers), sub-second latency, and a clean separation between your database and your event consumers. No polling, no scheduled jobs, no missed updates between windows.

The pattern scales from a single webhook for a microservice notification to fan-out across analytics, fulfillment, compliance, and AI pipelines — all driven from a single replication slot.

Next steps:

Tags:
  • cdc
  • webhooks
  • real-time
  • streaming
  • postgres
  • change-data-capture
  • pipeline