Skip to main content

Direct Pull

SchemaBounce connects directly to your database and queries for changes. This is the simplest setup method: no agents, no replication configuration, just connection credentials.

Best for Managed Cloud Databases

Direct pull works well with RDS, Cloud SQL, Azure SQL, and other managed databases that may not expose transaction logs directly.

How It Works​

SchemaBounce periodically connects to your database and queries for changes using timestamp-based or cursor-based tracking.

SchemaBounce Cloud --[(1) Connect via secure tunnel]--> Your Database (RDS, Cloud SQL)
Your Database --[(2) Query: SELECT * FROM orders WHERE updated_at > ?]--> Changed Records
Changed Records --[(3) Stream to pipeline]--> Sinks (Kafka, Webhooks, etc.)
warning

Direct pull requires an updated_at timestamp column or similar cursor field on the tables you want to sync. Without this, only full refreshes are possible.

Configuration​

source:
  type: direct_pull
  connection:
    host: your-db.rds.amazonaws.com
    port: 5432
    database: production
    user: schemabounce_reader
    password: ${DB_PASSWORD}
    ssl_mode: require

  tables:
    - name: orders
      cursor_field: updated_at
      primary_key: id
    - name: customers
      cursor_field: modified_at
      primary_key: customer_id

  sync:
    interval: 60s # Poll every 60 seconds
    batch_size: 1000 # Records per batch
    mode: incremental # or 'full_refresh'

Supported Databases​

DatabaseManaged OptionsMin Interval
PostgreSQLRDS, Cloud SQL, Azure, Supabase, Neon10s
MySQLRDS, Cloud SQL, Azure, PlanetScale10s
SQL ServerRDS, Azure SQL, Azure Managed Instance30s
SnowflakeSnowflake Cloud60s
BigQueryGoogle BigQuery60s
RedshiftAWS Redshift, Redshift Serverless60s

See the provider references for PostgreSQL and MySQL.

Latency Expectations​

Poll IntervalAvg LatencyUse Case
10 seconds~5-15sNear real-time dashboards
60 seconds~30-90sGeneral analytics, reporting
5 minutes~2.5-7.5mBatch ETL, data warehouse sync
1 hour~30-90mDaily aggregations, cost optimization
note

Need sub-second latency? Consider Database CDC, which reads directly from transaction logs.

Direct Pull vs CDC​

Direct Pull

  • Simple setup: just credentials
  • Works with managed databases
  • No replication config needed
  • Higher latency (seconds to minutes)
  • Requires a cursor or timestamp column
  • Adds query load to the database

Database CDC

  • Sub-50ms latency
  • Zero query load on the database
  • Captures all changes in order
  • Requires replication setup
  • May not work with all managed databases
  • More complex configuration

When to Use Direct Pull​

Good fit:

  • Managed databases without log access
  • Quick proof-of-concept setup
  • Minute-level latency is acceptable
  • Small to medium data volumes
  • Tables have timestamp columns

Consider alternatives:

  • Need sub-second latency: use Database CDC
  • High-volume tables: use Database CDC
  • No timestamp columns: use the Outbox Pattern
  • Network-restricted database: use an agent push method (see the Bridge documentation, linked below)

Connection Security​

  • TLS/SSL required: all connections use encrypted transport.
  • IP allowlisting: whitelist SchemaBounce IPs in your firewall.
  • Read-only user: create a dedicated user with SELECT-only permissions.
  • Credential encryption: passwords are stored with AES-256-GCM envelope encryption.
-- Recommended: Create read-only user for direct pull
CREATE USER schemabounce_reader WITH PASSWORD 'secure_password';
GRANT CONNECT ON DATABASE production TO schemabounce_reader;
GRANT USAGE ON SCHEMA public TO schemabounce_reader;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO schemabounce_reader;

-- Grant access to future tables
ALTER DEFAULT PRIVILEGES IN SCHEMA public
GRANT SELECT ON TABLES TO schemabounce_reader;