-
Notifications
You must be signed in to change notification settings - Fork 21
Open
Description
I'm checking if pg_flo suites my pipeline. Testing with docker-compose from the sample and configuration based on https://github.com/pgflo/pg_flo/blob/main/internal/pg-flo.yaml (with source and target connections and creds).
- pg_flo: v0.0.13 (from releases)
- OS: macOS Sequoia (apple silicon)
- Replicator:
./pg_flo replicator --config ./pg-flo.yml
- Worker:
./pg_flo worker postgres --config ./pg-flo.yml
- Super user account
- Postgres images versions: postgres:17
DDLs are synced manually, then I add/update/delete rows and target DB receives CDC updates. Checked with few updates, then 80K, that works.
However, when I run a bulk change (e.g. insert a million of dummy records):
/*
CREATE TABLE users (
id SERIAL PRIMARY KEY,
name TEXT
);
CREATE PUBLICATION pg_flo_group FOR TABLE users;
*/
INSERT INTO users (name)
SELECT gen_random_uuid()::TEXT
FROM generate_series(1, 10000000);
Data is synced, I see all rows appear in target, yet replicator fails with "receive message failed: unexpected EOF" and exits the process:
16:24:44.904 DBG Sent standby status update component=replicator lsn=0/6542E2D0
16:24:44.905 INF Relation message received component=replicator id=16390 table=users
16:24:44.983 ERR Error processing next message error="receive message failed: unexpected EOF" component=replicator
16:24:44.983 ERR Replication error occurred error="receive message failed: unexpected EOF"
16:24:44.983 INF Initiating graceful shutdown component=replicator
16:24:44.983 DBG Sent standby status update component=replicator lsn=0/6542E2D0
16:24:44.984 INF Closing database connections component=replicator
16:24:44.984 INF Base replicator shutdown completed component=replicator
On replicator restart, it sends messages again from the beginning and then fails again with the same error.
Metadata
Metadata
Assignees
Labels
No labels