Make sure that Postgres always get a consistent snapshot #929
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
judahrand
force-pushed
the
postgres-snapshot
branch
from
2e18c58 to
535f236
Compare
|
@Samira-El This PR is ready for review, I believe. |
judahrand
force-pushed
the
postgres-snapshot
branch
4 times, most recently
from
b9ee0f5 to
ce27325
Compare
Merged
judahrand
force-pushed
the
postgres-snapshot
branch
from
0b98a99 to
d59e5d5
Compare
judahrand
force-pushed
the
postgres-snapshot
branch
3 times, most recently
from
04eaffb to
c04292a
Compare
This change ensures that the LSN at which the replication slot is first consistent is availiable to the caller of the method.
There is a replication lag between a primary and replica database. We get information about what data the replication slot can provide from the primary but perform the initial data snapshot on the replica. Therefore, we must make sure that the replica has caught up to at least the LSN which we are going to begin streaming from the replication slot.
judahrand
force-pushed
the
postgres-snapshot
branch
from
c04292a to
f795486
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
It is currently possible for PipelineWise to select an LSN to stream from which will result in missing data once the initial snapshot has been conducted.
This is due to the current mechanism of just selecting either the most recent LSN on the primary or the most recent LSN which has been replayed on the replica: https://github.com/thread/pipelinewise/blob/cfd2c2091eb60a5b71ea38486594ccfb91a4fe1c/pipelinewise/fastsync/commons/tap_postgres.py#L297-L322
This is not a safe thing to do as it does not guarantee a consistent view of the database.
Fixes: #930
Proposed changes
Inspired by how Debezium (another CDC tool written in Java) deals with initial snapshots of Postgres databases we will take advantage of two behaviours:
confirmed_flush_lsnto this value). We will stream from that point forward but use the current state of the database as the snapshot.confirmed_flush_lsnto be the point at which we will start streaming.In both of these cases, we may have data which will also be streamed to us by the replication slot but this should not be an issue as we are relying on an 'at least once' delivery mechanism already.
Types of changes
What types of changes does your code introduce to PipelineWise?
Put an
xin the boxes that applyChecklist
setup.pyis an individual PR and not mixed with feature or bugfix PRs[AP-NNNN](if applicable. AP-NNNN = JIRA ID)AP-NNN(if applicable. AP-NNN = JIRA ID)