Skip to content
This repository has been archived by the owner on Sep 23, 2024. It is now read-only.

Partial update of records will set all non-PK columns to null #275

Open
Tolsto opened this issue May 10, 2022 · 0 comments
Open

Partial update of records will set all non-PK columns to null #275

Tolsto opened this issue May 10, 2022 · 0 comments
Labels
bug Something isn't working

Comments

@Tolsto
Copy link
Contributor

Tolsto commented May 10, 2022

Describe the bug
When using the Postgres tap with log-based replication a delete will generate a record message that only contains the primary key properties and metadata columns like _sdc_deleted_at. This will cause the Snowflake target to reset all other columns to null as the MERGE query does no distinguish between missing columns in the record message and columns that are truly null.

This behaviour doesn't matter much when you are using hard-delete and are going to delete the row in the target anyway. However, when you actually want to keep deleted source data in your DWH then it's a big problem.

To Reproduce
Steps to reproduce the behavior:

  1. Use the target with a tap that sends incomplete record message (e.g. Postgres with log-based replication when doing deletes)
  2. Generate an incomplete record message for an existing row in the target.

Expected behavior
A partial update should not set unaffected columns to null

Your environment

  • Version of target: 2.1.0
  • Version of python: 3.9.12
@Tolsto Tolsto added the bug Something isn't working label May 10, 2022
Tolsto added a commit to Tolsto/pipelinewise-target-snowflake that referenced this issue May 10, 2022
Record messages for some updates from certain taps, e.g.
Postgres with log-based replication when doing deletes,
will only contain metadata and pkey columns. The current
MERGE logic would then set all non-included columns in the
target to null.
Tolsto added a commit to Tolsto/pipelinewise-target-snowflake that referenced this issue May 11, 2022
Record messages for some updates from certain taps, e.g.
Postgres with log-based replication when doing deletes,
will only contain metadata and pkey columns. The current
MERGE logic would then set all non-included columns in the
target to null.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant