Skip to content
This repository has been archived by the owner on Sep 23, 2024. It is now read-only.

Enable use of wal2json format-version 2 #152

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

judahrand
Copy link
Contributor

@judahrand judahrand commented Jan 24, 2022

Problem

The use of write-in-chunks is a slightly ugly hack now that format-version 2 is available.

Proposed changes

Make it possible to use format-version 2 but still default to version 1. Ideally, deprecating the use of version 1 in the future would be good.

The changes implemented are based on this PR: singer-io/tap-postgres#91

Types of changes

What types of changes does your code introduce to PipelineWise?
Put an x in the boxes that apply

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation Update (if none of the other choices apply)

Checklist

  • Description above provides context of the change
  • I have added tests that prove my fix is effective or that my feature works
  • Unit tests for changes (not needed for documentation changes)
  • CI checks pass with my changes
  • Bumping version in setup.py is an individual PR and not mixed with feature or bugfix PRs
  • Commit message/PR title starts with [AP-NNNN] (if applicable. AP-NNNN = JIRA ID)
  • Branch name starts with AP-NNN (if applicable. AP-NNN = JIRA ID)
  • Commits follow "How to write a good git commit message"
  • Relevant documentation is updated including usage instructions

@judahrand judahrand force-pushed the wal2json-format-2 branch 5 times, most recently from eddcc77 to 4c174e6 Compare January 24, 2022 14:36
@judahrand
Copy link
Contributor Author

@Samira-El I wonder if really it is worth just ripping out the format-version 1 code? There should be no functional difference it is just the recommended way of using wal2json now and so it would probably be best to use it.

Do you think it would be worth switching to pgoutput at some point in the future?

@judahrand judahrand force-pushed the wal2json-format-2 branch 4 times, most recently from 4481570 to 7193bbc Compare January 24, 2022 16:13
This option removes the need to use `write-in-chunks`.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant