Skip to content
This repository has been archived by the owner on Sep 23, 2024. It is now read-only.

Add support for transferring postgres range types as json objects #179

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

gwax
Copy link

@gwax gwax commented Jun 8, 2022

Problem

tap-postgres silently drops postgres range columns

Proposed changes

Add schema and marshalling support for for postgres range types via custom marshalling. We convert the incoming psycopg2.extras.Range object to a json object of the form {"lower": 0, "upper": 5, "bounds": "[)"}. This structure can be parsed and used in a downstream transform to reconstruct the contents of the original source range.

Types of changes

What types of changes does your code introduce to PipelineWise?

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation Update (if none of the other choices apply)

Checklist

  • Description above provides context of the change
  • I have added tests that prove my fix is effective or that my feature works
  • Unit tests for changes (not needed for documentation changes)
  • CI checks pass with my changes
  • Bumping version in setup.py is an individual PR and not mixed with feature or bugfix PRs
  • Commit message/PR title starts with [AP-NNNN] (if applicable. AP-NNNN = JIRA ID)
  • Branch name starts with AP-NNN (if applicable. AP-NNN = JIRA ID)
  • Commits follow "How to write a good git commit message"
  • Relevant documentation is updated including usage instructions

@gwax gwax force-pushed the gwax/range-types branch from 0f91d30 to dbe17a2 Compare June 8, 2022 08:25
@gwax gwax changed the title Add support for transferring postgres range types as strings Add support for transferring postgres range types as json objects Jun 8, 2022
@gwax gwax force-pushed the gwax/range-types branch 2 times, most recently from 17fef8e to 9c9db6a Compare December 28, 2022 22:55
@gwax
Copy link
Author

gwax commented Dec 28, 2022

@Samira-El , @louis-pie : I have rebased and updated this PR against master and have added unit and integration tests to cover the behavior. We have also successfully used it in a production context.

Could you please review so that the functionality can make its way into the mainline codebase?

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant