Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: The json module is now avaiable to stream maps #2531

Merged
merged 8 commits into from
Jul 16, 2024

Conversation

grigi
Copy link
Contributor

@grigi grigi commented Jul 15, 2024

This allows using json.loads() and json.dumps() to convert fields between text and json representations.

Particularily useful moving data between MySQL and Postgres where MySQL stores json as text, and Postgres stores json as JSONB.

Usage in a stream_map:

        stream_maps:
          sample-reports:
            recipients_email_addresses: json.loads(recipients_email_addresses)

Closes #1502


📚 Documentation preview 📚: https://meltano-sdk--2531.org.readthedocs.build/en/2531/

This allows using `json.loads()` and `json.dumps()` to convert fields
between text and json representations.
@grigi grigi requested a review from edgarrmondragon as a code owner July 15, 2024 09:26
Copy link

codspeed-hq bot commented Jul 15, 2024

CodSpeed Performance Report

Merging #2531 will not alter performance

Comparing grigi:main (e6b9041) with main (33b2d4e)

Summary

✅ 6 untouched benchmarks

Copy link

codecov bot commented Jul 15, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 89.23%. Comparing base (33b2d4e) to head (e6b9041).
Report is 146 commits behind head on main.

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #2531   +/-   ##
=======================================
  Coverage   89.23%   89.23%           
=======================================
  Files          54       54           
  Lines        4782     4782           
  Branches      935      935           
=======================================
  Hits         4267     4267           
  Misses        359      359           
  Partials      156      156           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@grigi grigi changed the title Add json function to CustomStreamMap feat: Add json function to CustomStreamMap Jul 15, 2024
@edgarrmondragon edgarrmondragon self-assigned this Jul 15, 2024
@grigi
Copy link
Contributor Author

grigi commented Jul 15, 2024

Ah, you want me to add docs to https://meltano-sdk--2531.org.readthedocs.build/en/2531/stream_maps.html#built-in-functions ?

@grigi grigi requested review from a team as code owners July 15, 2024 21:47
@edgarrmondragon
Copy link
Collaborator

@grigi
Copy link
Contributor Author

grigi commented Jul 16, 2024

Thanks for those documentation improvements @edgarrmondragon 🙏

@edgarrmondragon edgarrmondragon changed the title feat: Add json function to CustomStreamMap feat: The json module is now avaiable to stream maps Jul 16, 2024
@edgarrmondragon
Copy link
Collaborator

Thanks @grigi!

@edgarrmondragon edgarrmondragon added this pull request to the merge queue Jul 16, 2024
Merged via the queue into meltano:main with commit 8348456 Jul 16, 2024
31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Feat: add the json module to stream maps context
2 participants