Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

change blast_stats.json schema to add new fields #4

Merged
merged 2 commits into from
Sep 17, 2024

Conversation

yujoy
Copy link
Contributor

@yujoy yujoy commented Sep 17, 2024

Change the blast_stats.json schema to add new fields from the documentation here

Ran a test meltano elt into nymag-analytics-157315.dev_meltano.dev_blast_stats with the new schema

@diwakarkasi
Copy link

Would we want to manually update the DDL for the target table? Also any changes needed downstream, like the dbt models?

Copy link

@diwakarkasi diwakarkasi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a comment for clarification

@yujoy
Copy link
Contributor Author

yujoy commented Sep 17, 2024

Would we want to manually update the DDL for the target table? Also any changes needed downstream, like the dbt models?

As discussed, rather than changing the DDL, feels easier to just delete the current prod tables and full refresh them.
As for dbt changes, yeah we'll need to make some to incorporate the new fields/schema, but ran a test on the stg model against my dev table and it ran successfully -

image

@yujoy
Copy link
Contributor Author

yujoy commented Sep 17, 2024

With the last commit, we're going to try ignore_parent_replication_key = True since the replication key for BlastStream is start_time and we don't expect that to really change per record. Adding this config will pull the latest blast_stats for each blast, so the ingestion will take longer but we will have the freshest data. Docs here

Also added a few more fields to the blast_stats.json schema.

@yujoy yujoy merged commit 1b940d1 into main Sep 17, 2024
0 of 4 checks passed
@yujoy yujoy deleted the add_new_schema_fields branch September 17, 2024 20:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants