Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a way of replacing a single partition within a table #2167

Open
jenkoian opened this issue Dec 19, 2024 · 0 comments
Open

Add a way of replacing a single partition within a table #2167

jenkoian opened this issue Dec 19, 2024 · 0 comments

Comments

@jenkoian
Copy link

Feature description

As far as I can tell the replace write_disposition only supports replacing an entire table, I would like to be able to replace a single partition within a table.

I can do this with a merge disposition with a merge key, but it does a lot more than I want/need. It creates temp tables and stages data and all that stuff, when I simply want to just delete and re-insert data, I am aware this means there is a non-zero period of time the data is not available.

Are you a dlt user?

Yes, I'm already a dlt user.

Use case

I can use the merge write_disposition but it does a bunch of stuff I don't really need and can be quite slow and expensive (when running on Snowflake anyways). I looked at using upsert but it doesn't handle the case of existing entries being deleted if they no longer exist in the source (afaict).

Proposed solution

Example:

Event
id
type
match_id
id type match_id
1 pass 123
2 shot 123
3 pass 456

Replace everything in Event with the match_id 123 would delete events 1 and 2 and re-add them, but event with id 3 would remain untouched.

Related issues

I think this is similar? #1094

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Todo
Development

No branches or pull requests

1 participant