Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

prepare_table does SDK handle this? If so migrate to it #256

Closed
visch opened this issue Dec 16, 2023 · 2 comments
Closed

prepare_table does SDK handle this? If so migrate to it #256

visch opened this issue Dec 16, 2023 · 2 comments

Comments

@visch
Copy link
Member

visch commented Dec 16, 2023

          @visch @sebastianswms this is great!

One thought when I saw this though was that I worked on similar improvements in the SDK (re: meltano/sdk#1779), not sure if its exactly the same or not. Maybe you've already done this recently but you might want to compare what youre doing in the overridden methods here to make sure its not already done in the SDK for you. Its been a while since this target was created and the SDK has come a long way. Overridden methods wont get to take advantage of ongoing improvements so removing them if theyre no longer needed could be a beneficial.

Originally posted by @pnadolny13 in #228 (comment)

@sebastianswms
Copy link
Contributor

I gave it a couple hours and wasn't able to get a reasonable implementation with the SDK's new prepare_table(). Many of the methods in target-postgres are overridden and have different expectations for input parameters than what is found in the SDK, meaning that it would take a lot to "unwind" the current target-postgres implementation (not as simple as just removing the prepare_table() override).

Based just on comparing the two codebases visually, I expect that the SDK does not address the same problems as #228. Following the path of execution, it's still O([number of columns]) db queries to prepare a table. Caching would speed things up after the initial run, but it would still fundamentally requires a large number of queries to set up.

@visch
Copy link
Member Author

visch commented Dec 21, 2023

Thanks for looking!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Archived in project
Development

No branches or pull requests

2 participants