Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🔧 change logic to see if ds or table exists #91

Merged
merged 5 commits into from
Jul 1, 2024

Conversation

AlejandroUPC
Copy link
Contributor

@AlejandroUPC AlejandroUPC commented Jun 16, 2024

Description

The goal of this MR is to tackle the issue mentioned here. The explanation is pretty much there.

Implementation

Right now to check if something exists the code tries to create the dataset with the parameter exists_ok, which in the internal bigquery library it just avoids raising an Exception as documented here.

The issue is that those calls are still created on the API and then handled in code, and your logs are going to be flooded with errors in your bigquery API.

With this MR the aim is to add an extra call to get the resources (dataset or table) and handle the NotFound exception itself. There is a potential cons that is making an additional call to the API to check if the resource actually exists.

Summoning @z3z1ma @blancaruano

@z3z1ma
Copy link
Owner

z3z1ma commented Jul 1, 2024

LGMT, I think I need to merge it for the tests to run
Also noticed maybe a bug in the other merged branch, I will follow up

@z3z1ma z3z1ma merged commit f9384ca into z3z1ma:main Jul 1, 2024
0 of 3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants