better handler API error parsing. #1510
Conversation
Possibly we encountered an error while parsing the API error JSON, like connection closes early etc and the received JSON is invalid. Better handle such scenario and build a valid error Dict to keep the exception stack flow running and raise this message to the application. extract the returned text so if we any bit of valid information at least we can pass that to the application. This could be helpfull to if we add calls to the API made for web browser that returns HTML error message, it will fail to decode the recieved error and raise a proper error. closes #1504 Signed-off-by: Alexandre Lavigne <[email protected]>
|
I could possibly manually test it, but that's quiet hard to do. otherwise, using automatic test no 😞 we do not regress in anything so far. |
|
we could make a test that uses a HTTP client and mock the response? |
Possibly yes 🤔 I'm worried how this is gonna play with the cassette recorder too 🤔 it's worth the try. |
|
appreciate the quick fix here, we started seeing these as well. any chance we'll get a release with this change? |
|
It's done ✔️ |
so far we can't tell, we have quite a few new commits but nothing minor/major except this one. |
|
Helpful change, thanks for fixing. We need this fix because suddenly we've been getting a lot of 502 errors from the Google Sheets API. The response on the 502 does not contain a json object but instead the HTML/CSS of an error page, which triggers a JSONDecodeError. We implemented gspreads Will be on the look out for a release. Will be massively helpful. Thanks. |
|
I have turned the release into an issue... we will do it soon :) |
| # and keep the exception raise flow running | ||
|
|
||
| error = { | ||
| "code": -1, |
There was a problem hiding this comment.
Hey @lavigne958 I'm wondering if the error code here should instead be response.status_code. I noticed this because the BackOffHTTPClient will not retry to send the request because -1 is not an expected error code here.
There was a problem hiding this comment.
Hi that's a good point.
Unfortunately we can't simply set the return code as we failed to parse the returned error.
This could be 2 reasons: 1. The response is incomplete/connection lost etc. 2. The response is in html or some other format we don't handle nor expect but the response is valid !
I will check with alifee what is best here, I have 2 ideas:
-
Instead of returning a mock http response, we raise a new Gspread issue and the user is in charge of retrying or not. We don't take a decision here.
-
We add a new setting for the http client to retry on -1 errors. So the user can enable that setting and the http client will retry on -1 error code too.
@alifeee what you do think ?
There was a problem hiding this comment.
For what it's worth, I imagine there would still be HTTP codes coming back with the scenarios you mentioned, e.g. 408 (timeout), 502 (server error, with html response).
Glad to know I'm not the only one seeing this the last few weeks. |
|
@Kirkman absolutely. It's almost 1 in 10 calls for me at this rate. Not something I'd expect from Google, even if it's a free service. I can't find much chatter about it online, and no acknowledgement from Google on any social media. Meanwhile my Google Cloud dashboard is massively undercounting the API error rate I'm seeing. But I've since forked the repo, added the status_code fix I mention above, implemented the BackOffHttpClient, and have been swimming problem-free since. |
|
The intermittent 502 errors seems to be a bug on Google's end, related issue: https://issuetracker.google.com/issues/369670473 |
Possibly we encountered an error while parsing the API error JSON, like connection closes early etc and the received JSON is invalid.
Better handle such scenario and build a valid error Dict to keep the exception stack flow running and raise this message to the application.
extract the returned text so if we any bit of valid information at least we can pass that to the application.
This could be helpfull to if we add calls to the API made for web browser that returns HTML error message, it will fail to decode the recieved error and raise a proper error.
closes #1504