-
Notifications
You must be signed in to change notification settings - Fork 71
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Error Handling and dead letter queues for targets #133
Comments
What is the status on this feature? Seems like a pretty useful usecase. |
@louis-vines a first pass for us would like be this issue: With better exit codes for SDK-based connectors we can start to handle each error better overall. Likely we need to break this issue up into specific proposals and make progress on those. cc @aaronsteers |
This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the |
Still relevant |
This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the |
Migrated from GitLab: https://gitlab.com/meltano/sdk/-/issues/134
Originally created by @vischous on 2021-05-26 17:40:34
Following up on our Office hours today. Not sure if we want this to be Target only or not your call @aaronsteers
Error Handling especially with SaaS style targets gets pretty interesting. Here's errors you'll hit at some point (one's that I can think about off the top of my head there's tons more, everything you can imagine when you run this stuff at scale)
Connection issues
Each of these errors needs to be handled slightly different. Some a simple retry with exponential backoff fixes your problem.
Data issues are something you can't get away from, and for a lot of SaaS apis (lots are not http based by the way, see Active Directory, and more) you'll get data errors that are masked as things like 500 errors.
Functionality that's probably needed:
Some of this "maybe all?" could be handling by a dead letter queue of some sort.
Use cases that I know about today:
The text was updated successfully, but these errors were encountered: