You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
When DS gets overwhelmed with too many catalog requests, catalog can fail due to ES timeouts, DB timeouts, lambda errors, which results in data that are in S3 but not cataloged and unavailable for search.
Fixing the root cause is something DS will work on i.e. setting up DLQ, retrying failed DB writes, processing a batch of messages with a single lambda instead of invoking hundreds of concurrent lambdas, etc. but providing users with an API endpoint to catalog data is still useful.
The text was updated successfully, but these errors were encountered:
When DS gets overwhelmed with too many catalog requests, catalog can fail due to ES timeouts, DB timeouts, lambda errors, which results in data that are in S3 but not cataloged and unavailable for search.
Fixing the root cause is something DS will work on i.e. setting up DLQ, retrying failed DB writes, processing a batch of messages with a single lambda instead of invoking hundreds of concurrent lambdas, etc. but providing users with an API endpoint to catalog data is still useful.
The text was updated successfully, but these errors were encountered: