Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Call to listObjectsV2 should be modified to handle +1K objects #267

Open
rdmurphy opened this issue Aug 23, 2020 · 0 comments
Open

Call to listObjectsV2 should be modified to handle +1K objects #267

rdmurphy opened this issue Aug 23, 2020 · 0 comments

Comments

@rdmurphy
Copy link
Collaborator

rdmurphy commented Aug 23, 2020

This is currently an intentional limitation of the library — the call to aws-sdk-js's s3.listObjectsV2 function only returns 1,000 objects, and expects you to further paginate to get more than this.

For the majority of projects this probably isn't an issue. Until it is! (FWIW we've never hit it at the LA Times, I think we hit it once or twice at Texas Tribune over the span of like 5-6 years.)

Now that it's acceptable to have our minimum supported Node version as v10, we can use async iterators to make this a cinch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant