Skip to content

Source error with sync delete all dest files #695

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
adrienyhuel opened this issue Jan 30, 2024 · 2 comments
Open

Source error with sync delete all dest files #695

adrienyhuel opened this issue Jan 30, 2024 · 2 comments
Labels

Comments

@adrienyhuel
Copy link

Hello,

I'm trying to synchronize the content of an S3 to a local directory.

If there is an error on source (like a network timeout to reach the S3, or the bucket doesn't exist), the existing files in the directory can be all deleted.

I suspect there is a race condition between listing source and dest, and the first generating an error will fail the sync process.
I think about that because if the dest folder contain only few files, then the files are deleted.
But if the dest folder contains lots of files (like /tmp dir), no files are deleted, because src s3 lisiting fail before dest local dir listing finish.

@grmrgecko
Copy link

I'm working this out, will have a pull request in once testing is done.

@grmrgecko
Copy link

Ok, I have two pull requests that are waiting on the maintainers for review.

This one should outright solve the issue for errors:
#698

This one is more of an idea I thought of as I use the trick on rsync:
#699

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: No status
Development

No branches or pull requests

3 participants