Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve download speed of folder hierarchies #10743

Closed
cyberduck opened this issue Jun 20, 2019 · 2 comments
Closed

Improve download speed of folder hierarchies #10743

cyberduck opened this issue Jun 20, 2019 · 2 comments
Assignees
Labels
enhancement fixed s3 AWS S3 Protocol Implementation
Milestone

Comments

@cyberduck
Copy link
Collaborator

4d59c66 created the issue

I had to restore a backup from Amazon S3 that was created with rsync. There are deep folder hierarchies with hundreds of files. Cyberduck is one of the few tools that can connect to S3 and download a folder hierarchy. But it's excruciatingly slow, and not on account of the network speed.

In particular there seems to be a very long delay after the file tree has been retrieved. First Cyberduck appears to be walking the file tree and building a list. Then it appears to be sorting or doing something in the background. There is no UI feedback for minutes before the actual download starts. The download itself also crawls along. Not sure if there's some code that is not optimized for a large folder hierarchy or if the S3 API gets in the way?

A copy of a folder hierarchy with 4,703 files and 248MB in size took about 41 minutes! As a single file this would have finished in under a minute with exact same configuration.

When restoring backups time is of the essence. Would be fantastic if this could be optimized.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

We have increased the default of concurrent connections used in 013a4b9 which should improve the performance for downloads with many small files.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Milestone renamed

@iterate-ch iterate-ch locked as resolved and limited conversation to collaborators Nov 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement fixed s3 AWS S3 Protocol Implementation
Projects
None yet
Development

No branches or pull requests

2 participants