Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Upload speed for many files #7927

Closed
cyberduck opened this issue May 1, 2014 · 2 comments
Closed

Upload speed for many files #7927

cyberduck opened this issue May 1, 2014 · 2 comments
Assignees
@dkocher
Labels
bug openstack OpenStack Swift Protocol Implementation worksforme

Comments

@cyberduck
Copy link
Collaborator

68e3fc5 created the issue

We are having trouble achieving a reasonable upload speed for a directory with many files. At the largest, we have to upload a new set of image tiles, each averaging ~25k, every 28 days. However, even with a test run of 500 files, Cyberduck goes through a very long process, indicating "Prepare xx.png" for each file, then the upload hovers around 12kb/sec.

We know that average upload speed will be much lower when you are uploading many small files, but the overall speed to upload our files to Cloud Files is a snail's pace.

What can be done to improve this speed?

Thanks!
@cyberduck
Copy link
Collaborator Author

@dkocher commented

Compiling the list of files to be included for the transfer depends on the local hard drive speed. Transferring many small files could be improved with multiple concurrent connections (#1135) for a transfer. This is not currently enabled by default as it hasn't seen enough testing. See Multiple connections per transfer on how to enable.

@cyberduck
Copy link
Collaborator Author

68e3fc5 commented

Thank you for your response. It's definitely not a hard drive speed issue as this is a new server (< 12 months old) with a 10k RPM RAID drive setup.

The hidden option sounds like it could help, however, how can that setting be found on the Windows version of Cyberduck?

The "prepare" sequence, however, still takes incredibly long and is not addressed by your answer.

For a reference point, it takes about 10 hours to upload 65k map tile image files to Rackspace Cloud Files via Cyberduck (when it completes, which is rare). In comparison, I can reliably upload 65k tiles to Amazon S3 via S3 Explorer in about 15 minutes. There is a vast discrepancy in performance.

@iterate-ch iterate-ch locked as resolved and limited conversation to collaborators Nov 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@dkocher dkocher

Labels
bug openstack OpenStack Swift Protocol Implementation worksforme
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dkocher @cyberduck