Cyberduck Mountain Duck CLI

#7927 closed defect (worksforme)

Upload speed for many files

Reported by: ifpjohn Owned by: dkocher
Priority: normal Milestone:
Component: openstack Version: 4.4.4
Severity: normal Keywords:
Cc: Architecture:
Platform: Windows 8

Description

We are having trouble achieving a reasonable upload speed for a directory with many files. At the largest, we have to upload a new set of image tiles, each averaging ~25k, every 28 days. However, even with a test run of 500 files, Cyberduck goes through a very long process, indicating "Prepare xx.png" for each file, then the upload hovers around 12kb/sec.

We know that average upload speed will be much lower when you are uploading many small files, but the overall speed to upload our files to Cloud Files is a snail's pace.

What can be done to improve this speed?

Thanks!

Change History (2)

comment:1 Changed on May 2, 2014 at 9:36:05 AM by dkocher

  • Resolution set to worksforme
  • Status changed from new to closed
  • Summary changed from Cyberduck Upload Speed for Many Files to Upload speed for many files

Compiling the list of files to be included for the transfer depends on the local hard drive speed. Transferring many small files could be improved with multiple concurrent connections (#1135) for a transfer. This is not currently enabled by default as it hasn't seen enough testing. See Multiple connections per transfer on how to enable.

comment:2 Changed on May 2, 2014 at 2:38:15 PM by ifpjohn

Thank you for your response. It's definitely not a hard drive speed issue as this is a new server (< 12 months old) with a 10k RPM RAID drive setup.

The hidden option sounds like it could help, however, how can that setting be found on the Windows version of Cyberduck?

The "prepare" sequence, however, still takes incredibly long and is not addressed by your answer.

For a reference point, it takes about 10 hours to upload 65k map tile image files to Rackspace Cloud Files via Cyberduck (when it completes, which is rare). In comparison, I can reliably upload 65k tiles to Amazon S3 via S3 Explorer in about 15 minutes. There is a vast discrepancy in performance.

Note: See TracTickets for help on using tickets.
swiss made software