Cyberduck Mountain Duck CLI

#10743 new enhancement

Improve download speed of folder hierarchies

Reported by: allklier Owned by: dkocher
Priority: normal Milestone:
Component: s3 Version: 7.0
Severity: normal Keywords:
Cc: Architecture:
Platform: macOS 10.13

Description

I had to restore a backup from Amazon S3 that was created with rsync. There are deep folder hierarchies with hundreds of files. Cyberduck is one of the few tools that can connect to S3 and download a folder hierarchy. But it's excruciatingly slow, and not on account of the network speed.

In particular there seems to be a very long delay after the file tree has been retrieved. First Cyberduck appears to be walking the file tree and building a list. Then it appears to be sorting or doing something in the background. There is no UI feedback for minutes before the actual download starts. The download itself also crawls along. Not sure if there's some code that is not optimized for a large folder hierarchy or if the S3 API gets in the way?

A copy of a folder hierarchy with 4,703 files and 248MB in size took about 41 minutes! As a single file this would have finished in under a minute with exact same configuration.

When restoring backups time is of the essence. Would be fantastic if this could be optimized.

Change History (1)

comment:1 Changed on Aug 27, 2019 at 12:41:24 PM by dkocher

  • Component changed from core to s3
  • Owner set to dkocher
  • Summary changed from Very slow download of folder hierarchies to Improve download speed of folder hierarchies
Note: See TracTickets for help on using tickets.
swiss made software