Cyberduck Mountain Duck CLI

#11614 reopened enhancement

Allow transfers skipping pre-scanning of folders and files to download

Reported by: marcus_zeng Owned by:
Priority: normal Milestone:
Component: core Version: 7.8.3
Severity: normal Keywords: S3, Slow Downloading, Downloading hanging
Cc: Architecture:
Platform: macOS 11

Description

When starting to download a S3 directory full of nested subdirectories and files, the download starts with a long hang. For a directory with size 1G, the hanging can be as long as 10 minutes. Downloading the directory with aws_cli or other file transfer tools didn't encounter the same issue.

We tried both Windows 10 and macOS 11 and experienced the same issue.

Let me know if more information is needed to reproduce the issue.

https://files.slack.com/files-pri/TGTU5N9MG-F01R4LDJEAK/screen_shot_2021-03-17_at_11.14.18_am.png https://files.slack.com/files-pri/TGTU5N9MG-F01R48BD97Z/image.png

Attachments (2)

screen_shot_2021-03-17_at_11.14.18_am.png (64.9 KB) - added by marcus_zeng on Mar 17, 2021 at 9:55:46 PM.
downloading hanging on macOS 11
image.png (106.8 KB) - added by marcus_zeng on Mar 17, 2021 at 9:56:01 PM.
downloading hanging on Windows

Download all attachments as: .zip

Change History (6)

Changed on Mar 17, 2021 at 9:55:46 PM by marcus_zeng

downloading hanging on macOS 11

Changed on Mar 17, 2021 at 9:56:01 PM by marcus_zeng

downloading hanging on Windows

comment:1 Changed on Mar 24, 2021 at 4:23:57 PM by dkocher

  • Resolution set to worksforme
  • Status changed from new to closed

We are reading remote file attributes prior a file transfer which causes the delay in the Prepare… status.

comment:2 Changed on Mar 24, 2021 at 7:31:51 PM by marcus_zeng

  • Resolution worksforme deleted
  • Status changed from closed to reopened

Is there a way to turn off the "pre-scanning" somewhere? It would be nice if it can start the download recursively just like aws-cli by options.

If this option is not available and you think this is a good feature to add in, let me know. I can look over and push an MR about this.

Last edited on Mar 24, 2021 at 8:26:31 PM by marcus_zeng (previous) (diff)

comment:3 Changed on Apr 7, 2021 at 10:22:21 AM by svogt

  • Type changed from defect to enhancement

There is currently no way to turn off or work around the pre-scanning process.

comment:4 Changed on Apr 8, 2021 at 7:21:13 AM by dkocher

  • Component changed from s3 to core
  • Summary changed from S3 Download Starts with A Long Hanging to Allow transfers skipping pre-scanning of folders and files to download
Note: See TracTickets for help on using tickets.