Cyberduck Mountain Duck CLI

#10849 closed enhancement (fixed)

Uploading to OneDrive Cryptomator vault is extremely slow if there are many files

Reported by: rootie Owned by: yla
Priority: normal Milestone: 7.9.0
Component: onedrive Version: 7.1
Severity: normal Keywords:
Cc: Architecture: PPC
Platform: macOS 10.15

Description

Hi,

there seems to be a bug in Cyberduck when lots of fairly small files are uploaded to a Cryptomator vault located in OneDrive. I've attached an example. Just unpack it and drag and drop it to the vault. It will be extremely slow.

Attachments (1)

Example.zip (584.0 KB) - added by rootie on Oct 19, 2019 at 11:34:24 AM.

Download all attachments as: .zip

Change History (24)

Changed on Oct 19, 2019 at 11:34:24 AM by rootie

comment:1 Changed on Oct 23, 2019 at 7:37:54 PM by dkocher

  • Component changed from core to onedrive

comment:2 Changed on Oct 23, 2019 at 7:38:12 PM by dkocher

  • Type changed from defect to enhancement

comment:3 Changed on Mar 17, 2020 at 9:12:28 AM by rootie

Hi,

what's the status on this? It still takes a very long time to browse through directories. Please give an update.

Thanks and best regards rootie

comment:4 Changed on Jul 17, 2020 at 2:13:54 PM by rootie

Hi,

is there any plan in improving this? This is clearly a bug and can most likely fixed fairly easily.

Please give an update.

Thanks and best regards rootie

comment:5 Changed on Apr 2, 2021 at 7:46:46 AM by panrag227

Can confirm, super slow, using Windows 10 latest ver Cyberduck 7.8.3 and Mountain duck 4.5.0, I suspect there also is something to do with Ticket 11582 because you get rate limited very often uploading. Tried uploading 20,000 small files and it took around 6 hours for even 2000 files to be uploaded! It's quite frustrating to be honest. I hope there is a fix soon. It doesn't look good that this has been open for over 2 years now with no resolution especially after i went out and just purchased this software for basically this reason :/

comment:6 Changed on Apr 2, 2021 at 8:02:35 AM by rootie

Yes unfortunately it seems that the developers are not interested in such things and this is very disappointing. The most disturbing thing is that there's not even a reaction - nothing.

As they call it 'enhancement' it's not declared a bug and hence won't be fixed anytime soon.

comment:7 follow-up: Changed on Apr 2, 2021 at 9:27:30 AM by panrag227

Yep, this is absolutely a defect, not a "enhancement". The software is unusable at this moment with OneDrive and basically over 1000 smallish files. Just went back and used the original OneDrive desktop client and Cryptomator desktop and it got my 20,000 files synced within an hour or two while Mountainduck and Cyberduck get stuck uploading for several more hours even if you have as few as 500 files! It's insane, absolutely is an issue with OneDrive syncing. I don't know whether to blame how slow it is to transfer many small files into a cryptomator vault on my awful computer or on Mountain Duck/Cyberduck though, Cryptomator desktop app seems to be blazing fast though. It's so weird. I was trialing this software for 2 weeks, it seemed alright with smaller things but it just can't handle a photo library, and I still went out and bought it today. I really didn't expect it to fail so hard today :/

comment:8 Changed on Apr 2, 2021 at 9:36:30 AM by rootie

  • Owner set to dkocher

comment:9 Changed on Apr 6, 2021 at 8:49:02 AM by dkocher

  • Milestone set to 8.0
  • Owner changed from dkocher to yla

comment:10 Changed on Apr 8, 2021 at 7:38:00 AM by dkocher

Related to #10990.

comment:11 Changed on Apr 8, 2021 at 7:41:36 AM by dkocher

  • Milestone changed from 8.0 to 7.8.6

comment:12 Changed on Apr 9, 2021 at 11:02:56 AM by dkocher

Fix in r50879 with missing cache.

Last edited on Apr 9, 2021 at 11:03:19 AM by dkocher (previous) (diff)

comment:13 Changed on Apr 9, 2021 at 11:04:43 AM by dkocher

Optimization in r50880 to add newly created directories to cache for later lookup of parent directory id when uploading containing files.

comment:14 Changed on Apr 12, 2021 at 5:52:17 PM by dkocher

Allow reuse of cached file id for previously created directory in r50894.

comment:15 Changed on Apr 12, 2021 at 5:55:43 PM by dkocher

New Auto default allows up to 20 concurrent transfers for an upload in r50899 to embrace parallelism when uploading many small files. The auto setting for FTP is at 1 connection (in r50900) because FTP is stateful and the number of allowed connections is usually limited by the server.

comment:16 Changed on Apr 12, 2021 at 6:00:16 PM by dkocher

The same issue affects all protocols where we have a file id to filename mapping.

comment:17 Changed on Apr 29, 2021 at 5:21:38 PM by dkocher

Optimization to reuse known file id when setting timestamp in r51085.

comment:18 Changed on Apr 29, 2021 at 5:25:26 PM by dkocher

Fix mapping of file id in decrypted file reference in r51147.

comment:19 Changed on Apr 29, 2021 at 5:25:52 PM by dkocher

Add internal cache in file id providers in r51124.

comment:20 Changed on Apr 29, 2021 at 5:26:47 PM by dkocher

Refactored in r51136 and r51131.

comment:21 Changed on Apr 29, 2021 at 5:27:24 PM by dkocher

  • Resolution set to fixed
  • Status changed from new to closed

In r51177.

comment:22 Changed on Apr 29, 2021 at 5:27:41 PM by dkocher

  • Milestone changed from 7.8.6 to 7.9.0

Milestone renamed

comment:23 in reply to: ↑ 7 Changed on Apr 29, 2021 at 7:37:34 PM by dkocher

Replying to panrag227:

Please try the current snapshot build by updating from within Cyberduck in Preferences → Update → Automatically check for updates in → Snapshot Builds.

Note: See TracTickets for help on using tickets.