Cyberduck Mountain Duck CLI

#11000 closed enhancement (fixed)

Increase OneDrive multipart chunk size

Reported by: Rjevski Owned by:
Priority: normal Milestone: 7.3.0
Component: onedrive Version: 7.2.5
Severity: normal Keywords:
Cc: Architecture: Intel
Platform: macOS 10.14

Description

Hello,

Just wondering if there are any plans to increase the OneDrive multipart chunk size?

The official client syncs quite fast at around 17MB/s. Uploading the same file using Cyberduck hovers at around 6MB/s (with the change of the "factor" introduced recently as in https://github.com/iterate-ch/cyberduck/blob/8080fc9e1c0b6f40ac23d1dca4f79a21598df375/core/src/main/java/ch/cyberduck/core/preferences/Preferences.java#L739 ) - my current build (7.2.5) doesn't seem to have it implemented so I just simulated it by setting the "onedrive.upload.multipart.partsize.minimum" value to minimum*20.

After setting the "minimum" part size setting to a ridiculously high value like 480000000 I am then able to get roughly the same speeds as the official client. This causes some weird behaviour with the transfer rate estimation and progress bar of Cyberduck as well as a "Java heap error" (probably out of memory?) when trying to upload a 1GB file. Reducing the value to half that (240000000) made the 1GB file upload fine and still had a speed comparable to the official client.

Is there any chance we can up this value further, potentially depending on the file size? The official docs set an upper limit of 60MB per chunk: https://docs.microsoft.com/en-us/graph/api/driveitem-createuploadsession?view=graph-rest-1.0#upload-bytes-to-the-upload-session - with your experience interfacing with different cloud storage providers, is there any reason *not* to use the max chunk size a provider allows, and what determines an optimal chunk size?

Regards.

Change History (1)

comment:1 Changed on Mar 24, 2020 at 8:58:30 AM by dkocher

  • Milestone set to 7.3.0
  • Resolution set to fixed
  • Status changed from new to closed
  • Type changed from defect to enhancement

In r48801. We do not want the chunk size to be too large as it uses system memory.

Note: See TracTickets for help on using tickets.