Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase OneDrive multipart chunk size #11000

Closed
cyberduck opened this issue Mar 24, 2020 · 1 comment
Closed

Increase OneDrive multipart chunk size #11000

cyberduck opened this issue Mar 24, 2020 · 1 comment
Labels
enhancement fixed onedrive OneDrive Protocol Implementation
Milestone

Comments

@cyberduck
Copy link
Collaborator

af75c61 created the issue

Hello,

Just wondering if there are any plans to increase the OneDrive multipart chunk size?

The official client syncs quite fast at around 17MB/s. Uploading the same file using Cyberduck hovers at around 6MB/s (with the change of the "factor" introduced recently as in

this.setDefault("onedrive.upload.multipart.partsize.factor", String.valueOf(20)); // ~6,25 MB
) - my current build (7.2.5) doesn't seem to have it implemented so I just simulated it by setting the "onedrive.upload.multipart.partsize.minimum" value to minimum*20.

After setting the "minimum" part size setting to a ridiculously high value like 480000000 I am then able to get roughly the same speeds as the official client. This causes some weird behaviour with the transfer rate estimation and progress bar of Cyberduck as well as a "Java heap error" (probably out of memory?) when trying to upload a 1GB file. Reducing the value to half that (240000000) made the 1GB file upload fine and still had a speed comparable to the official client.

Is there any chance we can up this value further, potentially depending on the file size? The official docs set an upper limit of 60MB per chunk: https://docs.microsoft.com/en-us/graph/api/driveitem-createuploadsession?view=graph-rest-1.0#upload-bytes-to-the-upload-session - with your experience interfacing with different cloud storage providers, is there any reason not to use the max chunk size a provider allows, and what determines an optimal chunk size?

Regards.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

In 3a72c7f. We do not want the chunk size to be too large as it uses system memory.

@iterate-ch iterate-ch locked as resolved and limited conversation to collaborators Nov 27, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement fixed onedrive OneDrive Protocol Implementation
Projects
None yet
Development

No branches or pull requests

1 participant