Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Files greater than 100GB fail to upload #9131

Closed
cyberduck opened this issue Nov 24, 2015 · 4 comments
Closed

Files greater than 100GB fail to upload #9131

cyberduck opened this issue Nov 24, 2015 · 4 comments
Assignees
Labels
bug fixed high priority openstack OpenStack Swift Protocol Implementation
Milestone

Comments

@cyberduck
Copy link
Collaborator

d67cfcd created the issue

Cyberduck has an arbitrary 100GB Object Size limit.

-Observed Behavior:*

When uploading an object larger than 100GB to Swift Storage, all segments will be transferred but the complete operation will fail at 100%.

Request Entity Too Large. 413 Request Entity Too Large

The Object manifest is not created.

-It appears to be related to the design decision below:*

We changed the part size for multipart uploads to 10MB in 7b77f76 to allow multipart uploads up to 100GB in total size due to the maximum number of parts restriction of 10'000 by S3. Please try with the latest ​snapshot build available and reopen this ticket if you are still having this issue.

-Cause:*

From my understanding you're failing at objects greater than 100GB to prevent more than 10,000 segment being created (an S3 limit).
OpenStack Swift does not have a hard upper limit on the total number of objects stored in a container.

-Workaround:*

I've been able to override the default user setting: openstack.upload.largeobject.size to 1,048,576,000 (1GB)
which is approximately the default segment size used by the Python Swift Client and increasing the Cyberduck object size limit to 1TB.

Manually editing the Cyberduck user.config xml file is a less than desirable solution for our users, who routinely upload 100GB or larger files.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Thanks for your detailed bug report.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

See also #9134.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

The 413 error response is possibly due to the large manifest files given the thousands of segment names.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Revert in 18555.

@iterate-ch iterate-ch locked as resolved and limited conversation to collaborators Nov 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug fixed high priority openstack OpenStack Swift Protocol Implementation
Projects
None yet
Development

No branches or pull requests

2 participants