Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing Large Object support #6056

Closed
cyberduck opened this issue Jul 7, 2011 · 18 comments
Closed

Missing Large Object support #6056

cyberduck opened this issue Jul 7, 2011 · 18 comments
Assignees
Labels
enhancement fixed openstack OpenStack Swift Protocol Implementation
Milestone

Comments

@cyberduck
Copy link
Collaborator

2f14288 created the issue

Hi,

I have just deployed a minimal OpenStack Swift Object Store v1.4.1 (5 storage nodes, 1 proxy server) and have been trying out CyberDuck from my Mac OS X Lion laptop. It all works fine with smaller files but when I try to upload a large file (6.1GiB, i.e. above the 5GiB single object limit in Swift) CyberDuck tried to upload the whole file as a single object - at least it appears so from the log on the swift proxy server which logged this:

Jul  7 09:55:16 swift proxy-server 131.111.10.57 131.111.10.57 07/Jul/2011/08/55/16 PUT /v1/AUTH_uxsup/UxSup-Test-Container/SnowLeopard_10A432_UserDVD_2Z693-6990-068.dmg HTTP/1.0 413 - Cyberduck/4.0.2%20%28Mac%20OS%20X/10.7%29%20%28i386%29 uxsup%2CAUTH_FOOBARAUTHKEYREPLACED - - - - - 214.0182

(Note I replaced the authentication token with FOOBARAUTHKEYREPLACED for obvious reasons!)

And cyberduck threw up an error message saying:

HTTP Error: Upload failed
Unexpected result.  Request Entity Too Large.

You can see an image of the error message and the activity window showing the 6.1GiB file that it uploaded unsuccessfully in the attachment.

As far as I can see large files must be uploaded in a segmented fashion as described here:

(http://swift.openstack.org/overview_large_objects.html)(http://swift.openstack.org/overview_large_objects.html)

Not sure whether to class this as a bug or feature request but given it results in an error I will submit it as a "defect" for now... Also, the Platform is Mac OS X 10.7 (GM, not a beta) and not 10.6 like I entered in the Platform field but you do not list 10.7 amongst the choices and 10.6 was the closest match. In any case I don't see how that would matter!

It would be great if this could be fixed/implemented so large file uploads work.

Thanks a lot in advance!

Best regards,

Anton


Attachments

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Thanks for the detailed bug report. Segmented uploads are not currently supported.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Milestone 4.2 deleted

@cyberduck
Copy link
Collaborator Author

@dkocher commented

#7069 closed as duplicate.

@cyberduck
Copy link
Collaborator Author

e2f2ab4 commented

In version 4.3.1 on Mac OS X , downloaded a segmented file (does not need to be larger than 5GB) from Openstack swift results in a 'Transfer incomplete' message even though the file is transferred successfully. I tested with a 1MB file which was uploaded in 500KB segments using the Dynamic Large Object (DLO) feature.

The Openstack swift proxy server handles sending clients all the segments of a file transparently when a user requests a manifest file (which represents a segmented upload).

I would imagine this is because the manifest file on the object store is 0 bytes is length (as reported correctly by Cyberduck in the file listing) but the actual file that is downloaded is > 0 bytes.

Would this issue be considered as part of #6056 or should it be split into a separate ticket ?

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Replying to [comment:7 david cole]:

In version 4.3.1 on Mac OS X , downloaded a segmented file (does not need to be larger than 5GB) from Openstack swift results in a 'Transfer incomplete' message even though the file is transferred successfully. I tested with a 1MB file which was uploaded in 500KB segments using the Dynamic Large Object (DLO) feature.

The Openstack swift proxy server handles sending clients all the segments of a file transparently when a user requests a manifest file (which represents a segmented upload).

I would imagine this is because the manifest file on the object store is 0 bytes is length (as reported correctly by Cyberduck in the file listing) but the actual file that is downloaded is > 0 bytes.

Would this issue be considered as part of #6056 or should it be split into a separate ticket ?

Handling manifest files for Dynamic Large Objects would be a separate issue from segmented uploads then. I assume Static Large Objects will work as the documentation says

The response’s Content-Length for a GET or HEAD on the manifest file will be the sum of all the segments in the / listing.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Referring to #7211. Does jclouds handle this transparently?

@cyberduck
Copy link
Collaborator Author

e2f2ab4 commented

Will you accept a patch for large file support to https://github.com/dkocher/java-openstack-swift ? We've not written it yet, but I want check if it's okay in principle !

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Replying to [comment:10 david cole]:

Will you accept a patch for large file support to https://github.com/dkocher/java-openstack-swift ? We've not written it yet, but I want check if it's okay in principle !

Sure! That would be fantastic.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Current API documentation.

@cyberduck
Copy link
Collaborator Author

e2f2ab4 commented

We think this should be implemented in two phases:

(1) Single TCP streams. On upload, split the large files into adjustable size pieces (segments) and each segment is uploaded serially. A JSON user defined manifest is then uploaded on the successful upload of all segments. This is the Static Large Object type of segmentation.

(2) Multiple TCP streams. On upload, split the large files into adjustable size pieces (segments) and each segment is uploaded in parallel. A JSON user defined manifest is then uploaded on the successful upload of all segments. This is the Static Large Object type of segmentation.

A PoC of phase 1 is what we are working towards in the short term.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Replying to [comment:13 david cole]:

We think this should be implemented in two phases:

+1

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Replying to [comment:13 david cole]:

We think this should be implemented in two phases:

Progress so far?

@cyberduck
Copy link
Collaborator Author

e2f2ab4 commented

Replying to [comment:15 dkocher]:

Replying to [comment:13 david cole]:

We think this should be implemented in two phases:

Progress so far?

Sorry for going quiet. We have been through the python-swiftclient code to understand the special steps required on upload & delete - in particular what to do if you upload a standard or segmented object over an existing segmented upload. We should have something for you in a week or so.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Ongoing work in this fork.

@cyberduck
Copy link
Collaborator Author

4f2e4bd commented

Replying to [comment:17 dkocher]:

Ongoing work in this fork.


Progress update:

I have made quite a bit of progress here by creating a new branch (phase2) in the above repository in which I've separated out some useful methods for dealing with large objects. Separately I have a checkout of the Cyberduck Subversion repository in which I have added a SwiftThresholdUploadService, a SwiftSmallObjectUploadFeature and a SwiftLargeObjectUploadFeature. I have also modified the SwiftDeleteFeature to remove large object segments. The behaviour of each of these features is configurable through the Preferences.

I am currently going through some testing, which is throwing up some interesting problems. All the features seem to be working against a devstack vm on my laptop, but I have hit some problems related to bulk deletes when testing against another system. Will update here when I've worked out what the issues are.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Replying to [comment:18 joel-wright]:

Replying to [comment:17 dkocher]:

Ongoing work in this fork.


Progress update:

I have made quite a bit of progress here by creating a new branch (phase2) in the above repository in which I've separated out some useful methods for dealing with large objects. Separately I have a checkout of the Cyberduck Subversion repository in which I have added a SwiftThresholdUploadService, a SwiftSmallObjectUploadFeature and a SwiftLargeObjectUploadFeature. I have also modified the SwiftDeleteFeature to remove large object segments. The behaviour of each of these features is configurable through the Preferences.

I am currently going through some testing, which is throwing up some interesting problems. All the features seem to be working against a devstack vm on my laptop, but I have hit some problems related to bulk deletes when testing against another system. Will update here when I've worked out what the issues are.

We can disable bulk deletes if required, this is not a blocker.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Added implementation in 4ad0539 and test in 54956bd.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Enabled in 985e573.

@iterate-ch iterate-ch locked as resolved and limited conversation to collaborators Nov 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement fixed openstack OpenStack Swift Protocol Implementation
Projects
None yet
Development

No branches or pull requests

2 participants