Cyberduck Mountain Duck CLI

Opened 7 years ago

Closed 5 years ago

#6056 closed enhancement (fixed)

Missing Large Object support

Reported by: Anton Altaparmakov Owned by: dkocher
Priority: normal Milestone: 4.4.1
Component: openstack Version: 4.0.2
Severity: critical Keywords: OpenStack Swift Large Object Upload
Cc: Architecture: Intel
Platform: Mac OS X 10.6

Description (last modified by dkocher)

Hi,

I have just deployed a minimal OpenStack Swift Object Store v1.4.1 (5 storage nodes, 1 proxy server) and have been trying out CyberDuck from my Mac OS X Lion laptop. It all works fine with smaller files but when I try to upload a large file (6.1GiB, i.e. above the 5GiB single object limit in Swift) CyberDuck tried to upload the whole file as a single object - at least it appears so from the log on the swift proxy server which logged this:

Jul  7 09:55:16 swift proxy-server 131.111.10.57 131.111.10.57 07/Jul/2011/08/55/16 PUT /v1/AUTH_uxsup/UxSup-Test-Container/SnowLeopard_10A432_UserDVD_2Z693-6990-068.dmg HTTP/1.0 413 - Cyberduck/4.0.2%20%28Mac%20OS%20X/10.7%29%20%28i386%29 uxsup%2CAUTH_FOOBARAUTHKEYREPLACED - - - - - 214.0182

(Note I replaced the authentication token with FOOBARAUTHKEYREPLACED for obvious reasons!)

And cyberduck threw up an error message saying:

HTTP Error: Upload failed
Unexpected result.  Request Entity Too Large.

You can see an image of the error message and the activity window showing the 6.1GiB file that it uploaded unsuccessfully in the attachment.

As far as I can see large files must be uploaded in a segmented fashion as described here:

http://swift.openstack.org/overview_large_objects.html

Not sure whether to class this as a bug or feature request but given it results in an error I will submit it as a "defect" for now... Also, the Platform is Mac OS X 10.7 (GM, not a beta) and not 10.6 like I entered in the Platform field but you do not list 10.7 amongst the choices and 10.6 was the closest match. In any case I don't see how that would matter!

It would be great if this could be fixed/implemented so large file uploads work.

Thanks a lot in advance!

Best regards,

Anton

Attachments (1)

SwiftLargeUploadFailure.jpg (59.0 KB) - added by Anton Altaparmakov 7 years ago.
Error message from CyberDuck on uploading 6.1GiB file to Swift.

Download all attachments as: .zip

Change History (22)

Changed 7 years ago by Anton Altaparmakov

Error message from CyberDuck on uploading 6.1GiB file to Swift.

comment:1 Changed 7 years ago by dkocher

  • Description modified (diff)

comment:2 Changed 7 years ago by dkocher

Thanks for the detailed bug report. Segmented uploads are not currently supported.

comment:3 Changed 7 years ago by dkocher

  • Milestone set to 4.2
  • Summary changed from Swift Large Object support missing? to Missing Large Object support

comment:4 Changed 7 years ago by dkocher

  • Type changed from defect to enhancement

comment:5 Changed 7 years ago by dkocher

  • Milestone changed from 4.2 to 4.3

Milestone 4.2 deleted

comment:6 Changed 6 years ago by dkocher

#7069 closed as duplicate.

comment:7 follow-up: Changed 5 years ago by david cole

In version 4.3.1 on Mac OS X , downloaded a segmented file (does not need to be larger than 5GB) from Openstack swift results in a 'Transfer incomplete' message even though the file is transferred successfully. I tested with a 1MB file which was uploaded in 500KB segments using the Dynamic Large Object (DLO) feature.

The Openstack swift proxy server handles sending clients all the segments of a file transparently when a user requests a manifest file (which represents a segmented upload).

I would imagine this is because the manifest file on the object store is 0 bytes is length (as reported correctly by Cyberduck in the file listing) but the actual file that is downloaded is > 0 bytes.

Would this issue be considered as part of #6056 or should it be split into a separate ticket ?

comment:8 in reply to: ↑ 7 Changed 5 years ago by dkocher

Replying to david cole:

In version 4.3.1 on Mac OS X , downloaded a segmented file (does not need to be larger than 5GB) from Openstack swift results in a 'Transfer incomplete' message even though the file is transferred successfully. I tested with a 1MB file which was uploaded in 500KB segments using the Dynamic Large Object (DLO) feature.

The Openstack swift proxy server handles sending clients all the segments of a file transparently when a user requests a manifest file (which represents a segmented upload).

I would imagine this is because the manifest file on the object store is 0 bytes is length (as reported correctly by Cyberduck in the file listing) but the actual file that is downloaded is > 0 bytes.

Would this issue be considered as part of #6056 or should it be split into a separate ticket ?

Handling manifest files for Dynamic Large Objects would be a separate issue from segmented uploads then. I assume Static Large Objects will work as the documentation says

The response’s Content-Length for a GET or HEAD on the manifest file will be the sum of all the segments in the <container>/<prefix> listing.

comment:9 Changed 5 years ago by dkocher

Referring to #7211. Does jclouds handle this transparently?

comment:10 follow-up: Changed 5 years ago by david cole

Will you accept a patch for large file support to https://github.com/dkocher/java-openstack-swift ? We've not written it yet, but I want check if it's okay in principle !

comment:11 in reply to: ↑ 10 Changed 5 years ago by dkocher

Replying to david cole:

Will you accept a patch for large file support to https://github.com/dkocher/java-openstack-swift ? We've not written it yet, but I want check if it's okay in principle !

Sure! That would be fantastic.

comment:12 Changed 5 years ago by dkocher

comment:13 follow-ups: Changed 5 years ago by david cole

We think this should be implemented in two phases:

(1) Single TCP streams. On upload, split the large files into adjustable size pieces (segments) and each segment is uploaded serially. A JSON user defined manifest is then uploaded on the successful upload of all segments. This is the Static Large Object type of segmentation.

(2) Multiple TCP streams. On upload, split the large files into adjustable size pieces (segments) and each segment is uploaded in parallel. A JSON user defined manifest is then uploaded on the successful upload of all segments. This is the Static Large Object type of segmentation.

A PoC of phase 1 is what we are working towards in the short term.

comment:14 in reply to: ↑ 13 Changed 5 years ago by dkocher

Replying to david cole:

We think this should be implemented in two phases:

+1

comment:15 in reply to: ↑ 13 ; follow-up: Changed 5 years ago by dkocher

Replying to david cole:

We think this should be implemented in two phases:

Progress so far?

comment:16 in reply to: ↑ 15 Changed 5 years ago by david cole

Replying to dkocher:

Replying to david cole:

We think this should be implemented in two phases:

Progress so far?

Sorry for going quiet. We have been through the python-swiftclient code to understand the special steps required on upload & delete - in particular what to do if you upload a standard or segmented object over an existing segmented upload. We should have something for you in a week or so.

comment:17 follow-up: Changed 5 years ago by dkocher

Ongoing work in this fork.

comment:18 in reply to: ↑ 17 ; follow-up: Changed 5 years ago by joel-wright

Replying to dkocher:

Ongoing work in this fork.


Progress update:

I have made quite a bit of progress here by creating a new branch (phase2) in the above repository in which I've separated out some useful methods for dealing with large objects. Separately I have a checkout of the Cyberduck Subversion repository in which I have added a SwiftThresholdUploadService, a SwiftSmallObjectUploadFeature and a SwiftLargeObjectUploadFeature. I have also modified the SwiftDeleteFeature to remove large object segments. The behaviour of each of these features is configurable through the Preferences.

I am currently going through some testing, which is throwing up some interesting problems. All the features seem to be working against a devstack vm on my laptop, but I have hit some problems related to bulk deletes when testing against another system. Will update here when I've worked out what the issues are.

comment:19 in reply to: ↑ 18 Changed 5 years ago by dkocher

Replying to joel-wright:

Replying to dkocher:

Ongoing work in this fork.


Progress update:

I have made quite a bit of progress here by creating a new branch (phase2) in the above repository in which I've separated out some useful methods for dealing with large objects. Separately I have a checkout of the Cyberduck Subversion repository in which I have added a SwiftThresholdUploadService, a SwiftSmallObjectUploadFeature and a SwiftLargeObjectUploadFeature. I have also modified the SwiftDeleteFeature to remove large object segments. The behaviour of each of these features is configurable through the Preferences.

I am currently going through some testing, which is throwing up some interesting problems. All the features seem to be working against a devstack vm on my laptop, but I have hit some problems related to bulk deletes when testing against another system. Will update here when I've worked out what the issues are.

We can disable bulk deletes if required, this is not a blocker.

comment:20 Changed 5 years ago by dkocher

Added implementation in r13761 and test in r13766.

comment:21 Changed 5 years ago by dkocher

  • Milestone changed from 4.5 to 4.4.1
  • Resolution set to fixed
  • Status changed from new to closed

Enabled in r13782.

Note: See TracTickets for help on using tickets.
swiss made software