Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Wrong size of large file manifest using swift #9636

Closed
cyberduck opened this issue Jul 28, 2016 · 5 comments
Closed

Wrong size of large file manifest using swift #9636

cyberduck opened this issue Jul 28, 2016 · 5 comments
Labels
bug openstack OpenStack Swift Protocol Implementation worksforme
Milestone
5.1

Comments

@cyberduck
Copy link
Collaborator

6b3e1e0 created the issue

Hello.

We have some problems with large files downloading process using cyberduck in our cloud storage (openstack swift).
File size is ~20G (20000000000).
Uploading process ends successfully, but downloading is impossible, because of wrong size of manifest file (4,0 KiB) in cyberduck client - (http://slc.tl/zhy6t)
If we get HEAD info with python-swiftclient or an API everything is correct ( Content Length: 20000000000).

$ swift stat -U XXXX_cyberduck -K XXXXX -A http://api.selcdn.ru/auth/v1.0 cyberducktest 20G
                      Account: SEL_XXXXX
                    Container: cyberducktest
                       Object: 20G
                 Content Type: text/plain; charset=utf-8
               Content Length: 20000000000
                         ETag: 80139eeb4bc466509b6fed49a1475837
Access-Control-Expose-Headers: X-Backend-Timestamp, Etag, Last-Modified, X-Object-Manifest, X-Timestamp
        X-Static-Large-Object: True

Thank you.
@cyberduck
Copy link
Collaborator Author

@dkocher commented

To me everything looks correct as described in Large object support.

A manifest object links the segment objects into one logical large object. When you download a manifest object, Object Storage concatenates and returns the contents of the segment objects in the response body of the request. The manifest object types are:

You will find the parts with path and size_bytes in the manifest files referenced. The manifest file itself is a JSON file referencing the large upload parts. Swift will transparently give you back the full content when requesting the manifest file with a GET request (Or as you mention yourself when doing a HEADrequest you get the 20G object size reported.

@cyberduck
Copy link
Collaborator Author

6b3e1e0 commented

To me everything looks correct as described in Large object support.
Yes, but if you try to download this file by cyberduck you'll get only 4K file, not 20G - http://slc.tl/2uykc

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Can you confirm the manifest file has the X-Static-Large-Object: True header set?

@cyberduck
Copy link
Collaborator Author

@dkocher commented

The GET for the manifest file should be handled transparently by the server and return the large object.

A GET request to the manifest object will return the concatenation of the objects from the manifest…

@cyberduck
Copy link
Collaborator Author

@dkocher commented

I cannot reproduce this issue here with a ~3.1GB file uploaded with 3 segments.

GET /v1/MossoCloudFS_59113590-c679-46c3-bf62-9d7c3d5176ee/test.cyberduck.ch/GOPR0110.MP4 HTTP/1.1
X-Auth-Token: –
Host: storage101.ord1.clouddrive.com
Connection: Keep-Alive
User-Agent: Cyberduck/5.1.0.20695 (Mac OS X/10.12) (x86_64)
HTTP/1.1 200 OK
Content-Length: 3084438505
Accept-Ranges: bytes
Last-Modified: Thu, 28 Jul 2016 14:44:19 GMT
Etag: "f24d12f7d8122318e737b7add0119026"
X-Timestamp: 1469717058.74802
X-Static-Large-Object: True
Content-Type: video/mp4
X-Trans-Id: tx2720462080b144c893e02-00579a1a70ord1
Date: Thu, 28 Jul 2016 14:45:04 GMT

@iterate-ch iterate-ch locked as resolved and limited conversation to collaborators Nov 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug openstack OpenStack Swift Protocol Implementation worksforme
Projects
None yet
Milestone
5.1
Development

No branches or pull requests
1 participant
@cyberduck