Cyberduck Mountain Duck CLI

#9636 closed defect (worksforme)

Wrong size of large file manifest using swift

Reported by: selectel Owned by:
Priority: normal Milestone: 5.1
Component: openstack Version: 5.0.9
Severity: normal Keywords: manifest swift large file
Cc: support@…, dolgov@…, melnikov@…, dzhurinskij@… Architecture: Intel
Platform:

Description

Hello.

We have some problems with large files downloading process using cyberduck in our cloud storage (openstack swift). File size is ~20G (20000000000). Uploading process ends successfully, but downloading is impossible, because of wrong size of manifest file (4,0 KiB) in cyberduck client - http://slc.tl/zhy6t If we get HEAD info with python-swiftclient or an API everything is correct ( Content Length: 20000000000).

$ swift stat -U XXXX_cyberduck -K XXXXX -A http://api.selcdn.ru/auth/v1.0 cyberducktest 20G
                      Account: SEL_XXXXX
                    Container: cyberducktest
                       Object: 20G
                 Content Type: text/plain; charset=utf-8
               Content Length: 20000000000
                         ETag: 80139eeb4bc466509b6fed49a1475837
Access-Control-Expose-Headers: X-Backend-Timestamp, Etag, Last-Modified, X-Object-Manifest, X-Timestamp
        X-Static-Large-Object: True

Thank you.

Change History (5)

comment:1 Changed on Jul 28, 2016 at 1:04:49 PM by dkocher

  • Milestone set to 5.1
  • Resolution set to worksforme
  • Status changed from new to closed

To me everything looks correct as described in Large object support.

A manifest object links the segment objects into one logical large object. When you download a manifest object, Object Storage concatenates and returns the contents of the segment objects in the response body of the request. The manifest object types are:

You will find the parts with path and size_bytes in the manifest files referenced. The manifest file itself is a JSON file referencing the large upload parts. Swift will transparently give you back the full content when requesting the manifest file with a GET request (Or as you mention yourself when doing a HEADrequest you get the 20G object size reported.

comment:2 Changed on Jul 28, 2016 at 1:39:08 PM by selectel

  • Resolution worksforme deleted
  • Status changed from closed to reopened

To me everything looks correct as described in Large object support.

Yes, but if you try to download this file by cyberduck you'll get only 4K file, not 20G - http://slc.tl/2uykc

comment:3 Changed on Jul 28, 2016 at 2:43:32 PM by dkocher

Can you confirm the manifest file has the X-Static-Large-Object: True header set?

comment:4 Changed on Jul 28, 2016 at 2:44:47 PM by dkocher

The `GET` for the manifest file should be handled transparently by the server and return the large object.

A GET request to the manifest object will return the concatenation of the objects from the manifest…

comment:5 Changed on Jul 28, 2016 at 2:46:16 PM by dkocher

  • Resolution set to worksforme
  • Status changed from reopened to closed

I cannot reproduce this issue here with a ~3.1GB file uploaded with 3 segments.

GET /v1/MossoCloudFS_59113590-c679-46c3-bf62-9d7c3d5176ee/test.cyberduck.ch/GOPR0110.MP4 HTTP/1.1
X-Auth-Token: –
Host: storage101.ord1.clouddrive.com
Connection: Keep-Alive
User-Agent: Cyberduck/5.1.0.20695 (Mac OS X/10.12) (x86_64)
HTTP/1.1 200 OK
Content-Length: 3084438505
Accept-Ranges: bytes
Last-Modified: Thu, 28 Jul 2016 14:44:19 GMT
Etag: "f24d12f7d8122318e737b7add0119026"
X-Timestamp: 1469717058.74802
X-Static-Large-Object: True
Content-Type: video/mp4
X-Trans-Id: tx2720462080b144c893e02-00579a1a70ord1
Date: Thu, 28 Jul 2016 14:45:04 GMT

Note: See TracTickets for help on using tickets.
swiss made software