Cyberduck Mountain Duck CLI

#7496 closed defect (fixed)

Swift S3 emulation: could not upload file bigger then 10MB

Reported by: din Owned by: dkocher
Priority: normal Milestone: 4.4.1
Component: s3 Version: Nightly Build
Severity: normal Keywords:
Cc: Architecture: Intel
Platform: Windows 7

Description

Hi, I have problem with uploading files bigger then 10MB to my swift cluster with s3 emulation https://github.com/fujita/swift3. When uploading file smaller then 10MB cyberduck generate PUT request and everything is ok, but when file > 10MB there is a bunch of POST request with ?uploads parameter which gets 503 responses from server (example of upload 100MB file):

 [Window Title]
Error

[Main Instruction]
Upload failed (/write/testfile100MB).

[Content]
Encountered too many Internal Server errors (6), aborting request.

[Expanded Information]
CONNECT example.com:443 HTTP/1.1
Host: example.com:443
Proxy-Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (13515) (Windows 7/6.1) (x86)
HTTP/1.1 200 Connection Established
FiddlerGateway: Direct
StartTime: 18:45:13.939
Connection: close
GET / HTTP/1.1
Date: Tue, 22 Oct 2013 16:45:13 GMT
Authorization: AWS TestUser1234:admin:rvyjGd0jqRHXoifUJx9RLVcYW2Q=
Host: example.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (13515) (Windows 7/6.1) (x86)
HTTP/1.1 200 OK
Content-Length: 795
Content-Type: application/xml
X-Trans-Id: tx3bf0e548772a41e5bc8fa89856cbd7a2
Date: Tue, 22 Oct 2013 16:45:15 GMT
Connection: keep-alive
HEAD /write/testfile100MB HTTP/1.1
Date: Tue, 22 Oct 2013 16:45:15 GMT
Authorization: AWS TestUser1234:admin:NVdo4WwBPlvgMOrnCusm3M4QCAM=
Host: example.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (13515) (Windows 7/6.1) (x86)
HTTP/1.1 404 Not Found
Content-Length: 149
Content-Type: text/xml
X-Trans-Id: tx738f9abceef7471ba6d6654362c78d58
Date: Tue, 22 Oct 2013 16:45:15 GMT
Connection: keep-alive
HEAD /write/testfile100MB HTTP/1.1
Date: Tue, 22 Oct 2013 16:45:15 GMT
Authorization: AWS TestUser1234:admin:NVdo4WwBPlvgMOrnCusm3M4QCAM=
Host: example.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (13515) (Windows 7/6.1) (x86)
HTTP/1.1 404 Not Found
Content-Length: 149
Content-Type: text/xml
X-Trans-Id: tx16b49faa75804429b5d4ca00f2413577
Date: Tue, 22 Oct 2013 16:45:15 GMT
Connection: keep-alive
POST /write/testfile100MB?uploads HTTP/1.1
Date: Tue, 22 Oct 2013 16:45:15 GMT
Content-Type: application/octet-stream
Authorization: AWS TestUser1234:admin:w0PDP7/4vYjvEgsvqn12/yTk3vw=
Content-Length: 0
Host: example.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (13515) (Windows 7/6.1) (x86)
HTTP/1.1 503 Service Unavailable
Content-Length: 148
Content-Type: text/xml
X-Trans-Id: txfb609970cfad4d45849328e8584375d9
Date: Tue, 22 Oct 2013 16:45:15 GMT
Connection: keep-alive
POST /write/testfile100MB?uploads HTTP/1.1
Date: Tue, 22 Oct 2013 16:45:15 GMT
Content-Type: application/octet-stream
Authorization: AWS TestUser1234:admin:w0PDP7/4vYjvEgsvqn12/yTk3vw=
Content-Length: 0
Host: example.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (13515) (Windows 7/6.1) (x86)
HTTP/1.1 503 Service Unavailable
Content-Length: 148
Content-Type: text/xml
X-Trans-Id: txc330270b6b814e0dbfe995d74ac1cf4e
Date: Tue, 22 Oct 2013 16:45:15 GMT
Connection: keep-alive
POST /write/testfile100MB?uploads HTTP/1.1
Date: Tue, 22 Oct 2013 16:45:15 GMT
Content-Type: application/octet-stream
Authorization: AWS TestUser1234:admin:w0PDP7/4vYjvEgsvqn12/yTk3vw=
Content-Length: 0
Host: example.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (13515) (Windows 7/6.1) (x86)
HTTP/1.1 503 Service Unavailable
Content-Length: 148
Content-Type: text/xml
X-Trans-Id: txf917b7bc98e94bbdbc9dc991ae9445c1
Date: Tue, 22 Oct 2013 16:45:16 GMT
Connection: keep-alive
POST /write/testfile100MB?uploads HTTP/1.1
Date: Tue, 22 Oct 2013 16:45:16 GMT
Content-Type: application/octet-stream
Authorization: AWS TestUser1234:admin:qi9JL4fsX+vWTBZix14klLdTgkE=
Content-Length: 0
Host: example.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (13515) (Windows 7/6.1) (x86)
HTTP/1.1 503 Service Unavailable
Content-Length: 148
Content-Type: text/xml
X-Trans-Id: txe8adddff51894a64b60f4656eadddcd3
Date: Tue, 22 Oct 2013 16:45:16 GMT
Connection: keep-alive

With s3curl I can upload files bigger then 10MB. In stable version problem not exist.

Change History (4)

comment:1 Changed on Oct 22, 2013 at 5:47:20 PM by dkocher

This is multipart upload we are using to make uploads resumable on S3. Refer to #6208.

comment:2 Changed on Oct 23, 2013 at 8:07:44 AM by din

Do you think it is possible fallback to PUT request if POST /ObjectName?uploads doesn't return UploadId?

comment:3 Changed on Oct 23, 2013 at 8:08:36 AM by dkocher

  • Milestone changed from 4.4 to 4.4.1
  • Status changed from new to assigned

comment:4 Changed on Oct 28, 2013 at 2:12:54 PM by dkocher

  • Resolution set to fixed
  • Status changed from assigned to closed

In r13618.

Note: See TracTickets for help on using tickets.
swiss made software