Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Swift S3 emulation: could not upload file bigger then 10MB #7496

Closed
cyberduck opened this issue Oct 22, 2013 · 3 comments
Closed

Swift S3 emulation: could not upload file bigger then 10MB #7496

cyberduck opened this issue Oct 22, 2013 · 3 comments
Assignees
@dkocher
Labels
bug fixed s3 AWS S3 Protocol Implementation
Milestone
4.4.1

Comments

@cyberduck
Copy link
Collaborator

da06755 created the issue

Hi, I have problem with uploading files bigger then 10MB to my swift cluster with s3 emulation (https://github.com/fujita/swift3).
When uploading file smaller then 10MB cyberduck generate PUT request and everything is ok, but when file > 10MB there is a bunch of POST request with ?uploads parameter which gets 503 responses from server (example of upload 100MB file):

 [Window Title]
Error

[Main Instruction]
Upload failed (/write/testfile100MB).

[Content]
Encountered too many Internal Server errors (6), aborting request.

[Expanded Information]
CONNECT example.com:443 HTTP/1.1
Host: example.com:443
Proxy-Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (13515) (Windows 7/6.1) (x86)
HTTP/1.1 200 Connection Established
FiddlerGateway: Direct
StartTime: 18:45:13.939
Connection: close
GET / HTTP/1.1
Date: Tue, 22 Oct 2013 16:45:13 GMT
Authorization: AWS TestUser1234:admin:rvyjGd0jqRHXoifUJx9RLVcYW2Q=
Host: example.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (13515) (Windows 7/6.1) (x86)
HTTP/1.1 200 OK
Content-Length: 795
Content-Type: application/xml
X-Trans-Id: tx3bf0e548772a41e5bc8fa89856cbd7a2
Date: Tue, 22 Oct 2013 16:45:15 GMT
Connection: keep-alive
HEAD /write/testfile100MB HTTP/1.1
Date: Tue, 22 Oct 2013 16:45:15 GMT
Authorization: AWS TestUser1234:admin:NVdo4WwBPlvgMOrnCusm3M4QCAM=
Host: example.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (13515) (Windows 7/6.1) (x86)
HTTP/1.1 404 Not Found
Content-Length: 149
Content-Type: text/xml
X-Trans-Id: tx738f9abceef7471ba6d6654362c78d58
Date: Tue, 22 Oct 2013 16:45:15 GMT
Connection: keep-alive
HEAD /write/testfile100MB HTTP/1.1
Date: Tue, 22 Oct 2013 16:45:15 GMT
Authorization: AWS TestUser1234:admin:NVdo4WwBPlvgMOrnCusm3M4QCAM=
Host: example.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (13515) (Windows 7/6.1) (x86)
HTTP/1.1 404 Not Found
Content-Length: 149
Content-Type: text/xml
X-Trans-Id: tx16b49faa75804429b5d4ca00f2413577
Date: Tue, 22 Oct 2013 16:45:15 GMT
Connection: keep-alive
POST /write/testfile100MB?uploads HTTP/1.1
Date: Tue, 22 Oct 2013 16:45:15 GMT
Content-Type: application/octet-stream
Authorization: AWS TestUser1234:admin:w0PDP7/4vYjvEgsvqn12/yTk3vw=
Content-Length: 0
Host: example.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (13515) (Windows 7/6.1) (x86)
HTTP/1.1 503 Service Unavailable
Content-Length: 148
Content-Type: text/xml
X-Trans-Id: txfb609970cfad4d45849328e8584375d9
Date: Tue, 22 Oct 2013 16:45:15 GMT
Connection: keep-alive
POST /write/testfile100MB?uploads HTTP/1.1
Date: Tue, 22 Oct 2013 16:45:15 GMT
Content-Type: application/octet-stream
Authorization: AWS TestUser1234:admin:w0PDP7/4vYjvEgsvqn12/yTk3vw=
Content-Length: 0
Host: example.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (13515) (Windows 7/6.1) (x86)
HTTP/1.1 503 Service Unavailable
Content-Length: 148
Content-Type: text/xml
X-Trans-Id: txc330270b6b814e0dbfe995d74ac1cf4e
Date: Tue, 22 Oct 2013 16:45:15 GMT
Connection: keep-alive
POST /write/testfile100MB?uploads HTTP/1.1
Date: Tue, 22 Oct 2013 16:45:15 GMT
Content-Type: application/octet-stream
Authorization: AWS TestUser1234:admin:w0PDP7/4vYjvEgsvqn12/yTk3vw=
Content-Length: 0
Host: example.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (13515) (Windows 7/6.1) (x86)
HTTP/1.1 503 Service Unavailable
Content-Length: 148
Content-Type: text/xml
X-Trans-Id: txf917b7bc98e94bbdbc9dc991ae9445c1
Date: Tue, 22 Oct 2013 16:45:16 GMT
Connection: keep-alive
POST /write/testfile100MB?uploads HTTP/1.1
Date: Tue, 22 Oct 2013 16:45:16 GMT
Content-Type: application/octet-stream
Authorization: AWS TestUser1234:admin:qi9JL4fsX+vWTBZix14klLdTgkE=
Content-Length: 0
Host: example.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (13515) (Windows 7/6.1) (x86)
HTTP/1.1 503 Service Unavailable
Content-Length: 148
Content-Type: text/xml
X-Trans-Id: txe8adddff51894a64b60f4656eadddcd3
Date: Tue, 22 Oct 2013 16:45:16 GMT
Connection: keep-alive

With s3curl I can upload files bigger then 10MB. In stable version problem not exist.
@cyberduck
Copy link
Collaborator Author

@dkocher commented

This is multipart upload we are using to make uploads resumable on S3. Refer to #6208.

@cyberduck
Copy link
Collaborator Author

da06755 commented

Do you think it is possible fallback to PUT request if POST /ObjectName?uploads doesn't return UploadId?

@cyberduck
Copy link
Collaborator Author

@dkocher commented

In 710bda0.

@iterate-ch iterate-ch locked as resolved and limited conversation to collaborators Nov 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@dkocher dkocher

Labels
bug fixed s3 AWS S3 Protocol Implementation
Projects
None yet
Milestone
4.4.1
Development

No branches or pull requests
2 participants
@dkocher @cyberduck