Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Error: S3 Server "Failed to Respond" Following 26 GB Upload #8697

Closed
cyberduck opened this issue Mar 25, 2015 · 12 comments
Closed

Error: S3 Server "Failed to Respond" Following 26 GB Upload #8697

cyberduck opened this issue Mar 25, 2015 · 12 comments
Assignees
@dkocher
Labels
bug duplicate s3 AWS S3 Protocol Implementation
Milestone
4.7

Comments

@cyberduck
Copy link
Collaborator

9ba9865 created the issue

Hello,

Big fans of Cyberduck! For work, it's very important that we be able to upload files >10GB to our S3 server. Cyberduck worked great for this on our old system, but on a new MacPro with Yosemite, we keep getting a perplexing error message.

Every time the upload is almost finished (99% or more) it gives us an error message that server "Failed to respond". This happens every time. Oddly, if I hit refresh the entire file does appear to be on the server, but I'm not sure if it's complete and functional. I've included the transcript below...

GET /?delimiter=%2F&max-keys=1000&prefix=31831%2F HTTP/1.1
Date: Wed, 25 Mar 2015 00:53:48 GMT
Authorization: AWS 0ERCJ8PGP9FCHANHFD82:mXvqNTbiOLAgzj7xCsMXfwKcZcE=
Host: mubi-encoder-source.s3.amazonaws.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.6.5 (Mac OS X/10.10.1) (x86_64)
GET /?delimiter=%2F&max-keys=1000&prefix=31831%2F HTTP/1.1
Date: Wed, 25 Mar 2015 00:53:48 GMT
Authorization: AWS 0ERCJ8PGP9FCHANHFD82:mXvqNTbiOLAgzj7xCsMXfwKcZcE=
Host: mubi-encoder-source.s3.amazonaws.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.6.5 (Mac OS X/10.10.1) (x86_64)
HTTP/1.1 200 OK
x-amz-id-2: oUi1p7FQm/yeSipnS1tIhzQsUhsjlLcMfT5OnT1WVCCXC8zK0KA0n+sxXfVVFO+taqGcm/WTkLk=
x-amz-request-id: 401AC64CD027DD33
Date: Wed, 25 Mar 2015 00:53:50 GMT
Content-Type: application/xml
Transfer-Encoding: chunked
Server: AmazonS3
GET /?delimiter=%2F&max-keys=1000&prefix=31831%2F HTTP/1.1
Date: Wed, 25 Mar 2015 17:14:29 GMT
Authorization: AWS 0ERCJ8PGP9FCHANHFD82:v1RjsA2+YoOSFNf+3af+ad9gc3o=
Host: mubi-encoder-source.s3.amazonaws.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.6.5 (Mac OS X/10.10.1) (x86_64)
GET /?delimiter=%2F&max-keys=1000&prefix=31831%2F HTTP/1.1
Date: Wed, 25 Mar 2015 17:14:29 GMT
Authorization: AWS 0ERCJ8PGP9FCHANHFD82:v1RjsA2+YoOSFNf+3af+ad9gc3o=
Host: mubi-encoder-source.s3.amazonaws.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.6.5 (Mac OS X/10.10.1) (x86_64)
HTTP/1.1 200 OK
x-amz-id-2: PBPuVh0OKLkbpKpT/fcFeb3094Dh5HqiI4sqv3UOIlgl/LPI1qpq4lvAKSpsQjh6k2TBe6JBIWI=
x-amz-request-id: 9325FEAADABE0892
Date: Wed, 25 Mar 2015 17:14:31 GMT
Content-Type: application/xml
Transfer-Encoding: chunked
Server: AmazonS3

Thank you. Keep up the great work, and let us know if there's any info you need from us!
@cyberduck
Copy link
Collaborator Author

@dkocher commented

Do you get the error message in the Transfer or in the Browser window?

@cyberduck
Copy link
Collaborator Author

9ba9865 commented

Replying to [comment:2 dkocher]:

Do you get the error message in the Transfer or in the Browser window?

The Transfer window.

Thanks!

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Please try a7a75bd or later.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Reopening in response to #8731.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

We reverted changeset a7a75bd with 0d8b40a defaulting to use defaulting to enable connection reuse with HTTP Keep-Alive. The issue that connections were not validated prior reuse was supposed to be fixed with 157d5ce instead.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Set the hidden setting http.connections.reuse to false to disable the reuse of HTTP connections with Keep-Alive headers and let us know if this makes a difference. This would also apply if you use a build prior to 0d8b40a.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Replying to [8697 dag02005]:

Every time the upload is almost finished (99% or more) it gives us an error message that server "Failed to respond". This happens every time. Oddly, if I hit refresh the entire file does appear to be on the server, but I'm not sure if it's complete and functional.
If the file does appear in the file listing the multipart upload was successful and checksums verified.

@cyberduck
Copy link
Collaborator Author

9ba9865 commented

Thanks for the fast reply! I quit out of Cyberduck, and in the Terminal, I entered...

defaults write ch.sudo.cyberduck http.connections.reuse false

After relaunching Cyberduck, I tried the upload again and the error still happened.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Please post the transcript from the log drawer of the Transfers window. Choose ⌘-L on Mac or right-click the toolbar from the Transfers window and choose Log on Windows

@cyberduck
Copy link
Collaborator Author

9ba9865 commented

Sure thing. I tried to upload it again to reproduce the problem, and it gave a more complex, drawn-out version of the bug. Here's the transcript of the end...

Date: Thu, 09 Apr 2015 19:05:30 GMT
Expect: 100-continue
Content-Type: application/octet-stream
Authorization: AWS 0ERCJ8PGP9FCHANHFD82:+qVMfVguAofNl3CODyZT8HVb5S4=
Content-Length: 10485760
Host: mubi-encoder-source.s3.amazonaws.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.7 (Mac OS X/10.10.1) (x86_64)
PUT /31339%2Fim-still-here_en_xx_1920x1080_30723.mp4?uploadId=lg_Lht5_WOHUIWMjRkU.JKwjyI_SUBqN8Zf2TW3O4uACPRaqd5qsyHbzZkL4JEdH5xTWAhcKYTmF_sLzvfKfnw--&partNumber=1610 HTTP/1.1
Date: Thu, 09 Apr 2015 19:05:30 GMT
Expect: 100-continue
Content-Type: application/octet-stream
Authorization: AWS 0ERCJ8PGP9FCHANHFD82:iwlFb14KcZKf7C6q3SjPapZbX5k=
Content-Length: 10485760
Host: mubi-encoder-source.s3.amazonaws.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.7 (Mac OS X/10.10.1) (x86_64)
PUT /31339%2Fim-still-here_en_xx_1920x1080_30723.mp4?uploadId=lg_Lht5_WOHUIWMjRkU.JKwjyI_SUBqN8Zf2TW3O4uACPRaqd5qsyHbzZkL4JEdH5xTWAhcKYTmF_sLzvfKfnw--&partNumber=1611 HTTP/1.1
Date: Thu, 09 Apr 2015 19:05:31 GMT
Expect: 100-continue
Content-Type: application/octet-stream
Authorization: AWS 0ERCJ8PGP9FCHANHFD82:Jw4qYr/SrAMd3kspZPmv+hP/Biw=
Content-Length: 10485760
Host: mubi-encoder-source.s3.amazonaws.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.7 (Mac OS X/10.10.1) (x86_64)
PUT /31339%2Fim-still-here_en_xx_1920x1080_30723.mp4?uploadId=lg_Lht5_WOHUIWMjRkU.JKwjyI_SUBqN8Zf2TW3O4uACPRaqd5qsyHbzZkL4JEdH5xTWAhcKYTmF_sLzvfKfnw--&partNumber=1613 HTTP/1.1
Date: Thu, 09 Apr 2015 19:05:31 GMT
Expect: 100-continue
Content-Type: application/octet-stream
Authorization: AWS 0ERCJ8PGP9FCHANHFD82:ULBrj0jzSGvNXZL+r0z/JAAEwh4=
Content-Length: 10485760
Host: mubi-encoder-source.s3.amazonaws.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.7 (Mac OS X/10.10.1) (x86_64)
PUT /31339%2Fim-still-here_en_xx_1920x1080_30723.mp4?uploadId=lg_Lht5_WOHUIWMjRkU.JKwjyI_SUBqN8Zf2TW3O4uACPRaqd5qsyHbzZkL4JEdH5xTWAhcKYTmF_sLzvfKfnw--&partNumber=1614 HTTP/1.1
Date: Thu, 09 Apr 2015 19:05:31 GMT
Expect: 100-continue
Content-Type: application/octet-stream
Authorization: AWS 0ERCJ8PGP9FCHANHFD82:pjkXB46nzb+u2KXw00I/KAZZ9G4=
Content-Length: 10485760
Host: mubi-encoder-source.s3.amazonaws.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.7 (Mac OS X/10.10.1) (x86_64)
PUT /31339%2Fim-still-here_en_xx_1920x1080_30723.mp4?uploadId=lg_Lht5_WOHUIWMjRkU.JKwjyI_SUBqN8Zf2TW3O4uACPRaqd5qsyHbzZkL4JEdH5xTWAhcKYTmF_sLzvfKfnw--&partNumber=1612 HTTP/1.1
Date: Thu, 09 Apr 2015 19:05:31 GMT
Expect: 100-continue
Content-Type: application/octet-stream
Authorization: AWS 0ERCJ8PGP9FCHANHFD82:qL2Hz71cvP8fddJvh4PFox8cWNo=
Content-Length: 10485760
Host: mubi-encoder-source.s3.amazonaws.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.7 (Mac OS X/10.10.1) (x86_64)
HTTP/1.1 200 OK
x-amz-id-2: BO3Mtwq40EaRc2V6n16m8A22fHTLJKrYlvLlLuzBM1+kOHjN9gyG8smhBKG1VMq3tL65r0hEiRI=
x-amz-request-id: 026DF13E9B97DB92
Date: Thu, 09 Apr 2015 19:05:33 GMT
ETag: "1f0600ec3570148574781806e0d23fb7"
Content-Length: 0
Server: AmazonS3
PUT /31339%2Fim-still-here_en_xx_1920x1080_30723.mp4?uploadId=lg_Lht5_WOHUIWMjRkU.JKwjyI_SUBqN8Zf2TW3O4uACPRaqd5qsyHbzZkL4JEdH5xTWAhcKYTmF_sLzvfKfnw--&partNumber=1615 HTTP/1.1
Date: Thu, 09 Apr 2015 19:05:31 GMT
Expect: 100-continue
Content-Type: application/octet-stream
Authorization: AWS 0ERCJ8PGP9FCHANHFD82:waViQEDCJHIc9KgoVCnC9+3paew=
Content-Length: 10485760
Host: mubi-encoder-source.s3.amazonaws.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.7 (Mac OS X/10.10.1) (x86_64)
HTTP/1.1 200 OK
x-amz-id-2: 1rv0US1iT0soBN5gY0caKWhIEelIpT2BOioXt+uFWgW09k6wY7CkAIyt3P6bhrvYVAn0h+lbJdY=
x-amz-request-id: A906A17AC629536F
Date: Thu, 09 Apr 2015 19:05:27 GMT
ETag: "621dc2b1bc89e6b2b10f5c440b578669"
Content-Length: 0
Server: AmazonS3
PUT /31339%2Fim-still-here_en_xx_1920x1080_30723.mp4?uploadId=lg_Lht5_WOHUIWMjRkU.JKwjyI_SUBqN8Zf2TW3O4uACPRaqd5qsyHbzZkL4JEdH5xTWAhcKYTmF_sLzvfKfnw--&partNumber=1616 HTTP/1.1
Date: Thu, 09 Apr 2015 19:05:31 GMT
Expect: 100-continue
Content-Type: application/octet-stream
Authorization: AWS 0ERCJ8PGP9FCHANHFD82:Pq+6yfg2UlZxJsFXx1Qu7PsGfkY=
Content-Length: 10485760
Host: mubi-encoder-source.s3.amazonaws.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.7 (Mac OS X/10.10.1) (x86_64)
PUT /31339%2Fim-still-here_en_xx_1920x1080_30723.mp4?uploadId=lg_Lht5_WOHUIWMjRkU.JKwjyI_SUBqN8Zf2TW3O4uACPRaqd5qsyHbzZkL4JEdH5xTWAhcKYTmF_sLzvfKfnw--&partNumber=1613 HTTP/1.1
Date: Thu, 09 Apr 2015 19:05:31 GMT
Expect: 100-continue
Content-Type: application/octet-stream
Authorization: AWS 0ERCJ8PGP9FCHANHFD82:ULBrj0jzSGvNXZL+r0z/JAAEwh4=
Content-Length: 10485760
Host: mubi-encoder-source.s3.amazonaws.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.7 (Mac OS X/10.10.1) (x86_64)
HTTP/1.1 200 OK
x-amz-id-2: pY2ju8xfLG/r2Tt84swvI0FM+Zwm5FIkSpbmfvb6nBxRx4zr9ZiNUSFMpavM23WbgSvlP0FQ9BE=
x-amz-request-id: 1CDF868276781344
Date: Thu, 09 Apr 2015 19:05:33 GMT
ETag: "2f3fe830138656e3fa4a92f8ea1fc99d"
Content-Length: 0
Server: AmazonS3
PUT /31339%2Fim-still-here_en_xx_1920x1080_30723.mp4?uploadId=lg_Lht5_WOHUIWMjRkU.JKwjyI_SUBqN8Zf2TW3O4uACPRaqd5qsyHbzZkL4JEdH5xTWAhcKYTmF_sLzvfKfnw--&partNumber=1617 HTTP/1.1
Date: Thu, 09 Apr 2015 19:05:31 GMT
Expect: 100-continue
Content-Type: application/octet-stream
Authorization: AWS 0ERCJ8PGP9FCHANHFD82:7rO3sqezSbwwj22E2/fD1vZwmMg=
Content-Length: 10485760
Host: mubi-encoder-source.s3.amazonaws.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.7 (Mac OS X/10.10.1) (x86_64)
HTTP/1.1 200 OK
x-amz-id-2: 2+3EPSb9Ast4wWhDQrzPR5US34fLB4Fj3j4IXGK+8XRKkibW7FGCi3OM3IXUfRMNkk0vZr2z0G4=
x-amz-request-id: AB5B94E375B58B04
Date: Thu, 09 Apr 2015 19:05:50 GMT
ETag: "3ff605522c04a0b71c94f4454b3786c7"
Content-Length: 0
Server: AmazonS3
PUT /31339%2Fim-still-here_en_xx_1920x1080_30723.mp4?uploadId=lg_Lht5_WOHUIWMjRkU.JKwjyI_SUBqN8Zf2TW3O4uACPRaqd5qsyHbzZkL4JEdH5xTWAhcKYTmF_sLzvfKfnw--&partNumber=1618 HTTP/1.1
Date: Thu, 09 Apr 2015 19:05:31 GMT
Expect: 100-continue
Content-Type: application/octet-stream
Authorization: AWS 0ERCJ8PGP9FCHANHFD82:cJlAFR2zgG+1thFV4PNSdk0d7Zc=
Content-Length: 10485760
Host: mubi-encoder-source.s3.amazonaws.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.7 (Mac OS X/10.10.1) (x86_64)
HTTP/1.1 200 OK
x-amz-id-2: O7BWpQ7UaSOVeHYwfcf0Mi4Obr2kR4aRIBRmkIE+1YEya/sQ6bw+t+irQZdhAIXihA+l+CHbryA=
x-amz-request-id: E86EFC1925AD87A1
Date: Thu, 09 Apr 2015 19:05:48 GMT
ETag: "6edfc7bb6f85b5a5355d5871bb5d5792"
Content-Length: 0
Server: AmazonS3
PUT /31339%2Fim-still-here_en_xx_1920x1080_30723.mp4?uploadId=lg_Lht5_WOHUIWMjRkU.JKwjyI_SUBqN8Zf2TW3O4uACPRaqd5qsyHbzZkL4JEdH5xTWAhcKYTmF_sLzvfKfnw--&partNumber=1619 HTTP/1.1
Date: Thu, 09 Apr 2015 19:05:31 GMT
Expect: 100-continue
Content-Type: application/octet-stream
Authorization: AWS 0ERCJ8PGP9FCHANHFD82:38vXO8dKiRfYWyeJkdAeWghtnqM=
Content-Length: 10485760
Host: mubi-encoder-source.s3.amazonaws.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.7 (Mac OS X/10.10.1) (x86_64)
PUT /31339%2Fim-still-here_en_xx_1920x1080_30723.mp4?uploadId=lg_Lht5_WOHUIWMjRkU.JKwjyI_SUBqN8Zf2TW3O4uACPRaqd5qsyHbzZkL4JEdH5xTWAhcKYTmF_sLzvfKfnw--&partNumber=1620 HTTP/1.1
Date: Thu, 09 Apr 2015 19:05:32 GMT
Expect: 100-continue
Content-Type: application/octet-stream
Authorization: AWS 0ERCJ8PGP9FCHANHFD82:Pj68mjunbhHGQgm0XUihK1FASlk=
Content-Length: 10485760
Host: mubi-encoder-source.s3.amazonaws.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.7 (Mac OS X/10.10.1) (x86_64)
PUT /31339%2Fim-still-here_en_xx_1920x1080_30723.mp4?uploadId=lg_Lht5_WOHUIWMjRkU.JKwjyI_SUBqN8Zf2TW3O4uACPRaqd5qsyHbzZkL4JEdH5xTWAhcKYTmF_sLzvfKfnw--&partNumber=1614 HTTP/1.1
Date: Thu, 09 Apr 2015 19:05:32 GMT
Expect: 100-continue
Content-Type: application/octet-stream
Authorization: AWS 0ERCJ8PGP9FCHANHFD82:ms3ShLDTdaiJDzNUBZeYBTuEJaY=
Content-Length: 10485760
Host: mubi-encoder-source.s3.amazonaws.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.7 (Mac OS X/10.10.1) (x86_64)
PUT /31339%2Fim-still-here_en_xx_1920x1080_30723.mp4?uploadId=lg_Lht5_WOHUIWMjRkU.JKwjyI_SUBqN8Zf2TW3O4uACPRaqd5qsyHbzZkL4JEdH5xTWAhcKYTmF_sLzvfKfnw--&partNumber=1615 HTTP/1.1
Date: Thu, 09 Apr 2015 19:05:32 GMT
Expect: 100-continue
Content-Type: application/octet-stream
Authorization: AWS 0ERCJ8PGP9FCHANHFD82:6VaZvZODjILgIukOAOtR10r+//k=
Content-Length: 10485760
Host: mubi-encoder-source.s3.amazonaws.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.7 (Mac OS X/10.10.1) (x86_64)
PUT /31339%2Fim-still-here_en_xx_1920x1080_30723.mp4?uploadId=lg_Lht5_WOHUIWMjRkU.JKwjyI_SUBqN8Zf2TW3O4uACPRaqd5qsyHbzZkL4JEdH5xTWAhcKYTmF_sLzvfKfnw--&partNumber=1621 HTTP/1.1
Date: Thu, 09 Apr 2015 19:05:32 GMT
Expect: 100-continue
Content-Type: application/octet-stream
Authorization: AWS 0ERCJ8PGP9FCHANHFD82:yreJF3FXGUuhmMpmoyNqYMRVmX4=
Content-Length: 10485760
Host: mubi-encoder-source.s3.amazonaws.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.7 (Mac OS X/10.10.1) (x86_64)
PUT /31339%2Fim-still-here_en_xx_1920x1080_30723.mp4?uploadId=lg_Lht5_WOHUIWMjRkU.JKwjyI_SUBqN8Zf2TW3O4uACPRaqd5qsyHbzZkL4JEdH5xTWAhcKYTmF_sLzvfKfnw--&partNumber=1616 HTTP/1.1
Date: Thu, 09 Apr 2015 19:05:32 GMT
Expect: 100-continue
Content-Type: application/octet-stream
Authorization: AWS 0ERCJ8PGP9FCHANHFD82:S7Y9PF0VjH0Qm8Zm2v+/i362Huk=
Content-Length: 10485760
Host: mubi-encoder-source.s3.amazonaws.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.7 (Mac OS X/10.10.1) (x86_64)
HTTP/1.1 200 OK
x-amz-id-2: TGJVWKWKUZYINtxdlx0mwthClQj8Hr4Rn9e9OwhVJPxNJfCBqZxCl6FCqi73qhypHstbuRNSH3U=
x-amz-request-id: D56FEE768AE769B0
Date: Thu, 09 Apr 2015 19:05:53 GMT
ETag: "5ea3cc27c5d35bed0cbed8ad2f5093c1"
Content-Length: 0
Server: AmazonS3
GET /?max-keys=1000&prefix=31339%2F&delimiter=%2F HTTP/1.1
Date: Thu, 09 Apr 2015 19:05:32 GMT
Authorization: AWS 0ERCJ8PGP9FCHANHFD82:IARiG3GwAB+OHzGnxVvH77tRLHo=
Host: mubi-encoder-source.s3.amazonaws.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.7 (Mac OS X/10.10.1) (x86_64)
HTTP/1.1 200 OK
x-amz-id-2: 0mSXU4Cp72ef6kAkEmMljbvLK4O3GDVeZ4dHXftoC/Lv9nGVXRpqQW2Qz+eQZIKnTfqqIZdm8Kk=
x-amz-request-id: 722EA6BAD3991EA2
Date: Thu, 09 Apr 2015 19:06:09 GMT
Content-Type: application/xml
Transfer-Encoding: chunked
Server: AmazonS3

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Looks like the Connection: Keep-Alive header is still sent. Can you try again to disable with

defaults write ~/Library/Preferences/ch.sudo.cyberduck.plist http.connections.reuse false

and also update to the latest snaphot build.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Duplicate for #8634.

@iterate-ch iterate-ch locked as resolved and limited conversation to collaborators Nov 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@dkocher dkocher

Labels
bug duplicate s3 AWS S3 Protocol Implementation
Projects
None yet
Milestone
4.7
Development

No branches or pull requests
2 participants
@dkocher @cyberduck