Navigation Menu

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Large uploads requiring checksum of parts fail with network timeout #10516

Closed
cyberduck opened this issue Oct 23, 2018 · 13 comments
Closed

Large uploads requiring checksum of parts fail with network timeout #10516

cyberduck opened this issue Oct 23, 2018 · 13 comments
Assignees
@dkocher
Labels
bug fixed high priority s3 AWS S3 Protocol Implementation
Milestone
6.8.3

Comments

@cyberduck
Copy link
Collaborator

e8c8a40 created the issue

Hello We have had this problem, since the release of 6.8.0 (it was fine on 6.7.x) and 6.8.1 or 6.8.2 updates did not fix it.

When we try to upload a large file to S3 Share, the Checksum gets computer at the begining - that takes a really long time. (but that is how its always been) now when the actuall transfer begins, all the CPU's on the computer are pegged out at 100% and then the upload eventually fails.

The interesting thing is that it happens on Macs and PC's , i tried to test this on multiple computers. (same or similar issue is happening)

If I can help in any way to try to send some logs, i could do that.

Thank you very much for looking at this issue.

-Marek

Attachments
@cyberduck
Copy link
Collaborator Author

@dkocher commented

Can you please attach the error message displayed.

@cyberduck
Copy link
Collaborator Author

e8c8a40 commented

I forgot to add that the transfer speeds are slowing down from about 45 MB/sec down to 5-6.5 MB/sec and then it fails. This is very easy reproducable, the files 2GB or less will transfer okay , (but still with visible slow down toward the end of the transfer ) ..... anything larger than 2GB, its pretty upsetting how it slows down and eventually slow down.

Thank you for looking into this,

Marek

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Screen Shot 2018-10-30 at 3.18.41 PM.png

@cyberduck
Copy link
Collaborator Author

@dkocher commented

For the issue that MD5 checksum calculation is taking a long time and could be skipped for multipart uploads refer to #10278.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Regression from 1ab744c which decreases performance significantly when skipping from the file input stream which is used extensively when calculating checksums for parts of a multipart upload.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

In 4f64fbe.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

#10491 closed as duplicate.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

#10492 closed as duplicate.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

#10465 closed as duplicate.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

#10485 closed as duplicate.

@cyberduck
Copy link
Collaborator Author

e8c8a40 commented

David,

Again, thanks for taking a look at this. I noticed that you closed this ticket, as it did resurface previously (large data fail upload to S3) ... Is there something you would like me to do? Test it once you guys release 6.8.3? or test it with a nightly update? Not rushing or anything. Just offering to do more testing.

Thank you,
Marek

@cyberduck
Copy link
Collaborator Author

@dkocher commented

#10429 closed as duplicate.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Replying to [comment:15 Chicago_Marek]:

David,

Again, thanks for taking a look at this. I noticed that you closed this ticket, as it did resurface previously (large data fail upload to S3) ... Is there something you would like me to do? Test it once you guys release 6.8.3? or test it with a nightly update? Not rushing or anything. Just offering to do more testing.

Thank you,
Marek
Please update to the latest beta build available. Version 6.8.3 will be released tomorrow.

@iterate-ch iterate-ch locked as resolved and limited conversation to collaborators Nov 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@dkocher dkocher

Labels
bug fixed high priority s3 AWS S3 Protocol Implementation
Projects
None yet
Milestone
6.8.3
Development

No branches or pull requests
2 participants
@dkocher @cyberduck