Cyberduck Mountain Duck CLI

Opened 8 weeks ago

Closed 6 weeks ago

Last modified 6 weeks ago

#10516 closed defect (fixed)

Large uploads requiring checksum of parts fail with network timeout

Reported by: Chicago_Marek Owned by: dkocher
Priority: high Milestone: 6.8.3
Component: s3 Version: 6.8.2
Severity: major Keywords: S3 Uploads Fail
Cc: Architecture:
Platform: macOS 10.13

Description

Hello We have had this problem, since the release of 6.8.0 (it was fine on 6.7.x) and 6.8.1 or 6.8.2 updates did not fix it.

When we try to upload a large file to S3 Share, the Checksum gets computer at the begining - that takes a really long time. (but that is how its always been) now when the actuall transfer begins, all the CPU's on the computer are pegged out at 100% and then the upload eventually fails.

The interesting thing is that it happens on Macs and PC's , i tried to test this on multiple computers. (same or similar issue is happening)

If I can help in any way to try to send some logs, i could do that.

Thank you very much for looking at this issue.

-Marek

Attachments (3)

Screen Shot 2018-10-30 at 3.18.41 PM.png (177.7 KB) - added by Chicago_Marek 6 weeks ago.
Error that i got after turning on "debugging" Normally files larger than 2GB fail to upload
Screen Shot 2018-10-30 at 3.19.46 PM.png (208.2 KB) - added by Chicago_Marek 6 weeks ago.
also , before the debugging they would fail and say "Transfer Incomplete". CPU always is pegged at 100% when these uploads fail. The CPU was not doing that prior to version 6.8.0
Cyberduck_2018-10-23-114719_MBABALA-MAC.cpu_resource.diag (6.7 KB) - added by Chicago_Marek 6 weeks ago.
CPU peg error . from Diagnostics Directory

Download all attachments as: .zip

Change History (20)

comment:1 Changed 7 weeks ago by dkocher

  • Milestone set to 6.8.3

comment:2 Changed 7 weeks ago by dkocher

Can you please attach the error message displayed.

comment:3 Changed 7 weeks ago by dkocher

  • Owner set to dkocher
  • Status changed from new to assigned

Changed 6 weeks ago by Chicago_Marek

Error that i got after turning on "debugging" Normally files larger than 2GB fail to upload

Changed 6 weeks ago by Chicago_Marek

also , before the debugging they would fail and say "Transfer Incomplete". CPU always is pegged at 100% when these uploads fail. The CPU was not doing that prior to version 6.8.0

Changed 6 weeks ago by Chicago_Marek

CPU peg error . from Diagnostics Directory

comment:4 Changed 6 weeks ago by Chicago_Marek

I forgot to add that the transfer speeds are slowing down from about 45 MB/sec down to 5-6.5 MB/sec and then it fails. This is very easy reproducable, the files 2GB or less will transfer okay , (but still with visible slow down toward the end of the transfer ) ..... anything larger than 2GB, its pretty upsetting how it slows down and eventually slow down.

Thank you for looking into this,

Marek

comment:5 Changed 6 weeks ago by dkocher

  • Summary changed from Uploads Fail to the Amazon AWS S3 Bucket to Multipart uploads fail with network timeout

comment:6 Changed 6 weeks ago by dkocher

Error that i got after turning on "debugging"  Normally files larger than 2GB fail to upload

comment:7 Changed 6 weeks ago by dkocher

For the issue that MD5 checksum calculation is taking a long time and could be skipped for multipart uploads refer to #10278.

comment:8 Changed 6 weeks ago by dkocher

  • Priority changed from normal to high

Regression from r45102 which decreases performance significantly when skipping from the file input stream which is used extensively when calculating checksums for parts of a multipart upload.

comment:9 Changed 6 weeks ago by dkocher

  • Summary changed from Multipart uploads fail with network timeout to Large uploads requiring checksum of parts fail with network timeout

comment:10 Changed 6 weeks ago by dkocher

  • Resolution set to fixed
  • Status changed from assigned to closed

In r45403.

comment:11 Changed 6 weeks ago by dkocher

#10491 closed as duplicate.

comment:12 Changed 6 weeks ago by dkocher

#10492 closed as duplicate.

comment:13 Changed 6 weeks ago by dkocher

#10465 closed as duplicate.

comment:14 Changed 6 weeks ago by dkocher

#10485 closed as duplicate.

comment:15 follow-up: Changed 6 weeks ago by Chicago_Marek

David,

Again, thanks for taking a look at this. I noticed that you closed this ticket, as it did resurface previously (large data fail upload to S3) ... Is there something you would like me to do? Test it once you guys release 6.8.3? or test it with a nightly update? Not rushing or anything. Just offering to do more testing.

Thank you, Marek

comment:16 Changed 6 weeks ago by dkocher

#10429 closed as duplicate.

comment:17 in reply to: ↑ 15 Changed 6 weeks ago by dkocher

Replying to Chicago_Marek:

David,

Again, thanks for taking a look at this. I noticed that you closed this ticket, as it did resurface previously (large data fail upload to S3) ... Is there something you would like me to do? Test it once you guys release 6.8.3? or test it with a nightly update? Not rushing or anything. Just offering to do more testing.

Thank you, Marek

Please update to the latest beta build available. Version 6.8.3 will be released tomorrow.

Note: See TracTickets for help on using tickets.
swiss made software