Cyberduck Mountain Duck CLI

#7410 closed defect (fixed)

Large S3 uploads fail

Reported by: jspafford Owned by: dkocher
Priority: highest Milestone: 4.4
Component: s3 Version: 4.3.1
Severity: blocker Keywords:
Cc: Architecture: Intel
Platform: Mac OS X 10.8

Description

I've been getting this error for years. I use Cloudberry running a VM but hate doing it. Thought I'd try again after a year of not using CD but it still fails.

2GB files... right now trying a 15GB one on a very fast internet connection. Getting 5-6MB per second up.

Same error I've been getting for years: I/O Error: upload failed. Skipped X bytes instead of X.

And you can't resume an upload. (as you should be able to with multipart uploads)

AAnnyyywwayyy. Screenshot attached.

Attachments (2)

Screen Shot 2013-08-21 at 10.48.28 PM.png (65.7 KB) - added by jspafford on Aug 22, 2013 at 6:03:10 AM.
SS
Screen Shot 2013-08-27 at 6.35.02 PM.png (102.9 KB) - added by jspafford on Aug 28, 2013 at 1:39:30 AM.

Download all attachments as: .zip

Change History (12)

comment:1 Changed on Aug 22, 2013 at 9:05:46 AM by dkocher

  • Milestone set to 4.4
  • Resolution set to duplicate
  • Status changed from new to closed

Please update to the latest snapshot build available. Duplicate for #6208. Reopen this ticket if you are still having issues.

comment:2 Changed on Aug 23, 2013 at 3:07:00 AM by jspafford

  • Resolution duplicate deleted
  • Status changed from closed to reopened

I've updated to the snapshot build and did a test with a 15.7GB zip file. Took about an hour going to S3 East coast from LA. (I love how the east coast server is just as or sometimes faster than the west coast ones. Bigger tube I guess) I'm going to keep using the same file as a test for this ticket so everything remains same. (same file, internet connection, time of day... conditions)

The upload completed successfully according to the Transfers window. I refreshed the bucket and the file size read 0. Closed restarted... same thing. Checked the bucket with another computer through a different IP... same thing. When downloading, it is just a 0 bite zip file.

When I try to resume, I get transfer incomplete again and again with 419.4MB of 15.7GB.

I would assume the pieces in the upload got lost and there's no map to say what goes where. I've successfully uploaded this file a few times (one zipped and one a QT MOV file) via CloudBerry. So I know it's possible.

comment:3 Changed on Aug 23, 2013 at 3:08:19 AM by jspafford

Oh and I've tried this twice. Same result.

comment:4 Changed on Aug 23, 2013 at 6:40:16 AM by dkocher

  • Status changed from reopened to new

comment:5 Changed on Aug 23, 2013 at 6:40:29 AM by dkocher

  • Status changed from new to assigned

comment:6 Changed on Aug 26, 2013 at 7:19:05 AM by dkocher

Added tests in r12582.

comment:7 Changed on Aug 28, 2013 at 1:40:24 AM by jspafford

Version 4.4 (12588) - Same result. Screenshot attached.

comment:8 follow-up: Changed on Aug 28, 2013 at 8:25:07 AM by probus

I have similar (if not the same) problems. The uploads complete successfully but result in 0KB files. This doesn't happen every time and sometimes happens with smaller (~15MB) files as well. I don't get any error messages and I've also had this problem before, I just rarely use S3.

This week I have uploaded 28 ~2GB files and over a hundred 15MB files. Every 15MB file was uploaded succesfully and maybe 6 of the bigger files resulted in 0KB files.

I have tried to upload and overwrite a 0Kb file but it doesn't help. If I delete the 0KB file first and then try to upload again it might work, usually doesn't. If I rename the file or delete and wait a while before uploading again I seem to get better odds that the file is actually written to S3.

I'm running Versio 4.4 (12520) on OSX 10.8.4

comment:9 in reply to: ↑ 8 Changed on Aug 28, 2013 at 12:59:05 PM by dkocher

Replying to probus:

I have similar (if not the same) problems. The uploads complete successfully but result in 0KB files. This doesn't happen every time and sometimes happens with smaller (~15MB) files as well. I don't get any error messages and I've also had this problem before, I just rarely use S3.

This week I have uploaded 28 ~2GB files and over a hundred 15MB files. Every 15MB file was uploaded succesfully and maybe 6 of the bigger files resulted in 0KB files.

I have tried to upload and overwrite a 0Kb file but it doesn't help. If I delete the 0KB file first and then try to upload again it might work, usually doesn't. If I rename the file or delete and wait a while before uploading again I seem to get better odds that the file is actually written to S3.

I'm running Versio 4.4 (12520) on OSX 10.8.4

Thanks for your valuable report. We have made further fixes in r12593 and r12594 to make multipart uploads more reliable. Please try the latest build (r12603 or later). We will add more tests before closing this issue.

comment:10 Changed on Sep 2, 2013 at 7:41:36 PM by dkocher

  • Resolution set to fixed
  • Status changed from assigned to closed

Removed placeholder object in r12645 which could possibly lead to these arbitrary results due to the eventual consistency of S3.

Note: See TracTickets for help on using tickets.
swiss made software