Cyberduck Mountain Duck CLI

#5487 closed enhancement (fixed)

Support for Multipart Uploads

Reported by: https://www.google.com/accounts/o8/id?id=aitoawmezscd8syasm3f-juyissmym1nzqfrznm Owned by: dkocher
Priority: high Milestone: 4.0
Component: s3 Version: 3.7
Severity: normal Keywords:
Cc: Architecture:
Platform:

Description (last modified by dkocher)

S3 recently added a new feature: multi-part uploads.

The benefits of using this feature with Cyberduck include:

  • Resumable uploads
  • Better performance (potentially) if multiple parts are uploaded in parallel

This added functionality comes almost for free in the AWS SDK for Java mentioned in ticket #5486, so it is probably useful to consider this as blocked on that ticket.

I am also willing to contribute code for this feature. In all honesty it should be nearly trivial once #5486 is resolved.

Change History (12)

comment:1 Changed on Dec 1, 2010 at 11:12:00 AM by dkocher

  • Milestone set to 4.1
  • Status changed from new to assigned

This is also discussed in this thread on the JetS3t mailing list.

comment:2 Changed on Dec 10, 2010 at 4:38:18 PM by dkocher

  • Description modified (diff)

comment:3 Changed on Dec 10, 2010 at 4:47:27 PM by dkocher

In addition this would allow to support the new 5 TB Object Size Limit.

comment:4 Changed on Dec 28, 2010 at 8:51:19 AM by dkocher

#5592 closed as duplicate.

comment:5 Changed on Jan 5, 2011 at 7:08:12 PM by dkocher

  • Milestone changed from 4.1 to 4.0
  • Resolution set to fixed
  • Status changed from assigned to closed

In r8197. Resuming is not yet supported. Currently the default parallelism is 10 requests. Files larger than 16MB are uploaded using multiparts.

Last edited on Jan 5, 2011 at 7:08:26 PM by dkocher (previous) (diff)

comment:6 follow-up: Changed on Jan 22, 2011 at 8:06:25 AM by stickenhoffen

I have tried several times but am unable to upload a 16GB file to Amazon S3 Singapore. Any ideas? The upload seems to run to the end, but then I get an Upload Failed error.

Thanks.

-Mark

comment:7 in reply to: ↑ 6 Changed on Jan 22, 2011 at 10:09:51 AM by dkocher

Replying to stickenhoffen:

I have tried several times but am unable to upload a 16GB file to Amazon S3 Singapore. Any ideas? The upload seems to run to the end, but then I get an Upload Failed error.

Thanks.

-Mark

Are you running the latest snapshot build? Please post the HTTP transcript from the log drawer in the Transfers window.

Last edited on Jan 22, 2011 at 10:10:34 AM by dkocher (previous) (diff)

comment:8 follow-up: Changed on Jan 22, 2011 at 11:52:23 AM by stickenhoffen

Hi, yes I am using the very latest snapshot. Can I email the log to you rather than have it archived here?

Thanks so much.

-Mark

comment:9 in reply to: ↑ 8 Changed on Jan 22, 2011 at 11:58:30 AM by dkocher

Replying to stickenhoffen:

Hi, yes I am using the very latest snapshot. Can I email the log to you rather than have it archived here?

Thanks so much.

-Mark

Sure, feedback@….

comment:10 follow-up: Changed on Jan 22, 2011 at 12:34:25 PM by dkocher

S3 is returning a 404 Not Found for some PUTrequests for parts to be uploaded. According to the documentation, this means The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed. which is not the case here as later requests succeed with the same upload ID.

Have to do some more testing with this. Analyzing is a pain because the log is mangled with concurrent requests, though.

comment:11 in reply to: ↑ 10 Changed on Jan 26, 2011 at 3:19:51 PM by dkocher

Replying to dkocher:

S3 is returning a 404 Not Found for some PUTrequests for parts to be uploaded. According to the documentation, this means The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed. which is not the case here as later requests succeed with the same upload ID.

Have to do some more testing with this. Analyzing is a pain because the log is mangled with concurrent requests, though.

There is possibly nothing we can do about these failures but to support resuming parts of the transfer that failed.

comment:12 Changed on Jan 26, 2011 at 5:19:14 PM by dkocher

We have prepared but not yet enabled support for resuming parts of a multipart upload in r8289. The problem is that failed uploads take up space and must be explicitly cancelled if the transfer should be aborted altogether. Currently we do this for every incomplete upload. If resuming should be possible we obviously can't do this but need an option for the user to cancel later instead of resuming.

Note: See TracTickets for help on using tickets.
swiss made software