Cyberduck Mountain Duck CLI

Opened 8 years ago

Closed 8 years ago

Last modified 8 years ago

#5487 closed enhancement (fixed)

Support for Multipart Uploads

Reported by: https://www.google.com/accounts/o8/id?id=aitoawmezscd8syasm3f-juyissmym1nzqfrznm Owned by: dkocher
Priority: high Milestone: 4.0
Component: s3 Version: 3.7
Severity: normal Keywords:
Cc: Architecture:
Platform:

Description (last modified by dkocher)

S3 recently added a new feature: multi-part uploads.

The benefits of using this feature with Cyberduck include:

  • Resumable uploads
  • Better performance (potentially) if multiple parts are uploaded in parallel

This added functionality comes almost for free in the AWS SDK for Java mentioned in ticket #5486, so it is probably useful to consider this as blocked on that ticket.

I am also willing to contribute code for this feature. In all honesty it should be nearly trivial once #5486 is resolved.

Change History (12)

comment:1 Changed 8 years ago by dkocher

  • Milestone set to 4.1
  • Status changed from new to assigned

This is also discussed in this thread on the JetS3t mailing list.

comment:2 Changed 8 years ago by dkocher

  • Description modified (diff)

comment:3 Changed 8 years ago by dkocher

In addition this would allow to support the new 5 TB Object Size Limit.

comment:4 Changed 8 years ago by dkocher

#5592 closed as duplicate.

comment:5 Changed 8 years ago by dkocher

  • Milestone changed from 4.1 to 4.0
  • Resolution set to fixed
  • Status changed from assigned to closed

In r8197. Resuming is not yet supported. Currently the default parallelism is 10 requests. Files larger than 16MB are uploaded using multiparts.

Last edited 8 years ago by dkocher (previous) (diff)

comment:6 follow-up: Changed 8 years ago by stickenhoffen

I have tried several times but am unable to upload a 16GB file to Amazon S3 Singapore. Any ideas? The upload seems to run to the end, but then I get an Upload Failed error.

Thanks.

-Mark

comment:7 in reply to: ↑ 6 Changed 8 years ago by dkocher

Replying to stickenhoffen:

I have tried several times but am unable to upload a 16GB file to Amazon S3 Singapore. Any ideas? The upload seems to run to the end, but then I get an Upload Failed error.

Thanks.

-Mark

Are you running the latest snapshot build? Please post the HTTP transcript from the log drawer in the Transfers window.

Last edited 8 years ago by dkocher (previous) (diff)

comment:8 follow-up: Changed 8 years ago by stickenhoffen

Hi, yes I am using the very latest snapshot. Can I email the log to you rather than have it archived here?

Thanks so much.

-Mark

comment:9 in reply to: ↑ 8 Changed 8 years ago by dkocher

Replying to stickenhoffen:

Hi, yes I am using the very latest snapshot. Can I email the log to you rather than have it archived here?

Thanks so much.

-Mark

Sure, feedback@….

comment:10 follow-up: Changed 8 years ago by dkocher

S3 is returning a 404 Not Found for some PUTrequests for parts to be uploaded. According to the documentation, this means The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed. which is not the case here as later requests succeed with the same upload ID.

Have to do some more testing with this. Analyzing is a pain because the log is mangled with concurrent requests, though.

comment:11 in reply to: ↑ 10 Changed 8 years ago by dkocher

Replying to dkocher:

S3 is returning a 404 Not Found for some PUTrequests for parts to be uploaded. According to the documentation, this means The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed. which is not the case here as later requests succeed with the same upload ID.

Have to do some more testing with this. Analyzing is a pain because the log is mangled with concurrent requests, though.

There is possibly nothing we can do about these failures but to support resuming parts of the transfer that failed.

comment:12 Changed 8 years ago by dkocher

We have prepared but not yet enabled support for resuming parts of a multipart upload in r8289. The problem is that failed uploads take up space and must be explicitly cancelled if the transfer should be aborted altogether. Currently we do this for every incomplete upload. If resuming should be possible we obviously can't do this but need an option for the user to cancel later instead of resuming.

Note: See TracTickets for help on using tickets.
swiss made software