Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for Multipart Uploads #5487

Closed
cyberduck opened this issue Dec 1, 2010 · 11 comments
Closed

Support for Multipart Uploads #5487

cyberduck opened this issue Dec 1, 2010 · 11 comments
Assignees
Labels
enhancement fixed high priority s3 AWS S3 Protocol Implementation
Milestone

Comments

@cyberduck
Copy link
Collaborator

6883dc5 created the issue

S3 recently added a new feature: multi-part uploads.

The benefits of using this feature with Cyberduck include:

  • Resumable uploads
  • Better performance (potentially) if multiple parts are uploaded in parallel

This added functionality comes almost for free in the AWS SDK for Java mentioned in ticket #5486, so it is probably useful to consider this as blocked on that ticket.

I am also willing to contribute code for this feature. In all honesty it should be nearly trivial once #5486 is resolved.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

This is also discussed in this thread on the JetS3t mailing list.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

In addition this would allow to support the new 5 TB Object Size Limit.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

#5592 closed as duplicate.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

In d6f5bd8. Resuming is not yet supported. Currently the default parallelism is 10 requests. Files larger than 16MB are uploaded using multiparts.

@cyberduck
Copy link
Collaborator Author

454a57d commented

I have tried several times but am unable to upload a 16GB file to Amazon S3 Singapore. Any ideas? The upload seems to run to the end, but then I get an Upload Failed error.

Thanks.

-Mark

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Replying to [comment:6 stickenhoffen]:

I have tried several times but am unable to upload a 16GB file to Amazon S3 Singapore. Any ideas? The upload seems to run to the end, but then I get an Upload Failed error.

Thanks.

-Mark

Are you running the latest snapshot build? Please post the HTTP transcript from the log drawer in the Transfers window.

@cyberduck
Copy link
Collaborator Author

454a57d commented

Hi, yes I am using the very latest snapshot. Can I email the log to you rather than have it archived here?

Thanks so much.

-Mark

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Replying to [comment:8 stickenhoffen]:

Hi, yes I am using the very latest snapshot. Can I email the log to you rather than have it archived here?

Thanks so much.

-Mark

Sure, [mailto:feedback@cyberduck.ch].

@cyberduck
Copy link
Collaborator Author

@dkocher commented

S3 is returning a 404 Not Found for some PUTrequests for parts to be uploaded. According to the documentation, this means The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed. which is not the case here as later requests succeed with the same upload ID.

Have to do some more testing with this. Analyzing is a pain because the log is mangled with concurrent requests, though.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Replying to [comment:10 dkocher]:

S3 is returning a 404 Not Found for some PUTrequests for parts to be uploaded. According to the documentation, this means The specified multipart upload does not exist. The upload ID might be invalid, or the multipart upload might have been aborted or completed. which is not the case here as later requests succeed with the same upload ID.

Have to do some more testing with this. Analyzing is a pain because the log is mangled with concurrent requests, though.

There is possibly nothing we can do about these failures but to support resuming parts of the transfer that failed.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

We have prepared but not yet enabled support for resuming parts of a multipart upload in 346e367. The problem is that failed uploads take up space and must be explicitly cancelled if the transfer should be aborted altogether. Currently we do this for every incomplete upload. If resuming should be possible we obviously can't do this but need an option for the user to cancel later instead of resuming.

@iterate-ch iterate-ch locked as resolved and limited conversation to collaborators Nov 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement fixed high priority s3 AWS S3 Protocol Implementation
Projects
None yet
Development

No branches or pull requests

2 participants