Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Large S3 uploads fail #7410

Closed
cyberduck opened this issue Aug 22, 2013 · 8 comments
Closed

Large S3 uploads fail #7410

cyberduck opened this issue Aug 22, 2013 · 8 comments
Assignees
Labels
bug fixed s3 AWS S3 Protocol Implementation
Milestone

Comments

@cyberduck
Copy link
Collaborator

5ccb4e8 created the issue

I've been getting this error for years. I use Cloudberry running a VM but hate doing it. Thought I'd try again after a year of not using CD but it still fails.

2GB files... right now trying a 15GB one on a very fast internet connection. Getting 5-6MB per second up.

Same error I've been getting for years: I/O Error: upload failed. Skipped X bytes instead of X.

And you can't resume an upload. (as you should be able to with multipart uploads)

AAnnyyywwayyy. Screenshot attached.


Attachments

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Please update to the latest snapshot build available. Duplicate for #6208. Reopen this ticket if you are still having issues.

@cyberduck
Copy link
Collaborator Author

5ccb4e8 commented

I've updated to the snapshot build and did a test with a 15.7GB zip file. Took about an hour going to S3 East coast from LA. (I love how the east coast server is just as or sometimes faster than the west coast ones. Bigger tube I guess) I'm going to keep using the same file as a test for this ticket so everything remains same. (same file, internet connection, time of day... conditions)

The upload completed successfully according to the Transfers window. I refreshed the bucket and the file size read 0. Closed restarted... same thing. Checked the bucket with another computer through a different IP... same thing. When downloading, it is just a 0 bite zip file.

When I try to resume, I get transfer incomplete again and again with 419.4MB of 15.7GB.

I would assume the pieces in the upload got lost and there's no map to say what goes where. I've successfully uploaded this file a few times (one zipped and one a QT MOV file) via CloudBerry. So I know it's possible.

@cyberduck
Copy link
Collaborator Author

5ccb4e8 commented

Oh and I've tried this twice. Same result.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Added tests in 4063ebc.

@cyberduck
Copy link
Collaborator Author

5ccb4e8 commented

Version 4.4 (12588) - Same result. Screenshot attached.

@cyberduck
Copy link
Collaborator Author

308f661 commented

I have similar (if not the same) problems. The uploads complete successfully but result in 0KB files. This doesn't happen every time and sometimes happens with smaller (~15MB) files as well. I don't get any error messages and I've also had this problem before, I just rarely use S3.

This week I have uploaded 28 ~2GB files and over a hundred 15MB files. Every 15MB file was uploaded succesfully and maybe 6 of the bigger files resulted in 0KB files.

I have tried to upload and overwrite a 0Kb file but it doesn't help. If I delete the 0KB file first and then try to upload again it might work, usually doesn't. If I rename the file or delete and wait a while before uploading again I seem to get better odds that the file is actually written to S3.

I'm running Versio 4.4 (12520) on OSX 10.8.4

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Replying to [comment:8 probus]:

I have similar (if not the same) problems. The uploads complete successfully but result in 0KB files. This doesn't happen every time and sometimes happens with smaller (~15MB) files as well. I don't get any error messages and I've also had this problem before, I just rarely use S3.

This week I have uploaded 28 ~2GB files and over a hundred 15MB files. Every 15MB file was uploaded succesfully and maybe 6 of the bigger files resulted in 0KB files.

I have tried to upload and overwrite a 0Kb file but it doesn't help. If I delete the 0KB file first and then try to upload again it might work, usually doesn't. If I rename the file or delete and wait a while before uploading again I seem to get better odds that the file is actually written to S3.

I'm running Versio 4.4 (12520) on OSX 10.8.4

Thanks for your valuable report. We have made further fixes in 74ca4ff and 992b32f to make multipart uploads more reliable. Please try the latest build (0a48e4a or later). We will add more tests before closing this issue.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Removed placeholder object in bbf7e98 which could possibly lead to these arbitrary results due to the eventual consistency of S3.

@iterate-ch iterate-ch locked as resolved and limited conversation to collaborators Nov 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug fixed s3 AWS S3 Protocol Implementation
Projects
None yet
Development

No branches or pull requests

2 participants