Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Configurable part size #6367

Closed
cyberduck opened this issue Nov 14, 2011 · 12 comments
Closed

Configurable part size #6367

cyberduck opened this issue Nov 14, 2011 · 12 comments
Assignees
Labels
enhancement s3 AWS S3 Protocol Implementation worksforme

Comments

@cyberduck
Copy link
Collaborator

38a28fd created the issue

Hi!

Amazon S3 multipart upload default part size is 5MB. I dont know with this app is how much.

But if part size is small, upload price is higher, because PUT, COPY, POST, or LIST requests is much higher.

My quest: selectable part size of multipart upload in S3 options. Interval example: 5-100MB

Thanks.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

We split uploads into multiparts with a threshold of 5MB and make parts of 5MB size each.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Do you want to change the threshold or the individual part sizes?

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Documentation to disable in the wiki.

@cyberduck
Copy link
Collaborator Author

38a28fd commented

Replying to [comment:1 dkocher]:

We split uploads into multiparts with a threshold of 5MB and make parts of 5MB size each.

Thanks for this information!

@cyberduck
Copy link
Collaborator Author

38a28fd commented

Replying to [comment:2 dkocher]:

Do you want to change the threshold or the individual part sizes?

Hi!

My quest change to "individual" part size.

But I think treshold is changed automatic too,
example part size is 100MB, and upload file is 10MB, no sense, if treshold is 5/10/20 etc MB, because file size is smaller than part size.

This is mean, treshold and part size is equal automatic.

Thanks!

@cyberduck
Copy link
Collaborator Author

@dkocher commented

The settings are configurable with a hidden option as of 8aa5cf0.

s3.upload.multipart.threshold
s3.upload.multipart.size

@cyberduck
Copy link
Collaborator Author

@dkocher commented

We will not introduce a graphical preference for this.

@cyberduck
Copy link
Collaborator Author

38a28fd commented

Replying to [comment:7 dkocher]:

We will not introduce a graphical preference for this.

Thank you.

Hidden options is good for me, but in other app, example s3browser, part size selectable option is graphical (GUI).

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Replying to [comment:8 mikk]:

Replying to [comment:7 dkocher]:

We will not introduce a graphical preference for this.

Thank you.

Hidden options is good for me, but in other app, example s3browser, part size selectable option is graphical (GUI).

It's an option that should be optimized for the best performance but not something the user should need to care about.

@cyberduck
Copy link
Collaborator Author

38a28fd commented

Hi!

Replying to [comment:9 dkocher]:

It's an option that should be optimized for the best performance but not something the user should need to care about.

No, see ticket message, I think this is not performance options, this is "lower price" :) options.
But GUI of this options not needed for me, I wrote this.

Very thank you. I test this function if I can compile source or new version is arrived.

@cyberduck
Copy link
Collaborator Author

38a28fd commented

Hi!

This function works badly, I see 9241 version.

Example, I set threshold and size to 100. Start upload, file size about 1GB. Well, upload is extremly fast, about 210MB/s (!), this is impossible.
After I clik to stop button, I see "disconnecting from Amazon". But nothings happen. Remove button don't works. After I close this app, I get question: "There are files currently transferred, Quit anyway?"

Thanks.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

We will leave that with a hidden configuration option and optimize this in future builds. I don' think this is something we put in the graphical Preferences the user has to bother about.

@iterate-ch iterate-ch locked as resolved and limited conversation to collaborators Nov 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement s3 AWS S3 Protocol Implementation worksforme
Projects
None yet
Development

No branches or pull requests

2 participants