Cyberduck Mountain Duck CLI

#6367 closed enhancement (worksforme)

Configurable part size

Reported by: mikk Owned by: dkocher
Priority: normal Milestone:
Component: s3 Version: 4.1.3
Severity: normal Keywords:
Cc: Architecture: Intel
Platform: Windows 7

Description

Hi!

Amazon S3 multipart upload default part size is 5MB. I dont know with this app is how much.

But if part size is small, upload price is higher, because PUT, COPY, POST, or LIST requests is much higher.

My quest: selectable part size of multipart upload in S3 options. Interval example: 5-100MB

Thanks.

Change History (16)

comment:1 follow-up: Changed on Nov 14, 2011 at 1:29:49 PM by dkocher

We split uploads into multiparts with a threshold of 5MB and make parts of 5MB size each.

comment:2 follow-up: Changed on Nov 14, 2011 at 1:30:19 PM by dkocher

Do you want to change the threshold or the individual part sizes?

comment:3 Changed on Nov 14, 2011 at 1:33:15 PM by dkocher

Documentation to disable in the wiki.

comment:4 in reply to: ↑ 1 Changed on Nov 14, 2011 at 1:34:48 PM by mikk

Replying to dkocher:

We split uploads into multiparts with a threshold of 5MB and make parts of 5MB size each.

Thanks for this information!

comment:5 in reply to: ↑ 2 Changed on Nov 14, 2011 at 1:50:51 PM by mikk

Replying to dkocher:

Do you want to change the threshold or the individual part sizes?

Hi!

My quest change to "individual" part size.

But I think treshold is changed automatic too, example part size is 100MB, and upload file is 10MB, no sense, if treshold is 5/10/20 etc MB, because file size is smaller than part size.

This is mean, treshold and part size is equal automatic.

Thanks!

comment:6 Changed on Nov 15, 2011 at 12:28:24 PM by dkocher

The settings are configurable with a hidden option as of r9230.

s3.upload.multipart.threshold
s3.upload.multipart.size

comment:7 follow-up: Changed on Nov 15, 2011 at 12:28:49 PM by dkocher

  • Milestone set to 4.1.4
  • Resolution set to fixed
  • Status changed from new to closed

We will not introduce a graphical preference for this.

comment:8 in reply to: ↑ 7 ; follow-up: Changed on Nov 15, 2011 at 12:42:20 PM by mikk

Replying to dkocher:

We will not introduce a graphical preference for this.

Thank you.

Hidden options is good for me, but in other app, example s3browser, part size selectable option is graphical (GUI).

Last edited on Nov 15, 2011 at 12:42:31 PM by mikk (previous) (diff)

comment:9 in reply to: ↑ 8 ; follow-up: Changed on Nov 15, 2011 at 12:46:41 PM by dkocher

Replying to mikk:

Replying to dkocher:

We will not introduce a graphical preference for this.

Thank you.

Hidden options is good for me, but in other app, example s3browser, part size selectable option is graphical (GUI).

It's an option that should be optimized for the best performance but not something the user should need to care about.

comment:10 in reply to: ↑ 9 Changed on Nov 15, 2011 at 12:57:21 PM by mikk

Hi!

Replying to dkocher:

It's an option that should be optimized for the best performance but not something the user should need to care about.

No, see ticket message, I think this is not performance options, this is "lower price" :) options. But GUI of this options not needed for me, I worte this.

Very thank you. I test this function if I can compile source or new version is arrived.

Version 0, edited on Nov 15, 2011 at 12:57:21 PM by mikk (next)

comment:11 Changed on Nov 16, 2011 at 5:28:16 AM by mikk

Hi!

This function works badly, I see 9241 version.

Example, I set threshold and size to 100. Start upload, file size about 1GB. Well, upload is extremly fast, about 210MB/s (!), this is impossible. After I clik to stop button, I see "disconnecting from Amazon". But nothings happen. Remove button don't works. After I close this app, I get question: "There are files currently transferred, Quit anyway?"

Thanks.

comment:12 Changed on Nov 16, 2011 at 5:28:45 AM by mikk

  • Resolution fixed deleted
  • Status changed from closed to reopened

comment:13 Changed on Nov 18, 2011 at 11:13:10 PM by dkocher

  • Summary changed from Amazon S3 multipart upload - selectable part size to Configurable part size

comment:14 Changed on Nov 24, 2011 at 9:21:43 PM by dkocher

  • Milestone changed from 4.2 to 4.3

comment:15 Changed on Mar 22, 2013 at 9:56:08 AM by dkocher

  • Resolution set to worksforme
  • Status changed from reopened to closed

We will leave that with a hidden configuration option and optimize this in future builds. I don' think this is something we put in the graphical Preferences the user has to bother about.

comment:16 Changed on Mar 22, 2013 at 10:14:32 AM by dkocher

  • Milestone 4.4 deleted
Note: See TracTickets for help on using tickets.
swiss made software