Cyberduck Mountain Duck CLI

#10426 closed defect (fixed)

Listing directory is slow with versioned bucket when looking for delete markers

Reported by: riq363 Owned by: yla
Priority: normal Milestone: 6.9.0
Component: s3 Version: 6.7.0
Severity: critical Keywords: slow loading authentication
Cc: bruno.uy@… Architecture:
Platform: macOS 10.13

Description

Just updated to the latest version and it takes forever to login to my S3 bucket. When trying to upload files it is also very slow.

Before the update today everything worked super fast, as it should be.

Change History (14)

comment:1 Changed on Aug 8, 2018 at 1:18:32 PM by yla

Do you have a default path set to a bucket in your bookmark? If yes, is there versioning enabled for this bucket? This might be a consequence of our improved versioning support (#10357).

comment:2 Changed on Aug 8, 2018 at 1:30:06 PM by riq363

Hi yla,

Yes, I have a default path set and yes, my bucked has versioning enabled.

comment:3 Changed on Aug 8, 2018 at 1:57:25 PM by yla

Is there any chance to post a transcript (Cmd-L) from the authentication process?

comment:4 Changed on Aug 9, 2018 at 8:57:36 AM by riq363

yla,

Unfortunately I don't think I can share that, however, I did notice that in the authentication process it makes HTTP requests for all folders I have there. And I have a lot of folders:

Here's an example request: GET /?max-keys=1&versions&prefix=apps%2Fthirdparty%2Fumarket%2F&delimiter=%2F HTTP/1.1

And there are dozens of not hundreds of such requests in that log. I assume that's what started causing the delay. Thank you.

Version 0, edited on Aug 9, 2018 at 8:57:36 AM by riq363 (next)

comment:5 Changed on Aug 10, 2018 at 7:08:15 AM by dkocher

  • Component changed from core to s3
  • Owner set to dkocher
  • Summary changed from Cyberduck – very slow authenticating and files upload to S3 to Very slow authenticating

comment:6 Changed on Aug 10, 2018 at 9:16:48 AM by yla

Thanks, that's enough information. There reason for this behaviour is that in the latest version we try to properly decide if a folder is hidden or not for versioned buckets. For this decision we have to issue a list operation for each folder in order to see if there is at least one object without delete marker in the subtree.

comment:7 Changed on Aug 21, 2018 at 7:32:50 PM by dkocher

  • Owner changed from dkocher to yla
  • Summary changed from Very slow authenticating to Listing directory is slow with versioned bucket when looking for delete markers

comment:8 Changed on Sep 3, 2018 at 3:10:12 PM by bruno.uy

  • Cc bruno.uy@… added

comment:9 Changed on Nov 1, 2018 at 8:41:48 AM by dkocher

  • Milestone set to 7.0

comment:10 Changed on Dec 24, 2018 at 1:12:59 PM by riq363

Any updates here?

comment:11 Changed on Jan 10, 2019 at 12:51:44 PM by yla

  • Resolution set to fixed
  • Status changed from new to closed

In r45969.

comment:12 Changed on Jan 17, 2019 at 12:05:21 PM by dkocher

  • Milestone changed from 7.0 to 6.9.1

comment:13 Changed on Jan 20, 2019 at 7:54:57 PM by dkocher

  • Milestone changed from 6.9.1 to 6.9.2

Milestone renamed

comment:14 Changed on Jan 21, 2019 at 6:32:52 AM by dkocher

  • Milestone changed from 6.9.2 to 6.9.0
Note: See TracTickets for help on using tickets.
swiss made software