Cyberduck Mountain Duck CLI

#10811 closed defect (fixed)

Folder only showing 1000 files

Reported by: RobinRyme Owned by: dkocher
Priority: high Milestone: 7.1.2
Component: s3 Version: 7.1
Severity: normal Keywords:
Cc: Architecture:
Platform:

Description

S3 folder only showing 1000 files. I have folders where there are 20,000 files or more and I now can't access them anymore. To my knowledge, they are still there, just not showing in the Cyberduck Browser. I think the new update broke this. I'm on 7.1.0.31395

Change History (14)

comment:1 Changed on Sep 19, 2019 at 7:29:23 AM by dkocher

  • Component changed from core to s3
  • Owner set to dkocher

comment:2 Changed on Sep 19, 2019 at 7:29:38 AM by dkocher

  • Summary changed from S3 folder only showing 1000 files to Folder only showing 1000 files

comment:3 Changed on Sep 19, 2019 at 7:46:44 AM by dkocher

I cannot reproduce this issue with a bucket in eu-north-1 with 1037 files uploaded. We should issue multiple requests to fetch keys using marker

GET /?encoding-type=url&max-keys=1000&prefix=TRAC-10811%2F&delimiter=%2F HTTP/1.1
Date: Thu, 19 Sep 2019 07:45:21 GMT
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Host: test-eu-north-1-cyberduck.s3.amazonaws.com
x-amz-date: 20190919T074521Z
Authorization: ********
Connection: Keep-Alive
User-Agent: Cyberduck/7.1.1.31445 (Mac OS X/10.14.6) (x86_64)
HTTP/1.1 200 OK
x-amz-id-2: d8DbY3SvmfDop0DY9mtDQ9jwdt02XwaRPt2fblJlxyFBkNF0MMHzset8PBo4iPF0tLtzoCTMJvI=
x-amz-request-id: DF2F772BF05CD1D1
Date: Thu, 19 Sep 2019 07:45:22 GMT
x-amz-bucket-region: eu-north-1
Content-Type: application/xml
Transfer-Encoding: chunked
Server: AmazonS3
GET /?encoding-type=url&max-keys=1000&prefix=TRAC-10811%2F&delimiter=%2F&marker=TRAC-10811%2F1568878850200075000 HTTP/1.1
Date: Thu, 19 Sep 2019 07:45:21 GMT
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Host: test-eu-north-1-cyberduck.s3.amazonaws.com
x-amz-date: 20190919T074521Z
Authorization: ********
Connection: Keep-Alive
User-Agent: Cyberduck/7.1.1.31445 (Mac OS X/10.14.6) (x86_64)
HTTP/1.1 200 OK
x-amz-id-2: Who1ZcC3bPW2xN/dKHDUPOsxXXHgHpEZFI24w/NYAbDyiBjM3oJMmaP4fbD0C55myrcuJPenj88=
x-amz-request-id: 593DEA8EA4511C7F
Date: Thu, 19 Sep 2019 07:45:22 GMT
x-amz-bucket-region: eu-north-1
Content-Type: application/xml
Transfer-Encoding: chunked
Server: AmazonS3

comment:4 Changed on Sep 19, 2019 at 7:47:14 AM by dkocher

Please post the transcript from the log drawer (⌘-L).

comment:5 Changed on Sep 19, 2019 at 8:07:33 PM by RobinRyme

This is what I got from Ctrl-L and refreshing the browser window. Looks like the max keys are set to 1000? how do I change that. Never had an issue before.

GET /?encoding-type=url&max-keys=1000&prefix=Repertoire%2FRepertoire%20Current%2F&delimiter=%2F HTTP/1.1
Date: Thu, 19 Sep 2019 20:06:22 GMT
x-amz-request-payer: requester
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Host: dwmedia1.s3.amazonaws.com
x-amz-date: 20190919T200622Z
Authorization: ********
Connection: Keep-Alive
User-Agent: Cyberduck/7.1.0.31395 (Windows 8.1/6.3) (amd64)
HTTP/1.1 200 OK
x-amz-id-2: TJd6++ButicPmwUNPzule/GYppZUpNB9ywhNkcMbN3CDcPGTvdOH0jcuFep7RK7l4ZtZTMpxPAw=
x-amz-request-id: A7DB4C0E58E7E9AA
Date: Thu, 19 Sep 2019 20:06:31 GMT
x-amz-bucket-region: us-east-1
Content-Type: application/xml
Transfer-Encoding: chunked
Server: AmazonS3
GET /?encoding-type=url&max-keys=1000&prefix=Repertoire%2FRepertoire%20Current%2F&delimiter=%2F&marker=Repertoire%2FRepertoire%2BCurrent%2FHelp%2B-%2BThe%2BBeatles.pdf HTTP/1.1
Date: Thu, 19 Sep 2019 20:06:23 GMT
x-amz-request-payer: requester
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Host: dwmedia1.s3.amazonaws.com
x-amz-date: 20190919T200623Z
Authorization: ********
Connection: Keep-Alive
User-Agent: Cyberduck/7.1.0.31395 (Windows 8.1/6.3) (amd64)
HTTP/1.1 200 OK
x-amz-id-2: oIZyYuSF/4Z2BRguDqBTg4SgBijjo5Bt9UGCyOalm/UdTWaaY0umHJn2cjH6FzoHjnby3OrixqY=
x-amz-request-id: F9EC3F57D115AC92
Date: Thu, 19 Sep 2019 20:06:32 GMT
x-amz-bucket-region: us-east-1
Content-Type: application/xml
Transfer-Encoding: chunked
Server: AmazonS3
GET /?prefix=Repertoire%2FRepertoire%20Current%2F&uploads HTTP/1.1
Date: Thu, 19 Sep 2019 20:06:23 GMT
x-amz-request-payer: requester
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Host: dwmedia1.s3.amazonaws.com
x-amz-date: 20190919T200623Z
Authorization: ********
Connection: Keep-Alive
User-Agent: Cyberduck/7.1.0.31395 (Windows 8.1/6.3) (amd64)
HTTP/1.1 200 OK
x-amz-id-2: +12VIpOsviETMoY1EzFA6NMmbDwKf7JElVbe/RzOhDiWwMcNj/WWj8y154oWBel+7fzLD7P0U2A=
x-amz-request-id: 58F38262BC38AE26
Date: Thu, 19 Sep 2019 20:06:33 GMT
Content-Type: application/xml
Transfer-Encoding: chunked
Server: AmazonS3
Last edited on Sep 20, 2019 at 7:13:53 AM by dkocher (previous) (diff)

comment:6 Changed on Sep 20, 2019 at 7:15:36 AM by dkocher

I can confirm the max-keys is set to 1000 by default. Thus for directories containing more files we issue multiple GET request with a marker parameter.

GET /?encoding-type=url&max-keys=1000&prefix=Repertoire%2FRepertoire%20Current%2F&delimiter=%2F&marker=Repertoire%2FRepertoire%2BCurrent%2FHelp%2B-%2BThe%2BBeatles.pdf HTTP/1.1
Date: Thu, 19 Sep 2019 20:06:23 GMT

Given the two requests in the above transcript you should see more than 1000 files but less than 2000.

comment:7 Changed on Sep 20, 2019 at 3:49:29 PM by RobinRyme

Currently it brings up *exactly* 1000. I know barely anything about coding/your system (I took one coding class in python in college and can do very basic HTML/CSS). How can I make it pull all of my files up again?

comment:8 Changed on Oct 4, 2019 at 4:48:27 PM by RobinRyme

Hey, just wanted to check in. Any updates on this? Need anything more from me to figure it out? More than happy to adjust settings over here if needed, but couldn't find anything in the different menus that helped with this. Thank you!

comment:9 Changed on Oct 5, 2019 at 10:48:26 AM by dkocher

Please update to the latest snapshot build available and let us know if the issue can still be reproduced.

comment:10 Changed on Oct 10, 2019 at 6:09:11 PM by RobinRyme

Still having the same issues on the snapshot build 7.1.2.31581. No difference in how it handles the huge folders/buckets (only showing exactly 1000 files)

comment:11 Changed on Oct 14, 2019 at 8:46:36 AM by dkocher

  • Milestone set to 8.0
  • Status changed from new to assigned

comment:12 Changed on Oct 23, 2019 at 2:57:03 PM by dkocher

  • Milestone changed from 8.0 to 7.1.2

comment:13 Changed on Oct 23, 2019 at 3:22:13 PM by dkocher

  • Priority changed from normal to high

comment:14 Changed on Oct 23, 2019 at 7:29:02 PM by dkocher

  • Resolution set to fixed
  • Status changed from assigned to closed

In r47947.

Note: See TracTickets for help on using tickets.
swiss made software