Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Folder only showing 1000 files #10811

Closed
cyberduck opened this issue Sep 18, 2019 · 9 comments
Closed

Folder only showing 1000 files #10811

cyberduck opened this issue Sep 18, 2019 · 9 comments
Assignees
@dkocher
Labels
bug fixed high priority s3 AWS S3 Protocol Implementation
Milestone
7.1.2

Comments

@cyberduck
Copy link
Collaborator

8719699 created the issue

S3 folder only showing 1000 files. I have folders where there are 20,000 files or more and I now can't access them anymore. To my knowledge, they are still there, just not showing in the Cyberduck Browser. I think the new update broke this. I'm on 7.1.0.31395
@cyberduck
Copy link
Collaborator Author

@dkocher commented

I cannot reproduce this issue with a bucket in eu-north-1 with 1037 files uploaded. We should issue multiple requests to fetch keys using marker

GET /?encoding-type=url&max-keys=1000&prefix=TRAC-10811%2F&delimiter=%2F HTTP/1.1
Date: Thu, 19 Sep 2019 07:45:21 GMT
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Host: test-eu-north-1-cyberduck.s3.amazonaws.com
x-amz-date: 20190919T074521Z
Authorization: ********
Connection: Keep-Alive
User-Agent: Cyberduck/7.1.1.31445 (Mac OS X/10.14.6) (x86_64)
HTTP/1.1 200 OK
x-amz-id-2: d8DbY3SvmfDop0DY9mtDQ9jwdt02XwaRPt2fblJlxyFBkNF0MMHzset8PBo4iPF0tLtzoCTMJvI=
x-amz-request-id: DF2F772BF05CD1D1
Date: Thu, 19 Sep 2019 07:45:22 GMT
x-amz-bucket-region: eu-north-1
Content-Type: application/xml
Transfer-Encoding: chunked
Server: AmazonS3
GET /?encoding-type=url&max-keys=1000&prefix=TRAC-10811%2F&delimiter=%2F&marker=TRAC-10811%2F1568878850200075000 HTTP/1.1
Date: Thu, 19 Sep 2019 07:45:21 GMT
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Host: test-eu-north-1-cyberduck.s3.amazonaws.com
x-amz-date: 20190919T074521Z
Authorization: ********
Connection: Keep-Alive
User-Agent: Cyberduck/7.1.1.31445 (Mac OS X/10.14.6) (x86_64)
HTTP/1.1 200 OK
x-amz-id-2: Who1ZcC3bPW2xN/dKHDUPOsxXXHgHpEZFI24w/NYAbDyiBjM3oJMmaP4fbD0C55myrcuJPenj88=
x-amz-request-id: 593DEA8EA4511C7F
Date: Thu, 19 Sep 2019 07:45:22 GMT
x-amz-bucket-region: eu-north-1
Content-Type: application/xml
Transfer-Encoding: chunked
Server: AmazonS3

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Please post the transcript from the log drawer (⌘-L).

@cyberduck
Copy link
Collaborator Author

8719699 commented

This is what I got from Ctrl-L and refreshing the browser window. Looks like the max keys are set to 1000? how do I change that. Never had an issue before.

GET /?encoding-type=url&max-keys=1000&prefix=Repertoire%2FRepertoire%20Current%2F&delimiter=%2F HTTP/1.1
Date: Thu, 19 Sep 2019 20:06:22 GMT
x-amz-request-payer: requester
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Host: dwmedia1.s3.amazonaws.com
x-amz-date: 20190919T200622Z
Authorization: ********
Connection: Keep-Alive
User-Agent: Cyberduck/7.1.0.31395 (Windows 8.1/6.3) (amd64)
HTTP/1.1 200 OK
x-amz-id-2: TJd6++ButicPmwUNPzule/GYppZUpNB9ywhNkcMbN3CDcPGTvdOH0jcuFep7RK7l4ZtZTMpxPAw=
x-amz-request-id: A7DB4C0E58E7E9AA
Date: Thu, 19 Sep 2019 20:06:31 GMT
x-amz-bucket-region: us-east-1
Content-Type: application/xml
Transfer-Encoding: chunked
Server: AmazonS3
GET /?encoding-type=url&max-keys=1000&prefix=Repertoire%2FRepertoire%20Current%2F&delimiter=%2F&marker=Repertoire%2FRepertoire%2BCurrent%2FHelp%2B-%2BThe%2BBeatles.pdf HTTP/1.1
Date: Thu, 19 Sep 2019 20:06:23 GMT
x-amz-request-payer: requester
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Host: dwmedia1.s3.amazonaws.com
x-amz-date: 20190919T200623Z
Authorization: ********
Connection: Keep-Alive
User-Agent: Cyberduck/7.1.0.31395 (Windows 8.1/6.3) (amd64)
HTTP/1.1 200 OK
x-amz-id-2: oIZyYuSF/4Z2BRguDqBTg4SgBijjo5Bt9UGCyOalm/UdTWaaY0umHJn2cjH6FzoHjnby3OrixqY=
x-amz-request-id: F9EC3F57D115AC92
Date: Thu, 19 Sep 2019 20:06:32 GMT
x-amz-bucket-region: us-east-1
Content-Type: application/xml
Transfer-Encoding: chunked
Server: AmazonS3
GET /?prefix=Repertoire%2FRepertoire%20Current%2F&uploads HTTP/1.1
Date: Thu, 19 Sep 2019 20:06:23 GMT
x-amz-request-payer: requester
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Host: dwmedia1.s3.amazonaws.com
x-amz-date: 20190919T200623Z
Authorization: ********
Connection: Keep-Alive
User-Agent: Cyberduck/7.1.0.31395 (Windows 8.1/6.3) (amd64)
HTTP/1.1 200 OK
x-amz-id-2: +12VIpOsviETMoY1EzFA6NMmbDwKf7JElVbe/RzOhDiWwMcNj/WWj8y154oWBel+7fzLD7P0U2A=
x-amz-request-id: 58F38262BC38AE26
Date: Thu, 19 Sep 2019 20:06:33 GMT
Content-Type: application/xml
Transfer-Encoding: chunked
Server: AmazonS3

@cyberduck
Copy link
Collaborator Author

@dkocher commented

I can confirm the max-keys is set to 1000 by default. Thus for directories containing more files we issue multiple GET request with a marker parameter.

GET /?encoding-type=url&max-keys=1000&prefix=Repertoire%2FRepertoire%20Current%2F&delimiter=%2F&marker=Repertoire%2FRepertoire%2BCurrent%2FHelp%2B-%2BThe%2BBeatles.pdf HTTP/1.1
Date: Thu, 19 Sep 2019 20:06:23 GMT

Given the two requests in the above transcript you should see more than 1000 files but less than 2000.

@cyberduck
Copy link
Collaborator Author

8719699 commented

Currently it brings up exactly 1000. I know barely anything about coding/your system (I took one coding class in python in college and can do very basic HTML/CSS). How can I make it pull all of my files up again?

@cyberduck
Copy link
Collaborator Author

8719699 commented

Hey, just wanted to check in. Any updates on this? Need anything more from me to figure it out? More than happy to adjust settings over here if needed, but couldn't find anything in the different menus that helped with this. Thank you!

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Please update to the latest snapshot build available and let us know if the issue can still be reproduced.

@cyberduck
Copy link
Collaborator Author

8719699 commented

Still having the same issues on the snapshot build 7.1.2.31581. No difference in how it handles the huge folders/buckets (only showing exactly 1000 files)

@cyberduck
Copy link
Collaborator Author

@dkocher commented

In d61df59.

@iterate-ch iterate-ch locked as resolved and limited conversation to collaborators Nov 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@dkocher dkocher

Labels
bug fixed high priority s3 AWS S3 Protocol Implementation
Projects
None yet
Milestone
7.1.2
Development

No branches or pull requests
2 participants
@dkocher @cyberduck