Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Large directory on S3 not opening #10087

Closed
cyberduck opened this issue Sep 22, 2017 · 1 comment
Closed

Large directory on S3 not opening #10087

cyberduck opened this issue Sep 22, 2017 · 1 comment
Labels
bug s3 AWS S3 Protocol Implementation wontfix

Comments

@cyberduck
Copy link
Collaborator

7aeb860 created the issue

I have an S3 directory with a lot of files. About 400.000
CD is opening the directory in chunks of 1000 (according to the log). Just before the end (might be the last 1000 items), it stops.
Then I have max processor usage and no network activity.

Looks like CD retried 10 minutes later or so. Same activity in the log, and again stops just before the end.

First time statusbar message was: Authenticating, this time it is: Login succesful (Well in Dutch...)

Opening this directory used to work (although it took a lot of time, like 5 minutes or so), but of course the number of items is growing. Did I reach a maximum directory items threshold?

Log ends with:

GET /?max-keys=1000&versions&prefix=tl_assets%2F400%2F&delimiter=%2F&version-id-marker=null&key-marker=tl_assets%2F400%2Fyellowcab_yellowcabb_y37090_moss_winter_2016_3d24_400.jpg HTTP/1.1
Date: Fri, 22 Sep 2017 07:16:45 GMT
x-amz-request-payer: requester
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Host: production-publicbucket-xx1az459c85z.s3.amazonaws.com
x-amz-date: 20170922T071645Z
Authorization: ********
Connection: Keep-Alive
User-Agent: Cyberduck/6.2.6.26405 (Mac OS X/10.12.5) (x86_64)
HTTP/1.1 200 OK
x-amz-id-2: Su1iVBolRrntKrVrKr+LVWeRS9v/gsOsxw2tysbh8vQ/GWKCl7BVHcqVhBtq85IZZn6tOraBQtc=
x-amz-request-id: 889DD509E82DFDD9
Date: Fri, 22 Sep 2017 07:16:46 GMT
Content-Type: application/xml
Transfer-Encoding: chunked
Server: AmazonS3
GET /?max-keys=1000&versions&prefix=tl_assets%2F400%2F&delimiter=%2F&version-id-marker=null&key-marker=tl_assets%2F400%2Fzecchinodoro_zecchinodg_a191968_silver_zomer_2016_3d06_400.jpg HTTP/1.1
Date: Fri, 22 Sep 2017 07:16:45 GMT
x-amz-request-payer: requester
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Host: production-publicbucket-xx1az459c85z.s3.amazonaws.com
x-amz-date: 20170922T071645Z
Authorization: ********
Connection: Keep-Alive
User-Agent: Cyberduck/6.2.6.26405 (Mac OS X/10.12.5) (x86_64)
HTTP/1.1 200 OK
x-amz-id-2: P+9in7fBELqsSkosVqCdI/fJeIOoYm6FkrJ9LRrORIQnQYwRZ1YE+akLiQtcZx2hJDCeHIt12fw=
x-amz-request-id: C7EAA78031C5A49D
Date: Fri, 22 Sep 2017 07:16:46 GMT
Content-Type: application/xml
Transfer-Encoding: chunked
Server: AmazonS3
GET /?max-keys=1000&versions&prefix=tl_assets%2F400%2F&delimiter=%2F&version-id-marker=null&key-marker=tl_assets%2F400%2Fzecchinodoro_zecchinodg_n121251_51445041304_zomer_2017_3d33_400.jpg HTTP/1.1
Date: Fri, 22 Sep 2017 07:16:46 GMT
x-amz-request-payer: requester
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Host: production-publicbucket-xx1az459c85z.s3.amazonaws.com
x-amz-date: 20170922T071646Z
Authorization: ********
Connection: Keep-Alive
User-Agent: Cyberduck/6.2.6.26405 (Mac OS X/10.12.5) (x86_64)
HTTP/1.1 200 OK
x-amz-id-2: hLtIZGUT4NCc2LebAKrvazrLiqHTDml17eN4Es5m1IPAn9qB1TfqAD7Pbkl4EZieJg2cUKNQcQ4=
x-amz-request-id: AED983E421CE06E0
Date: Fri, 22 Sep 2017 07:16:47 GMT
Content-Type: application/xml
Transfer-Encoding: chunked
Server: AmazonS3
Connection: close
@cyberduck
Copy link
Collaborator Author

@dkocher commented

You are possibly hitting some limit with the performance of our browser interface. We suggest that you stop the directory listing when you get alerted with Continue listing directory with more than {0} files. and then choose to search for a specific file with the Filter & Search option. Alternatively group your files in folders.

@iterate-ch iterate-ch locked as resolved and limited conversation to collaborators Nov 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
bug s3 AWS S3 Protocol Implementation wontfix
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@cyberduck