Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Problem opening S3 buckets containing many thousands of files #1977

Closed
cyberduck opened this issue Apr 22, 2008 · 1 comment
Closed

Problem opening S3 buckets containing many thousands of files #1977

cyberduck opened this issue Apr 22, 2008 · 1 comment
Assignees
Labels
bug s3 AWS S3 Protocol Implementation worksforme
Milestone

Comments

@cyberduck
Copy link
Collaborator

dd53f20 created the issue

I have several S3 buckets that each contain tens of thousands of data objects. These data objects are named using the full path and filename of the original file (they are backups). For example dir1/dir2/dir3/somefile.txt

When Cyberduck opens the bucket it begins to download the file list and becomes un-responsive during this time. The refresh cannot be stopped using the stop button and the app must be force-quit.

Is it possible to prevent this by first checking the number of files in a bucket and warning the user before attempting to fetch the list? Alternately, is it possible to download the file list in stages (ie 1000 at a time)?. The last possible solution I can think of is to form a request to get only the first (pseudo) subdir. Although from my basic understanding of the S3 API, this is not possible.

Thanks for making Cyberduck!

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Please try a nightly build from http://update.cyberduck.ch/nightly with a revision 2fefddd or later. Duplicate of issue #1968.

@iterate-ch iterate-ch locked as resolved and limited conversation to collaborators Nov 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug s3 AWS S3 Protocol Implementation worksforme
Projects
None yet
Development

No branches or pull requests

2 participants