Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ListObjects for directories over 1000 files doesn't work for S3 compatible storage #10995

Closed
cyberduck opened this issue Mar 17, 2020 · 6 comments
Assignees
Labels
bug fixed s3 AWS S3 Protocol Implementation
Milestone

Comments

@cyberduck
Copy link
Collaborator

5bd4b7e created the issue

Starting with Cyberduck 7.1.0, I've noticed that when connecting to a private S3 compatible storage service, I can no longer list files in a directory where there are over 1000 files/keys present. The UI presents itself as if there were no files in the directory.

In Cyberduck 7.0.2, executing the same scenario results in displaying the >1000 list of keys.

When turning on debugging, I see the following:

> GET /bucket/?encoding-type=url&max-keys=1000&prefix=path%2Fto%2Flarge%2Fdirectory%2F&delimiter=%2F HTTP/1.1
This returns a 200.

I also see

> GET /bucket/?encoding-type=url&max-keys=1000&prefix=path%2Fto%2Flarge%2Fdirectory%2F&delimiter=%2F&marker=path%2Fto%2Flarge%2Fdirectory%2Fmarker.json HTTP/1.1

This also returns a 200.

Then I see the following

2020-03-17 13:00:23.199539-0400 0x7427b8   Default     0xaecc6e             88140  0    Cyberduck: (libcore.dylib) [ch.sudo.cyberduck:Thread-40 ch.cyberduck.core.threading.BackgroundCallable] Failure java.lang.IllegalArgumentException: URLDecoder: Illegal hex characters in escape (%) pattern - Error at index 1 in: "ap" running background task
java.lang.Exception
	at ch.cyberduck.core.threading.BackgroundCallable.<init>(BackgroundCallable.java:36)
	at ch.cyberduck.core.threading.DefaultBackgroundExecutor.execute(DefaultBackgroundExecutor.java:67)
	at ch.cyberduck.core.AbstractController.background(AbstractController.java:71)
	at ch.cyberduck.ui.cocoa.controller.BrowserController.reload(BrowserController.java:517)
	at ch.cyberduck.ui.cocoa.controller.BrowserController.setWorkdir(BrowserController.java:3064)
	at ch.cyberduck.ui.cocoa.controller.BrowserController.setWorkdir(BrowserController.java:3039)
	at ch.cyberduck.ui.cocoa.controller.BrowserController.insideButtonClicked(BrowserController.java:2705)
	at ch.cyberduck.ui.cocoa.controller.BrowserController$AbstractBrowserTableDelegate.tableRowDoubleClicked(BrowserController.java:3597)
	at java.base/jdk.internal.reflect.NativeMethodAccessorI
2020-03-17 13:00:23.199617-0400 0x7427b8   Info        0xaecc6e             88140  0    Cyberduck: (libcore.dylib) [ch.sudo.cyberduck:Thread-40 ch.cyberduck.core.threading.SessionBackgroundAction] Run alert callback ch.cyberduck.ui.cocoa.callback.PromptAlertCallback@8102284 for failure BackgroundException{class=class ch.cyberduck.core.exception.ConnectionCanceledException, file=null, message='Error', detail='Unknown application error. URLDecoder: Illegal hex characters in escape (%) pattern - Error at index 1 in: "ap".', cause='java.lang.IllegalArgumentException: URLDecoder: Illegal hex characters in escape (%) pattern - Error at index 1 in: "ap"'}

I'm having issues building Cyberduck on macOS 10.15.3 (most likely due to my own incompetence) so I can't quite debug the exact issue. If I can figure that out, I can attempt to create a patch fixing or working around my issue.

Reading through code, I think it may be related to 07e0e54.

I'm not blocked since I have a work around (using 7.0.2) but figured that I might create a ticket for this anyway to track.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Unknown application error. URLDecoder: Illegal hex characters in escape (%) pattern - Error at index 1 in: "ap".'

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Is it possible to get a temporary account for testing on the server?

@cyberduck
Copy link
Collaborator Author

5bd4b7e commented

Is it possible to get a temporary account for testing on the server?

Unlikely since it's a private instance. However, if you have a build, I'd be happy to test it for you.

@cyberduck
Copy link
Collaborator Author

@ylangisc commented

In bb45f1a.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Can you please test with the latest snapshot build 7.3.1.32695.

@cyberduck
Copy link
Collaborator Author

5bd4b7e commented

Replying to [comment:8 dkocher]:

Can you please test with the latest snapshot build 7.3.1.32695.

Yes. I tested it on Version 7.3.1 (32695) and the directory with >1000 keys in it returned all the files under it.
Are there other scenarios you would like me to verify in this build for this fix?

Thank you so much for the fix.

@iterate-ch iterate-ch locked as resolved and limited conversation to collaborators Nov 27, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug fixed s3 AWS S3 Protocol Implementation
Projects
None yet
Development

No branches or pull requests

2 participants