Cyberduck Mountain Duck CLI

#10995 closed defect (fixed)

ListObjects for directories over 1000 files doesn't work for S3 compatible storage

Reported by: skier5 Owned by: dkocher
Priority: normal Milestone: 7.3.1
Component: s3 Version: 7.1
Severity: normal Keywords:
Cc: Architecture:
Platform: macOS 10.15

Description (last modified by dkocher)

Starting with Cyberduck 7.1.0, I've noticed that when connecting to a private S3 compatible storage service, I can no longer list files in a directory where there are over 1000 files/keys present. The UI presents itself as if there were no files in the directory.

In Cyberduck 7.0.2, executing the same scenario results in displaying the >1000 list of keys.

When turning on debugging, I see the following:

> GET /bucket/?encoding-type=url&max-keys=1000&prefix=path%2Fto%2Flarge%2Fdirectory%2F&delimiter=%2F HTTP/1.1
This returns a 200.

I also see

> GET /bucket/?encoding-type=url&max-keys=1000&prefix=path%2Fto%2Flarge%2Fdirectory%2F&delimiter=%2F&marker=path%2Fto%2Flarge%2Fdirectory%2Fmarker.json HTTP/1.1

This also returns a 200.

Then I see the following

2020-03-17 13:00:23.199539-0400 0x7427b8   Default     0xaecc6e             88140  0    Cyberduck: (libcore.dylib) [ch.sudo.cyberduck:Thread-40 ch.cyberduck.core.threading.BackgroundCallable] Failure java.lang.IllegalArgumentException: URLDecoder: Illegal hex characters in escape (%) pattern - Error at index 1 in: "ap" running background task
java.lang.Exception
	at ch.cyberduck.core.threading.BackgroundCallable.<init>(BackgroundCallable.java:36)
	at ch.cyberduck.core.threading.DefaultBackgroundExecutor.execute(DefaultBackgroundExecutor.java:67)
	at ch.cyberduck.core.AbstractController.background(AbstractController.java:71)
	at ch.cyberduck.ui.cocoa.controller.BrowserController.reload(BrowserController.java:517)
	at ch.cyberduck.ui.cocoa.controller.BrowserController.setWorkdir(BrowserController.java:3064)
	at ch.cyberduck.ui.cocoa.controller.BrowserController.setWorkdir(BrowserController.java:3039)
	at ch.cyberduck.ui.cocoa.controller.BrowserController.insideButtonClicked(BrowserController.java:2705)
	at ch.cyberduck.ui.cocoa.controller.BrowserController$AbstractBrowserTableDelegate.tableRowDoubleClicked(BrowserController.java:3597)
	at java.base/jdk.internal.reflect.NativeMethodAccessorI
2020-03-17 13:00:23.199617-0400 0x7427b8   Info        0xaecc6e             88140  0    Cyberduck: (libcore.dylib) [ch.sudo.cyberduck:Thread-40 ch.cyberduck.core.threading.SessionBackgroundAction] Run alert callback ch.cyberduck.ui.cocoa.callback.PromptAlertCallback@8102284 for failure BackgroundException{class=class ch.cyberduck.core.exception.ConnectionCanceledException, file=null, message='Error', detail='Unknown application error. URLDecoder: Illegal hex characters in escape (%) pattern - Error at index 1 in: "ap".', cause='java.lang.IllegalArgumentException: URLDecoder: Illegal hex characters in escape (%) pattern - Error at index 1 in: "ap"'}

I'm having issues building Cyberduck on macOS 10.15.3 (most likely due to my own incompetence) so I can't quite debug the exact issue. If I can figure that out, I can attempt to create a patch fixing or working around my issue.

Reading through code, I think it may be related to https://github.com/iterate-ch/cyberduck/commit/07e0e548684b7a65f2d1bcbc424da639ffeb4396.

I'm not blocked since I have a work around (using 7.0.2) but figured that I might create a ticket for this anyway to track.

Change History (10)

comment:1 Changed on Mar 17, 2020 at 5:28:45 PM by skier5

  • Description modified (diff)

comment:2 Changed on Mar 17, 2020 at 7:33:02 PM by dkocher

  • Description modified (diff)

comment:3 Changed on Mar 22, 2020 at 1:31:59 PM by dkocher

  • Owner set to dkocher
  • Status changed from new to assigned
Unknown application error. URLDecoder: Illegal hex characters in escape (%) pattern - Error at index 1 in: "ap".'

comment:4 Changed on Apr 15, 2020 at 2:58:58 PM by dkocher

Is it possible to get a temporary account for testing on the server?

comment:5 Changed on Apr 15, 2020 at 3:05:31 PM by skier5

Is it possible to get a temporary account for testing on the server?

Unlikely since it's a private instance. However, if you have a build, I'd be happy to test it for you.

comment:6 Changed on Apr 15, 2020 at 3:11:44 PM by dkocher

  • Milestone set to 8.0

comment:7 Changed on Apr 16, 2020 at 7:04:56 AM by yla

  • Resolution set to fixed
  • Status changed from assigned to closed

In r49035.

comment:8 follow-up: Changed on Apr 16, 2020 at 8:49:01 AM by dkocher

Can you please test with the latest snapshot build 7.3.1.32695.

comment:9 Changed on Apr 16, 2020 at 9:15:07 AM by dkocher

  • Milestone changed from 8.0 to 7.3.1

comment:10 in reply to: ↑ 8 Changed on Apr 16, 2020 at 12:41:16 PM by skier5

Replying to dkocher:

Can you please test with the latest snapshot build 7.3.1.32695.

Yes. I tested it on Version 7.3.1 (32695) and the directory with >1000 keys in it returned all the files under it. Are there other scenarios you would like me to verify in this build for this fix?

Thank you so much for the fix.

Note: See TracTickets for help on using tickets.
swiss made software