Cyberduck Mountain Duck CLI

#10888 closed defect (fixed)

Interoperability with Minio

Reported by: si458 Owned by: dkocher
Priority: normal Milestone: 7.2.2
Component: s3 Version: 7.1.2
Severity: normal Keywords:
Cc: Architecture: Intel
Platform: macOS 10.15

Description

7.1.2 works without issues

7.2.0 onwards keeps getting an error on just simply listing a bucket contents

Request Error: null. Please contact your web hosting service provider for assistance.

server: minio 2019-10-12T01:39:57Z with --compat option

Change History (28)

comment:1 Changed on Nov 27, 2019 at 11:57:16 AM by si458

  • Priority changed from normal to high

THIS IS THE CONSOLE DEBUG OUTPUT IF I CONNECT TO THE MINIO SERVER IP AND PORT AND THEN ASK TO LIST CONTENTS TO MYBUCKET123

default	11:56:20.321525+0000	Cyberduck	Not supported to read versioning configuration for Path{path='/mybucket123', type=[directory, volume]} Request Error: null

default	11:56:20.323044+0000	Cyberduck	No retry for failure BackgroundException{file=Path{path='/mybucket123', type=[directory, volume]}, message='Listing directory mybucket123 failed.', detail='Request Error: null.', class='ch.cyberduck.core.exception.InteroperabilityException', cause='org.jets3t.service.ServiceException: Request Error: null'}

default	11:56:20.323921+0000	Cyberduck	Failure BackgroundException{file=Path{path='/mybucket123', type=[directory, volume]}, message='Listing directory mybucket123 failed.', detail='Request Error: null.', class='ch.cyberduck.core.exception.InteroperabilityException', cause='org.jets3t.service.ServiceException: Request Error: null'} running background task
java.lang.Exception
	at ch.cyberduck.core.threading.BackgroundCallable.<init>(BackgroundCallable.java:36)
	at ch.cyberduck.core.threading.DefaultBackgroundExecutor.execute(DefaultBackgroundExecutor.java:67)
	at ch.cyberduck.core.AbstractController.background(AbstractController.java:71)
	at ch.cyberduck.ui.cocoa.controller.BrowserController.reload(BrowserController.java:511)
	at ch.cyberduck.ui.cocoa.controller.BrowserController.setWorkdir(BrowserController.java:3041)
	at ch.cyberduck.ui.cocoa.controller.BrowserController.setWorkdir(BrowserController.java:3016)
	at ch.cyberduck.ui.cocoa.controller.BrowserController.insideButtonClicked(BrowserController.java:2682)
	at ch.cyberduck.ui.cocoa

default	11:56:21.481405+0000	Cyberduck	Ignore return code -1

default	11:56:21.512844+0000	Cyberduck	Missing result for worker SessionListWorker{directory=Path{path='/mybucket123', type=[directory, volume]}}. Use default value.

Last edited on Nov 27, 2019 at 12:59:00 PM by dkocher (previous) (diff)

comment:2 Changed on Nov 27, 2019 at 1:01:42 PM by si458

THIS IS THE CONSOLE DEBUG OUTPUT IF I USE THE VIRTUAL HOST METHOD IN MINIO e.g http://mybucket123.s3.myserver.com:9000

i have checked the XML output being returned from the minio server using Charles Proxy and its a VALID XML that gets returned with Contents listed for files

default	12:40:18.754559+0000	Cyberduck	No retry for failure BackgroundException{file=Path{path='/', type=[directory, volume]}, message='Listing directory / failed.', detail='Failed to parse XML document with handler class org.jets3t.service.impl.rest.XmlResponsesSaxParser$ListAllMyBucketsHandler.', class='ch.cyberduck.core.exception.InteroperabilityException', cause='org.jets3t.service.S3ServiceException: Failed to parse XML document with handler class org.jets3t.service.impl.rest.XmlResponsesSaxParser$ListAllMyBucketsHandler'}

default	12:40:18.755397+0000	Cyberduck	Failure BackgroundException{file=Path{path='/', type=[directory, volume]}, message='Listing directory / failed.', detail='Failed to parse XML document with handler class org.jets3t.service.impl.rest.XmlResponsesSaxParser$ListAllMyBucketsHandler.', class='ch.cyberduck.core.exception.InteroperabilityException', cause='org.jets3t.service.S3ServiceException: Failed to parse XML document with handler class org.jets3t.service.impl.rest.XmlResponsesSaxParser$ListAllMyBucketsHandler'} running background task
java.lang.Exception
	at ch.cyberduck.core.threading.BackgroundCallable.<init>(BackgroundCallable.java:36)
	at ch.cyberduck.core.threading.DefaultBackgroundExecutor.execute(DefaultBackgroundExecutor.java:67)
	at ch.cyberduck.core.AbstractController.background(AbstractController.java:71)
	at ch.cyberduck.ui.cocoa.controller.BrowserController$29.run(BrowserController.java:3070)
	at ch.cyberduck.ui.cocoa.controller.BrowserController$32.run(BrowserController.java:3189)
	at ch.cyberduck.ui.cocoa.controller.Bro
default	12:40:20.375029+0000	Cyberduck	Ignore return code -1
default	12:40:20.392688+0000	Cyberduck	Missing result for worker MountWorker{cache=ch.cyberduck.core.ReverseLookupCache@3eee3e2b}. Use default value.

comment:3 follow-up: Changed on Nov 27, 2019 at 1:04:50 PM by dkocher

Please make sure to enable ​virtual-host-style requests in their Minio configuration.

comment:4 Changed on Nov 27, 2019 at 1:05:55 PM by dkocher

  • Component changed from core to s3
  • Owner set to dkocher
  • Priority changed from high to normal
  • Severity changed from critical to normal

comment:5 in reply to: ↑ 3 Changed on Nov 27, 2019 at 1:12:15 PM by si458

Replying to dkocher:

Please make sure to enable ​virtual-host-style requests in their Minio configuration.

this is already enabled, as i couldnt access it first time round without setting the MINIO_DOMAIN in docker as it generated a different error

but after enabling it, im now getting this XML error as listed above

Last edited on Nov 27, 2019 at 1:12:24 PM by si458 (previous) (diff)

comment:6 Changed on Nov 27, 2019 at 1:27:01 PM by dkocher

  • Milestone set to 8.0
  • Summary changed from update 7.2.0 onwards seems to kill minio s3 support to Failed to parse XML document (Minio)

comment:7 Changed on Nov 27, 2019 at 1:32:55 PM by si458

  • Version changed from 7.2 to 7.1.2

ive just tried 7.1.2 with virtual-host-style and that doesn't work either

just keeps getting same error

Failed to parse XML document with handler class org.jets3t.service.impl.rest.XmlResponsesSaxParser$ListAllMyBucketsHandler. Please contact your web hosting service provider for assistance

but the log drawer show it returns HTTP/1.1 200 OK with a valid XML too

comment:8 Changed on Nov 27, 2019 at 1:34:25 PM by dkocher

Regression from r48152.

comment:9 follow-up: Changed on Nov 27, 2019 at 1:35:10 PM by dkocher

Can you provide the captured XML response?

comment:10 in reply to: ↑ 9 Changed on Nov 27, 2019 at 1:49:24 PM by si458

Replying to dkocher:

Can you provide the captured XML response?

https://pastebin.com/7knB6DcW

comment:11 follow-ups: Changed on Nov 27, 2019 at 2:04:15 PM by si458

ok i think i might have found the problem and maybe a bug but will try explain

  1. if you use an IP address instead of a hostname, it looks for http://wtf123.192.168.1.100:9000/ which is invalid,

so maybe a checker if i use an IP address instead of a hostname then ALWAYS USE PATH-STYLE !!!

  1. if i use a hostname with minio and have virtual-host-style DISABLED, cyberduck ALWAYS trys accessing the bucket using virtual-host-style regardless e.g: http://wtf123.s3.myserver.com:12345/

so i think it would be nice to have an option inside the connection settings to specify weather to use virtual-host-style or path-style (maybe per connection?)

  1. if i use the full virtual-host-style as the hostname e.g: http://wtf123.s3.myserver.com:12345/

i get an XML errror, even tho the XML is valid and returned correctly? https://pastebin.com/7knB6DcW

Last edited on Nov 27, 2019 at 2:09:49 PM by si458 (previous) (diff)

comment:12 Changed on Nov 27, 2019 at 2:43:13 PM by dkocher

Thanks for summarizing the issues. Accessing via the hostname set with MINIO_DOMAIN this cannot be reproduced.

comment:13 Changed on Nov 27, 2019 at 2:44:04 PM by dkocher

The error when accessing using the bucket name in Host configuration setting of the bookmark is that we expect the server to return a bucket list and not an object list.

comment:14 Changed on Nov 27, 2019 at 2:52:04 PM by dkocher

We have requested a documentation update for Minio in https://github.com/minio/cookbook/issues/280.

comment:15 follow-up: Changed on Nov 27, 2019 at 2:55:13 PM by si458

surely it would be a better option to give the user control over path-style or virtual-host-style than forcing the user to use virtual-host-style?

comment:16 in reply to: ↑ 15 ; follow-up: Changed on Nov 27, 2019 at 3:07:52 PM by dkocher

Replying to si458:

surely it would be a better option to give the user control over path-style or virtual-host-style than forcing the user to use virtual-host-style?

We will see if we can support this via an option in the connection profiles.

Last edited on Nov 27, 2019 at 3:40:11 PM by si458 (previous) (diff)

comment:17 in reply to: ↑ 16 Changed on Nov 27, 2019 at 3:40:21 PM by si458

Replying to dkocher:

Replying to si458:

surely it would be a better option to give the user control over path-style or virtual-host-style than forcing the user to use virtual-host-style?

We will see if we can support this via an option in the connection profiles.

thank you that would be helpful!

or even a tick box saying saying 'USE PATH-STYLE' in the edit bookmarks window/open connection window

comment:18 in reply to: ↑ 11 Changed on Nov 28, 2019 at 9:47:57 AM by dkocher

  • Resolution set to fixed
  • Status changed from new to closed

Replying to si458:

ok i think i might have found the problem and maybe a bug but will try explain

  1. if you use an IP address instead of a hostname, it looks for http://wtf123.192.168.1.100:9000/ which is invalid,

so maybe a checker if i use an IP address instead of a hostname then ALWAYS USE PATH-STYLE !!!

In r48257.

comment:19 Changed on Nov 28, 2019 at 9:55:04 AM by dkocher

  • Resolution fixed deleted
  • Status changed from closed to reopened

Pending change for Failed to parse XML document error to detect if we are connecting to a specific bucket and should list objects instead of all buckets.

comment:20 in reply to: ↑ 11 Changed on Nov 28, 2019 at 2:03:07 PM by dkocher

Replying to si458:

  1. if i use the full virtual-host-style as the hostname e.g: http://wtf123.s3.myserver.com:12345/

i get an XML errror, even tho the XML is valid and returned correctly? https://pastebin.com/7knB6DcW

Set the bucket name in the Path setting in the bookmark instead.

comment:21 Changed on Nov 28, 2019 at 8:14:56 PM by dkocher

  • Summary changed from Failed to parse XML document (Minio) to Interoperability with Minio

comment:22 Changed on Dec 6, 2019 at 2:25:43 PM by dkocher

  • Milestone changed from 8.0 to 7.2.2

comment:23 Changed on Dec 9, 2019 at 7:45:19 AM by yla

  • Resolution set to fixed
  • Status changed from reopened to closed

In r48333.

comment:24 follow-up: Changed on Jan 7, 2020 at 6:43:16 PM by dimm00

Trying to make cyberduck work with ceph S3, which only supports path style Tried using IP, which broke the cert and didn't work either Is there a way to reenable path style S3?

comment:25 in reply to: ↑ 24 ; follow-up: Changed on Jan 10, 2020 at 7:54:50 PM by dkocher

Replying to dimm00:

Trying to make cyberduck work with ceph S3, which only supports path style Tried using IP, which broke the cert and didn't work either Is there a way to reenable path style S3?

Is this Ceph instance publicly reachable over the Internet?

comment:26 in reply to: ↑ 25 ; follow-up: Changed on Jan 11, 2020 at 7:06:46 AM by dimm00

Replying to dkocher:

Replying to dimm00:

Trying to make cyberduck work with ceph S3, which only supports path style Tried using IP, which broke the cert and didn't work either Is there a way to reenable path style S3?

Is this Ceph instance publicly reachable over the Internet?

Yes. How can I send you credentials to access it?

comment:27 in reply to: ↑ 26 Changed on Jan 22, 2020 at 1:55:45 PM by dkocher

Replying to dimm00:

Replying to dkocher:

Replying to dimm00:

Trying to make cyberduck work with ceph S3, which only supports path style Tried using IP, which broke the cert and didn't work either Is there a way to reenable path style S3?

Is this Ceph instance publicly reachable over the Internet?

Yes. How can I send you credentials to access it?

Awesome. Please write to support@….

comment:28 Changed on Jan 22, 2020 at 1:56:52 PM by dkocher

Duplicate for #10956.

Note: See TracTickets for help on using tickets.
swiss made software