Cyberduck Mountain Duck CLI

#10888 closed defect (fixed)

Interoperability with Minio

Reported by: si458 Owned by: dkocher
Priority: normal Milestone: 7.2.2
Component: s3 Version: 7.1.2
Severity: normal Keywords:
Cc: cz@… Architecture: Intel
Platform: macOS 10.15

Description

7.1.2 works without issues

7.2.0 onwards keeps getting an error on just simply listing a bucket contents

Request Error: null. Please contact your web hosting service provider for assistance.

server: minio 2019-10-12T01:39:57Z with --compat option

Change History (33)

comment:1 Changed on Nov 27, 2019 at 11:57:16 AM by si458

  • Priority changed from normal to high

THIS IS THE CONSOLE DEBUG OUTPUT IF I CONNECT TO THE MINIO SERVER IP AND PORT AND THEN ASK TO LIST CONTENTS TO MYBUCKET123

default	11:56:20.321525+0000	Cyberduck	Not supported to read versioning configuration for Path{path='/mybucket123', type=[directory, volume]} Request Error: null

default	11:56:20.323044+0000	Cyberduck	No retry for failure BackgroundException{file=Path{path='/mybucket123', type=[directory, volume]}, message='Listing directory mybucket123 failed.', detail='Request Error: null.', class='ch.cyberduck.core.exception.InteroperabilityException', cause='org.jets3t.service.ServiceException: Request Error: null'}

default	11:56:20.323921+0000	Cyberduck	Failure BackgroundException{file=Path{path='/mybucket123', type=[directory, volume]}, message='Listing directory mybucket123 failed.', detail='Request Error: null.', class='ch.cyberduck.core.exception.InteroperabilityException', cause='org.jets3t.service.ServiceException: Request Error: null'} running background task
java.lang.Exception
	at ch.cyberduck.core.threading.BackgroundCallable.<init>(BackgroundCallable.java:36)
	at ch.cyberduck.core.threading.DefaultBackgroundExecutor.execute(DefaultBackgroundExecutor.java:67)
	at ch.cyberduck.core.AbstractController.background(AbstractController.java:71)
	at ch.cyberduck.ui.cocoa.controller.BrowserController.reload(BrowserController.java:511)
	at ch.cyberduck.ui.cocoa.controller.BrowserController.setWorkdir(BrowserController.java:3041)
	at ch.cyberduck.ui.cocoa.controller.BrowserController.setWorkdir(BrowserController.java:3016)
	at ch.cyberduck.ui.cocoa.controller.BrowserController.insideButtonClicked(BrowserController.java:2682)
	at ch.cyberduck.ui.cocoa

default	11:56:21.481405+0000	Cyberduck	Ignore return code -1

default	11:56:21.512844+0000	Cyberduck	Missing result for worker SessionListWorker{directory=Path{path='/mybucket123', type=[directory, volume]}}. Use default value.

Last edited on Nov 27, 2019 at 12:59:00 PM by dkocher (previous) (diff)

comment:2 Changed on Nov 27, 2019 at 1:01:42 PM by si458

THIS IS THE CONSOLE DEBUG OUTPUT IF I USE THE VIRTUAL HOST METHOD IN MINIO e.g http://mybucket123.s3.myserver.com:9000

i have checked the XML output being returned from the minio server using Charles Proxy and its a VALID XML that gets returned with Contents listed for files

default	12:40:18.754559+0000	Cyberduck	No retry for failure BackgroundException{file=Path{path='/', type=[directory, volume]}, message='Listing directory / failed.', detail='Failed to parse XML document with handler class org.jets3t.service.impl.rest.XmlResponsesSaxParser$ListAllMyBucketsHandler.', class='ch.cyberduck.core.exception.InteroperabilityException', cause='org.jets3t.service.S3ServiceException: Failed to parse XML document with handler class org.jets3t.service.impl.rest.XmlResponsesSaxParser$ListAllMyBucketsHandler'}

default	12:40:18.755397+0000	Cyberduck	Failure BackgroundException{file=Path{path='/', type=[directory, volume]}, message='Listing directory / failed.', detail='Failed to parse XML document with handler class org.jets3t.service.impl.rest.XmlResponsesSaxParser$ListAllMyBucketsHandler.', class='ch.cyberduck.core.exception.InteroperabilityException', cause='org.jets3t.service.S3ServiceException: Failed to parse XML document with handler class org.jets3t.service.impl.rest.XmlResponsesSaxParser$ListAllMyBucketsHandler'} running background task
java.lang.Exception
	at ch.cyberduck.core.threading.BackgroundCallable.<init>(BackgroundCallable.java:36)
	at ch.cyberduck.core.threading.DefaultBackgroundExecutor.execute(DefaultBackgroundExecutor.java:67)
	at ch.cyberduck.core.AbstractController.background(AbstractController.java:71)
	at ch.cyberduck.ui.cocoa.controller.BrowserController$29.run(BrowserController.java:3070)
	at ch.cyberduck.ui.cocoa.controller.BrowserController$32.run(BrowserController.java:3189)
	at ch.cyberduck.ui.cocoa.controller.Bro
default	12:40:20.375029+0000	Cyberduck	Ignore return code -1
default	12:40:20.392688+0000	Cyberduck	Missing result for worker MountWorker{cache=ch.cyberduck.core.ReverseLookupCache@3eee3e2b}. Use default value.

comment:3 follow-up: Changed on Nov 27, 2019 at 1:04:50 PM by dkocher

Please make sure to enable ​virtual-host-style requests in their Minio configuration.

comment:4 Changed on Nov 27, 2019 at 1:05:55 PM by dkocher

  • Component changed from core to s3
  • Owner set to dkocher
  • Priority changed from high to normal
  • Severity changed from critical to normal

comment:5 in reply to: ↑ 3 Changed on Nov 27, 2019 at 1:12:15 PM by si458

Replying to dkocher:

Please make sure to enable ​virtual-host-style requests in their Minio configuration.

this is already enabled, as i couldnt access it first time round without setting the MINIO_DOMAIN in docker as it generated a different error

but after enabling it, im now getting this XML error as listed above

Last edited on Nov 27, 2019 at 1:12:24 PM by si458 (previous) (diff)

comment:6 Changed on Nov 27, 2019 at 1:27:01 PM by dkocher

  • Milestone set to 8.0
  • Summary changed from update 7.2.0 onwards seems to kill minio s3 support to Failed to parse XML document (Minio)

comment:7 Changed on Nov 27, 2019 at 1:32:55 PM by si458

  • Version changed from 7.2 to 7.1.2

ive just tried 7.1.2 with virtual-host-style and that doesn't work either

just keeps getting same error

Failed to parse XML document with handler class org.jets3t.service.impl.rest.XmlResponsesSaxParser$ListAllMyBucketsHandler. Please contact your web hosting service provider for assistance

but the log drawer show it returns HTTP/1.1 200 OK with a valid XML too

comment:8 Changed on Nov 27, 2019 at 1:34:25 PM by dkocher

Regression from r48152.

comment:9 follow-up: Changed on Nov 27, 2019 at 1:35:10 PM by dkocher

Can you provide the captured XML response?

comment:10 in reply to: ↑ 9 Changed on Nov 27, 2019 at 1:49:24 PM by si458

Replying to dkocher:

Can you provide the captured XML response?

https://pastebin.com/7knB6DcW

comment:11 follow-ups: Changed on Nov 27, 2019 at 2:04:15 PM by si458

ok i think i might have found the problem and maybe a bug but will try explain

  1. if you use an IP address instead of a hostname, it looks for http://wtf123.192.168.1.100:9000/ which is invalid,

so maybe a checker if i use an IP address instead of a hostname then ALWAYS USE PATH-STYLE !!!

  1. if i use a hostname with minio and have virtual-host-style DISABLED, cyberduck ALWAYS trys accessing the bucket using virtual-host-style regardless e.g: http://wtf123.s3.myserver.com:12345/

so i think it would be nice to have an option inside the connection settings to specify weather to use virtual-host-style or path-style (maybe per connection?)

  1. if i use the full virtual-host-style as the hostname e.g: http://wtf123.s3.myserver.com:12345/

i get an XML errror, even tho the XML is valid and returned correctly? https://pastebin.com/7knB6DcW

Last edited on Nov 27, 2019 at 2:09:49 PM by si458 (previous) (diff)

comment:12 Changed on Nov 27, 2019 at 2:43:13 PM by dkocher

Thanks for summarizing the issues. Accessing via the hostname set with MINIO_DOMAIN this cannot be reproduced.

comment:13 Changed on Nov 27, 2019 at 2:44:04 PM by dkocher

The error when accessing using the bucket name in Host configuration setting of the bookmark is that we expect the server to return a bucket list and not an object list.

comment:14 Changed on Nov 27, 2019 at 2:52:04 PM by dkocher

We have requested a documentation update for Minio in https://github.com/minio/cookbook/issues/280.

comment:15 follow-up: Changed on Nov 27, 2019 at 2:55:13 PM by si458

surely it would be a better option to give the user control over path-style or virtual-host-style than forcing the user to use virtual-host-style?

comment:16 in reply to: ↑ 15 ; follow-up: Changed on Nov 27, 2019 at 3:07:52 PM by dkocher

Replying to si458:

surely it would be a better option to give the user control over path-style or virtual-host-style than forcing the user to use virtual-host-style?

We will see if we can support this via an option in the connection profiles.

Last edited on Nov 27, 2019 at 3:40:11 PM by si458 (previous) (diff)

comment:17 in reply to: ↑ 16 Changed on Nov 27, 2019 at 3:40:21 PM by si458

Replying to dkocher:

Replying to si458:

surely it would be a better option to give the user control over path-style or virtual-host-style than forcing the user to use virtual-host-style?

We will see if we can support this via an option in the connection profiles.

thank you that would be helpful!

or even a tick box saying saying 'USE PATH-STYLE' in the edit bookmarks window/open connection window

comment:18 in reply to: ↑ 11 Changed on Nov 28, 2019 at 9:47:57 AM by dkocher

  • Resolution set to fixed
  • Status changed from new to closed

Replying to si458:

ok i think i might have found the problem and maybe a bug but will try explain

  1. if you use an IP address instead of a hostname, it looks for http://wtf123.192.168.1.100:9000/ which is invalid,

so maybe a checker if i use an IP address instead of a hostname then ALWAYS USE PATH-STYLE !!!

In r48257.

comment:19 Changed on Nov 28, 2019 at 9:55:04 AM by dkocher

  • Resolution fixed deleted
  • Status changed from closed to reopened

Pending change for Failed to parse XML document error to detect if we are connecting to a specific bucket and should list objects instead of all buckets.

comment:20 in reply to: ↑ 11 Changed on Nov 28, 2019 at 2:03:07 PM by dkocher

Replying to si458:

  1. if i use the full virtual-host-style as the hostname e.g: http://wtf123.s3.myserver.com:12345/

i get an XML errror, even tho the XML is valid and returned correctly? https://pastebin.com/7knB6DcW

Set the bucket name in the Path setting in the bookmark instead.

comment:21 Changed on Nov 28, 2019 at 8:14:56 PM by dkocher

  • Summary changed from Failed to parse XML document (Minio) to Interoperability with Minio

comment:22 Changed on Dec 6, 2019 at 2:25:43 PM by dkocher

  • Milestone changed from 8.0 to 7.2.2

comment:23 Changed on Dec 9, 2019 at 7:45:19 AM by yla

  • Resolution set to fixed
  • Status changed from reopened to closed

In r48333.

comment:24 follow-ups: Changed on Jan 7, 2020 at 6:43:16 PM by dimm00

Trying to make cyberduck work with ceph S3, which only supports path style Tried using IP, which broke the cert and didn't work either Is there a way to reenable path style S3?

comment:25 in reply to: ↑ 24 ; follow-up: Changed on Jan 10, 2020 at 7:54:50 PM by dkocher

Replying to dimm00:

Trying to make cyberduck work with ceph S3, which only supports path style Tried using IP, which broke the cert and didn't work either Is there a way to reenable path style S3?

Is this Ceph instance publicly reachable over the Internet?

comment:26 in reply to: ↑ 25 ; follow-up: Changed on Jan 11, 2020 at 7:06:46 AM by dimm00

Replying to dkocher:

Replying to dimm00:

Trying to make cyberduck work with ceph S3, which only supports path style Tried using IP, which broke the cert and didn't work either Is there a way to reenable path style S3?

Is this Ceph instance publicly reachable over the Internet?

Yes. How can I send you credentials to access it?

comment:27 in reply to: ↑ 26 Changed on Jan 22, 2020 at 1:55:45 PM by dkocher

Replying to dimm00:

Replying to dkocher:

Replying to dimm00:

Trying to make cyberduck work with ceph S3, which only supports path style Tried using IP, which broke the cert and didn't work either Is there a way to reenable path style S3?

Is this Ceph instance publicly reachable over the Internet?

Yes. How can I send you credentials to access it?

Awesome. Please write to support@….

comment:28 Changed on Jan 22, 2020 at 1:56:52 PM by dkocher

Duplicate for #10956.

comment:29 in reply to: ↑ 24 ; follow-up: Changed on Apr 24, 2020 at 11:30:37 AM by zagy__

Replying to dimm00:

Trying to make cyberduck work with ceph S3, which only supports path style Tried using IP, which broke the cert and didn't work either Is there a way to reenable path style S3?

Was that solved? I'm seeing the same problem with Ceph here.

comment:30 Changed on Apr 24, 2020 at 11:30:52 AM by zagy__

  • Cc cz@… added

comment:31 in reply to: ↑ 29 Changed on Apr 24, 2020 at 11:41:14 AM by si458

i dont have this issue anymore as its been fixed in latest cyberduck version using the S3 (HTTPS) profile from the minio help page

the only issue i found was when using my home broadband 'virgin media' they take over my dns records,

so when cyberduck was looking for 'mybucket.s3.myserver.com' is should of returned blank A record as i only have an A record for 's3.myserver.com' which ment cyberduck would then reverted back to using the path format,

but 'virgin media' kept returning an IP address of theres for there stupid advanced search error page, so cyberduck kept trying to connect to them and just timing out instead of an error

once i disabled the 'advanced search error page' and rebooted our router and cleared my dns cache on laptop,

cyberduck started working without any issues

Last edited on Apr 24, 2020 at 11:42:18 AM by si458 (previous) (diff)

comment:32 Changed on Apr 24, 2020 at 1:22:24 PM by zagy__

Aha. What worked for me now, was to create a bookmark and add the bucket-url to the PATH there. So, worksforme now. Thanks.

comment:33 Changed on Apr 24, 2020 at 3:52:13 PM by dkocher

We have another fix for Minio (#11018) that will be included in version 7.3.1.

Note: See TracTickets for help on using tickets.
swiss made software