Cyberduck Mountain Duck CLI

#7450 closed defect (fixed)

Interoperability with Swift S3 emulation layer

Reported by: din Owned by: dkocher
Priority: normal Milestone: 4.4
Component: openstack Version: Nightly Build
Severity: normal Keywords:
Cc: Architecture: Intel
Platform: Windows 7

Description

After change in #7435 cyberduck use server value to connect but it cause exception.

cyberduck.log:

2013-09-16 09:58:49,361 [background-1] FATAL ch.cyberduck.ui.AbstractController - Unhandled exception running background task null
java.lang.NullPointerException
	at ch.cyberduck.core.s3.S3BucketListService.list(S3BucketListService.java:80)
	at ch.cyberduck.core.s3.S3Session.list(S3Session.java:323)
	at ch.cyberduck.core.s3.S3Session.login(S3Session.java:312)
	at ch.cyberduck.core.KeychainLoginService.login(KeychainLoginService.java:69)
	at ch.cyberduck.core.LoginConnectionService.login(LoginConnectionService.java:150)
	at ch.cyberduck.core.LoginConnectionService.connect(LoginConnectionService.java:127)
	at ch.cyberduck.core.LoginConnectionService.check(LoginConnectionService.java:67)
	at ch.cyberduck.core.threading.SessionBackgroundAction.call(SessionBackgroundAction.java:187)
	at ch.cyberduck.ui.AbstractController$BackgroundCallable.call(AbstractController.java:149)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:167)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:604)
	at ch.cyberduck.core.threading.NamedThreadFactory$1.run(NamedThreadFactory.java:50)
	at java.lang.Thread.run(Thread.java:883)
	at cli.System.Threading.ThreadHelper.ThreadStart_Context(Unknown Source)
	at cli.System.Threading.ExecutionContext.runTryCode(Unknown Source)
	at cli.System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(Unknown Source)

In fiddler for authentication request I see 200 OK response and list of containers/buckets:

<?xml version="1.0" encoding="UTF-8"?>
<ListAllMyBucketsResult xmlns="http://doc.s3.amazonaws.com/2006-03-01">
  <Buckets>
    <Bucket>
      <Name>clitest</Name>
      <CreationDate>2009-02-03T16:45:09.000Z</CreationDate>
    </Bucket>
    <Bucket>
      <Name>kontener</Name>
      <CreationDate>2009-02-03T16:45:09.000Z</CreationDate>
    </Bucket>
    <Bucket>
      <Name>lista</Name>
      <CreationDate>2009-02-03T16:45:09.000Z</CreationDate>
    </Bucket>
    <Bucket>
      <Name>rozmiar</Name>
      <CreationDate>2009-02-03T16:45:09.000Z</CreationDate>
    </Bucket>
    <Bucket>
      <Name>strona</Name>
      <CreationDate>2009-02-03T16:45:09.000Z</CreationDate>
    </Bucket>
    <Bucket>
      <Name>write</Name>
      <CreationDate>2009-02-03T16:45:09.000Z</CreationDate>
    </Bucket>
  </Buckets>
</ListAllMyBucketsResult>

Tested on build 4.4(12906)

Change History (18)

comment:1 Changed on Sep 16, 2013 at 9:55:42 AM by dkocher

  • Milestone set to 4.4
  • Status changed from new to assigned

comment:2 Changed on Sep 16, 2013 at 9:56:35 AM by dkocher

Regardless that this needs to fixed what is the reason you want to use the emulation layer?

comment:3 follow-up: Changed on Sep 16, 2013 at 10:39:52 AM by din

I have swift cluster with swauth authentication and as described here https://trac.cyberduck.ch/wiki/help/en/howto/openstack before you can connect using cyberduck you must edit authentication context path. For some users it is strange and sometimes they complains about this, on the other side when using s3 connection they don't have to modify files by hand.

Now I see that I can create profile file like for keystone authentication so it could be workaround.

comment:4 in reply to: ↑ 3 Changed on Sep 16, 2013 at 11:26:32 AM by dkocher

Replying to din:

I have swift cluster with swauth authentication and as described here https://trac.cyberduck.ch/wiki/help/en/howto/openstack before you can connect using cyberduck you must edit authentication context path. For some users it is strange and sometimes they complains about this, on the other side when using s3 connection they don't have to modify files by hand.

Now I see that I can create profile file like for keystone authentication so it could be workaround.

The profiling feature is intended for such setups. Let me know if you need assistance in setting up a working profile.

comment:5 Changed on Sep 16, 2013 at 3:07:57 PM by dkocher

Owner is an optional element in the ListBucketResult response (minOccurs=0) and we should check for null.

comment:6 Changed on Sep 16, 2013 at 3:09:45 PM by dkocher

  • Resolution set to fixed
  • Status changed from assigned to closed

In r12918.

comment:7 follow-up: Changed on Sep 19, 2013 at 8:22:40 AM by din

Now I can connect and list containers but when trying get content of container there is a request:

GET /container/?versioning

which is not handle by s3 emulation in swift: Amazon_S3_REST_API_Compatability and cause 403 Forbidden response, but I think it is a problem of implementation s3 api in swift.

Thank you for all your help, I think that profiling feature is a best option for me now (created one based on examples).

comment:8 in reply to: ↑ 7 Changed on Sep 19, 2013 at 8:27:38 AM by dkocher

Replying to din:

Now I can connect and list containers but when trying get content of container there is a request:

GET /container/?versioning

which is not handle by s3 emulation in swift: Amazon_S3_REST_API_Compatability and cause 403 Forbidden response, but I think it is a problem of implementation s3 api in swift.

Thank you for all your help, I think that profiling feature is a best option for me now (created one based on examples).

We fail gracefully when versioning is not supported by the S3 provider.

comment:9 Changed on Sep 19, 2013 at 8:47:20 AM by din

From Log Drawer:

GET / HTTP/1.1
Date: Thu, 19 Sep 2013 08:30:03 GMT
Authorization: AWS TestUser1234:admin:unk6iwxHdOi/WxR7WnQFYlvoO68=
Host: example.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (12996) (Windows 7/6.1) (x86)
HTTP/1.1 200 OK
Content-Type: application/xml; charset=UTF-8
Content-Length: 702
X-Trans-Id: txff2b3d026951463d996c7bd0853b53c5
Date: Thu, 19 Sep 2013 08:30:09 GMT
Connection: keep-alive

GET /lista/?delimiter=%2F&max-keys=1000&prefix HTTP/1.1
Date: Thu, 19 Sep 2013 08:30:15 GMT
Authorization: AWS TestUser1234:admin:5njQ5dgJkhjQ6LjvCQLNTwl0agg=
Host: example.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (12996) (Windows 7/6.1) (x86)
HTTP/1.1 200 OK
Content-Type: application/xml; charset=UTF-8
Content-Length: 450
X-Trans-Id: tx73f9666f6b21474d87bee3af90617808
Date: Thu, 19 Sep 2013 08:30:21 GMT
Connection: keep-alive

GET /lista/?versioning HTTP/1.1
Date: Thu, 19 Sep 2013 08:30:15 GMT
Authorization: AWS TestUser1234:admin:O6IDf+5SCMBpmjAiYTlH7NNbrM4=
Host: example.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (12996) (Windows 7/6.1) (x86)
HTTP/1.1 400 Bad Request
Content-Type: text/xml; charset=UTF-8
Content-Length: 142
X-Trans-Id: tx8b3f028c2dd74a68b1ea9124097175a3
Date: Thu, 19 Sep 2013 08:30:21 GMT
Connection: keep-alive

and another session

GET / HTTP/1.1
Date: Thu, 19 Sep 2013 08:35:38 GMT
Authorization: AWS TestUser1234:admin:AWj9GtjljRq8FQcasqVeRPkg17I=
Host: example.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (12996) (Windows 7/6.1) (x86)
HTTP/1.1 200 OK
Content-Type: application/xml; charset=UTF-8
Content-Length: 702
X-Trans-Id: tx99b3d7be15ca48e5b6ed788463435311
Date: Thu, 19 Sep 2013 08:35:44 GMT
Connection: keep-alive

GET /clitest/?delimiter=%2F&max-keys=1000&prefix HTTP/1.1
Date: Thu, 19 Sep 2013 08:35:58 GMT
Authorization: AWS TestUser1234:admin:f8235qva1ntjKebP+uXxiLNO/jo=
Host: example.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (12996) (Windows 7/6.1) (x86)
HTTP/1.1 200 OK
Content-Type: application/xml; charset=UTF-8
Content-Length: 504
X-Trans-Id: tx2c69f36f851f45e89ddd7300776e8929
Date: Thu, 19 Sep 2013 08:36:04 GMT
Connection: keep-alive

HEAD /clitest/katalog HTTP/1.1
Date: Thu, 19 Sep 2013 08:35:58 GMT
Authorization: AWS TestUser1234:admin:q35JoqWEvey1hL8zmI8HxNT604A=
Host: example.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (12996) (Windows 7/6.1) (x86)
HTTP/1.1 200 OK
Last-Modified: Sun, 09 Jun 2013 13:32:33 GMT
Content-Length: 0
Etag: d41d8cd98f00b204e9800998ecf8427e
Content-Type: application/directory
X-Trans-Id: txc4d4f7fc5ef64649b20827d2e7a45cc9
Date: Thu, 19 Sep 2013 08:36:04 GMT
Connection: keep-alive

GET /clitest/?versioning HTTP/1.1
Date: Thu, 19 Sep 2013 08:35:58 GMT
Authorization: AWS TestUser1234:admin:HL06OFHlbAE6OOGkd4/PyBLIyTU=
Host: example.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (12996) (Windows 7/6.1) (x86)
HTTP/1.1 403 Forbidden
Content-Type: text/xml; charset=UTF-8
Content-Length: 124
X-Trans-Id: tx826f30b0a1984d04b2d6fb1ca60c01fd
Date: Thu, 19 Sep 2013 08:36:04 GMT
Connection: keep-alive

comment:10 follow-up: Changed on Sep 19, 2013 at 8:54:26 AM by dkocher

Does this cause a connection failure with an error message?

comment:11 in reply to: ↑ 10 ; follow-up: Changed on Sep 19, 2013 at 8:55:01 AM by dkocher

Replying to dkocher:

Does this cause a connection failure with an error message?

I see we only handle permission failures silently.

comment:12 in reply to: ↑ 11 Changed on Sep 19, 2013 at 8:58:02 AM by dkocher

Replying to dkocher:

Replying to dkocher:

Does this cause a connection failure with an error message?

I see we only handle permission failures silently.

In r12998.

comment:13 Changed on Oct 7, 2013 at 9:32:14 AM by din

  • Resolution fixed deleted
  • Status changed from closed to reopened

sorry for late response but problem seems still exist when try list container content I get error:

Cannot read container configuration

Log Drawer:

GET /write/?delimiter=%2F&max-keys=1000&prefix HTTP/1.1
Date: Mon, 07 Oct 2013 09:02:18 GMT
Authorization: AWS TestUser1234:admin:FrJ7iRMUV5sOsfJHym+GQdMLaIY=
Host: example.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (13235) (Windows 7/6.1) (x86)
HTTP/1.1 200 OK
Content-Type: application/xml; charset=UTF-8
Content-Length: 450
X-Trans-Id: tx26910a1a0e8c4c489501c954130bcfc8
Date: Mon, 07 Oct 2013 09:02:19 GMT
Connection: keep-alive

GET /write/?versioning HTTP/1.1
Date: Mon, 07 Oct 2013 09:02:19 GMT
Authorization: AWS TestUser1234:admin:2EtrCC+NeBMLgr4Mw6JUjfQkFYQ=
Host: example.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (13235) (Windows 7/6.1) (x86)
HTTP/1.1 403 Forbidden
Content-Type: text/xml; charset=UTF-8
Content-Length: 124
X-Trans-Id: tx9c1682e206b14d7fb25d35da166b3adb
Date: Mon, 07 Oct 2013 09:02:19 GMT
Connection: keep-alive

with s3curl (which not use versioning) retrieving container content works.

comment:14 Changed on Oct 7, 2013 at 9:43:31 AM by dkocher

With the patch in r12998 we expect a 400 error response if the versioning feature is not supported.

comment:15 Changed on Oct 7, 2013 at 9:44:05 AM by dkocher

  • Component changed from s3 to openstack
  • Summary changed from Connect to swift S3 emulation cause exception to Interoperability with Swift S3 emulation layer

comment:16 Changed on Oct 7, 2013 at 9:50:05 AM by dkocher

Handling 403 respnoses was supposed to be fixed already (and errors silently ignored) in r12517.

comment:17 Changed on Oct 8, 2013 at 9:58:47 AM by dkocher

  • Resolution set to fixed
  • Status changed from reopened to closed

In r13276.

comment:18 Changed on Oct 8, 2013 at 2:25:39 PM by din

Thank you for fast fix. Find out that this problem was also fixed in newer version of s3 emulation middleware in swift: https://github.com/fujita/swift3/commit/8883a4ef639229ccffa58595f4164365e2c9dd67 which unfortunately is not in ubuntu 12.04 repositories.

In test environment (SAIO ) where is newer swift3 version get

HTTP/1.1 200 OK 
<VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"/>

when request container with versioning query using s3curl.

Note: See TracTickets for help on using tickets.
swiss made software