Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Interoperability with Swift S3 emulation layer #7450

Closed
cyberduck opened this issue Sep 16, 2013 · 16 comments
Closed

Interoperability with Swift S3 emulation layer #7450

cyberduck opened this issue Sep 16, 2013 · 16 comments
Assignees
Labels
bug fixed openstack OpenStack Swift Protocol Implementation
Milestone

Comments

@cyberduck
Copy link
Collaborator

da06755 created the issue

After change in #7435 cyberduck use server value to connect but it cause exception.

cyberduck.log:

2013-09-16 09:58:49,361 [background-1] FATAL ch.cyberduck.ui.AbstractController - Unhandled exception running background task null
java.lang.NullPointerException
	at ch.cyberduck.core.s3.S3BucketListService.list(S3BucketListService.java:80)
	at ch.cyberduck.core.s3.S3Session.list(S3Session.java:323)
	at ch.cyberduck.core.s3.S3Session.login(S3Session.java:312)
	at ch.cyberduck.core.KeychainLoginService.login(KeychainLoginService.java:69)
	at ch.cyberduck.core.LoginConnectionService.login(LoginConnectionService.java:150)
	at ch.cyberduck.core.LoginConnectionService.connect(LoginConnectionService.java:127)
	at ch.cyberduck.core.LoginConnectionService.check(LoginConnectionService.java:67)
	at ch.cyberduck.core.threading.SessionBackgroundAction.call(SessionBackgroundAction.java:187)
	at ch.cyberduck.ui.AbstractController$BackgroundCallable.call(AbstractController.java:149)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
	at java.util.concurrent.FutureTask.run(FutureTask.java:167)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:604)
	at ch.cyberduck.core.threading.NamedThreadFactory$1.run(NamedThreadFactory.java:50)
	at java.lang.Thread.run(Thread.java:883)
	at cli.System.Threading.ThreadHelper.ThreadStart_Context(Unknown Source)
	at cli.System.Threading.ExecutionContext.runTryCode(Unknown Source)
	at cli.System.Runtime.CompilerServices.RuntimeHelpers.ExecuteCodeWithGuaranteedCleanup(Unknown Source)

In fiddler for authentication request I see 200 OK response and list of containers/buckets:

<?xml version="1.0" encoding="UTF-8"?>
<ListAllMyBucketsResult xmlns="http://doc.s3.amazonaws.com/2006-03-01">
  <Buckets>
    <Bucket>
      <Name>clitest</Name>
      <CreationDate>2009-02-03T16:45:09.000Z</CreationDate>
    </Bucket>
    <Bucket>
      <Name>kontener</Name>
      <CreationDate>2009-02-03T16:45:09.000Z</CreationDate>
    </Bucket>
    <Bucket>
      <Name>lista</Name>
      <CreationDate>2009-02-03T16:45:09.000Z</CreationDate>
    </Bucket>
    <Bucket>
      <Name>rozmiar</Name>
      <CreationDate>2009-02-03T16:45:09.000Z</CreationDate>
    </Bucket>
    <Bucket>
      <Name>strona</Name>
      <CreationDate>2009-02-03T16:45:09.000Z</CreationDate>
    </Bucket>
    <Bucket>
      <Name>write</Name>
      <CreationDate>2009-02-03T16:45:09.000Z</CreationDate>
    </Bucket>
  </Buckets>
</ListAllMyBucketsResult>

Tested on build 4.4(12906)

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Regardless that this needs to fixed what is the reason you want to use the emulation layer?

@cyberduck
Copy link
Collaborator Author

da06755 commented

I have swift cluster with swauth authentication and as described here https://trac.cyberduck.ch/wiki/help/en/howto/openstack before you can connect using cyberduck you must edit authentication context path. For some users it is strange and sometimes they complains about this, on the other side when using s3 connection they don't have to modify files by hand.

Now I see that I can create profile file like for keystone authentication so it could be workaround.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Replying to [comment:3 din]:

I have swift cluster with swauth authentication and as described here https://trac.cyberduck.ch/wiki/help/en/howto/openstack before you can connect using cyberduck you must edit authentication context path. For some users it is strange and sometimes they complains about this, on the other side when using s3 connection they don't have to modify files by hand.

Now I see that I can create profile file like for keystone authentication so it could be workaround.

The profiling feature is intended for such setups. Let me know if you need assistance in setting up a working profile.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Owner is an optional element in the ListBucketResult response (minOccurs=0) and we should check for null.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

In 1e3b7c4.

@cyberduck
Copy link
Collaborator Author

da06755 commented

Now I can connect and list containers but when trying get content of container there is a request:

GET /container/?versioning

which is not handle by s3 emulation in swift: Amazon_S3_REST_API_Compatability and cause 403 Forbidden response, but I think it is a problem of implementation s3 api in swift.

Thank you for all your help, I think that profiling feature is a best option for me now (created one based on examples).

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Replying to [comment:7 din]:

Now I can connect and list containers but when trying get content of container there is a request:

GET /container/?versioning

which is not handle by s3 emulation in swift: Amazon_S3_REST_API_Compatability and cause 403 Forbidden response, but I think it is a problem of implementation s3 api in swift.

Thank you for all your help, I think that profiling feature is a best option for me now (created one based on examples).

We fail gracefully when versioning is not supported by the S3 provider.

@cyberduck
Copy link
Collaborator Author

da06755 commented

From Log Drawer:

GET / HTTP/1.1
Date: Thu, 19 Sep 2013 08:30:03 GMT
Authorization: AWS TestUser1234:admin:unk6iwxHdOi/WxR7WnQFYlvoO68=
Host: example.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (12996) (Windows 7/6.1) (x86)
HTTP/1.1 200 OK
Content-Type: application/xml; charset=UTF-8
Content-Length: 702
X-Trans-Id: txff2b3d026951463d996c7bd0853b53c5
Date: Thu, 19 Sep 2013 08:30:09 GMT
Connection: keep-alive

GET /lista/?delimiter=%2F&max-keys=1000&prefix HTTP/1.1
Date: Thu, 19 Sep 2013 08:30:15 GMT
Authorization: AWS TestUser1234:admin:5njQ5dgJkhjQ6LjvCQLNTwl0agg=
Host: example.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (12996) (Windows 7/6.1) (x86)
HTTP/1.1 200 OK
Content-Type: application/xml; charset=UTF-8
Content-Length: 450
X-Trans-Id: tx73f9666f6b21474d87bee3af90617808
Date: Thu, 19 Sep 2013 08:30:21 GMT
Connection: keep-alive

GET /lista/?versioning HTTP/1.1
Date: Thu, 19 Sep 2013 08:30:15 GMT
Authorization: AWS TestUser1234:admin:O6IDf+5SCMBpmjAiYTlH7NNbrM4=
Host: example.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (12996) (Windows 7/6.1) (x86)
HTTP/1.1 400 Bad Request
Content-Type: text/xml; charset=UTF-8
Content-Length: 142
X-Trans-Id: tx8b3f028c2dd74a68b1ea9124097175a3
Date: Thu, 19 Sep 2013 08:30:21 GMT
Connection: keep-alive

and another session

GET / HTTP/1.1
Date: Thu, 19 Sep 2013 08:35:38 GMT
Authorization: AWS TestUser1234:admin:AWj9GtjljRq8FQcasqVeRPkg17I=
Host: example.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (12996) (Windows 7/6.1) (x86)
HTTP/1.1 200 OK
Content-Type: application/xml; charset=UTF-8
Content-Length: 702
X-Trans-Id: tx99b3d7be15ca48e5b6ed788463435311
Date: Thu, 19 Sep 2013 08:35:44 GMT
Connection: keep-alive

GET /clitest/?delimiter=%2F&max-keys=1000&prefix HTTP/1.1
Date: Thu, 19 Sep 2013 08:35:58 GMT
Authorization: AWS TestUser1234:admin:f8235qva1ntjKebP+uXxiLNO/jo=
Host: example.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (12996) (Windows 7/6.1) (x86)
HTTP/1.1 200 OK
Content-Type: application/xml; charset=UTF-8
Content-Length: 504
X-Trans-Id: tx2c69f36f851f45e89ddd7300776e8929
Date: Thu, 19 Sep 2013 08:36:04 GMT
Connection: keep-alive

HEAD /clitest/katalog HTTP/1.1
Date: Thu, 19 Sep 2013 08:35:58 GMT
Authorization: AWS TestUser1234:admin:q35JoqWEvey1hL8zmI8HxNT604A=
Host: example.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (12996) (Windows 7/6.1) (x86)
HTTP/1.1 200 OK
Last-Modified: Sun, 09 Jun 2013 13:32:33 GMT
Content-Length: 0
Etag: d41d8cd98f00b204e9800998ecf8427e
Content-Type: application/directory
X-Trans-Id: txc4d4f7fc5ef64649b20827d2e7a45cc9
Date: Thu, 19 Sep 2013 08:36:04 GMT
Connection: keep-alive

GET /clitest/?versioning HTTP/1.1
Date: Thu, 19 Sep 2013 08:35:58 GMT
Authorization: AWS TestUser1234:admin:HL06OFHlbAE6OOGkd4/PyBLIyTU=
Host: example.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (12996) (Windows 7/6.1) (x86)
HTTP/1.1 403 Forbidden
Content-Type: text/xml; charset=UTF-8
Content-Length: 124
X-Trans-Id: tx826f30b0a1984d04b2d6fb1ca60c01fd
Date: Thu, 19 Sep 2013 08:36:04 GMT
Connection: keep-alive

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Does this cause a connection failure with an error message?

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Replying to [comment:10 dkocher]:

Does this cause a connection failure with an error message?

I see we only handle permission failures silently.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Replying to [comment:11 dkocher]:

Replying to [comment:10 dkocher]:

Does this cause a connection failure with an error message?

I see we only handle permission failures silently.

In 5eba276.

@cyberduck
Copy link
Collaborator Author

da06755 commented

sorry for late response but problem seems still exist when try list container content I get error:

Cannot read container configuration

Log Drawer:

GET /write/?delimiter=%2F&max-keys=1000&prefix HTTP/1.1
Date: Mon, 07 Oct 2013 09:02:18 GMT
Authorization: AWS TestUser1234:admin:FrJ7iRMUV5sOsfJHym+GQdMLaIY=
Host: example.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (13235) (Windows 7/6.1) (x86)
HTTP/1.1 200 OK
Content-Type: application/xml; charset=UTF-8
Content-Length: 450
X-Trans-Id: tx26910a1a0e8c4c489501c954130bcfc8
Date: Mon, 07 Oct 2013 09:02:19 GMT
Connection: keep-alive

GET /write/?versioning HTTP/1.1
Date: Mon, 07 Oct 2013 09:02:19 GMT
Authorization: AWS TestUser1234:admin:2EtrCC+NeBMLgr4Mw6JUjfQkFYQ=
Host: example.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.4 (13235) (Windows 7/6.1) (x86)
HTTP/1.1 403 Forbidden
Content-Type: text/xml; charset=UTF-8
Content-Length: 124
X-Trans-Id: tx9c1682e206b14d7fb25d35da166b3adb
Date: Mon, 07 Oct 2013 09:02:19 GMT
Connection: keep-alive

with s3curl (which not use versioning) retrieving container content works.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

With the patch in 5eba276 we expect a 400 error response if the versioning feature is not supported.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Handling 403 respnoses was supposed to be fixed already (and errors silently ignored) in 88fbfd2.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

In b203bde.

@cyberduck
Copy link
Collaborator Author

da06755 commented

Thank you for fast fix. Find out that this problem was also fixed in newer version of s3 emulation middleware in swift: (fujita/swift3@8883a4e) which unfortunately is not in ubuntu 12.04 repositories.

In test environment (SAIO ) where is newer swift3 version get

HTTP/1.1 200 OK 
<VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/"/>

when request container with versioning query using s3curl.

@iterate-ch iterate-ch locked as resolved and limited conversation to collaborators Nov 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug fixed openstack OpenStack Swift Protocol Implementation
Projects
None yet
Development

No branches or pull requests

2 participants