Reference in ​Identity API v3.

​How do I migrate from v2.0 to v3.

In ​New implementations to authenticate with OpenStack Identity API v3.

You will need to create a custom ​connection profile with the authentication path /v3/tokens set in Context Path. Adapt from ​Openstack Swift (Keystone).cyberduckprofile.

In r17511.

@dkocher thanks for the fast reply. Is there a way to test with this code now? Or if not, when will v4.8 be available?

Please update to the latest ​snapshot build available.

Please confirm if this works as we haven't done any integration testing with a Keystone v3 deployment.

New profile in r17553.

Test Results on Cyberduck Version 4.8 (17513) .

Tried out Version 4.8 (17513) with 2 different Profiles for HTTP without success

Profile Keystone v3 HTTP(/v3/tokens)

<plist version="1.0">
<dict>
<key>Protocol</key>
<string>swift</string>
<key>Vendor</key>
<string>cyberduck</string>
<key>Context</key>
<string>/v3/tokens</string>
<key>Description</key>
<string>Openstack Swift (Keystone 3)</string>
<key>Username Placeholder</key>
<string>Project:Username</string>
<key>Password Placeholder</key>
<string>Password</string>
<key>Scheme</key>
<string>http</string>
</dict>
</plist>

Log output (/v3/tokens)

POST /v3/tokens HTTP/1.1
Accept: application/json
Content-Type: application/json
Content-Length: 197
Host: 9.18.76.136:35357
Connection: Keep-Alive
User-Agent: Cyberduck/4.8 (17513).17513 (Windows 7/6.1) (x86)
HTTP/1.1 404 Not Found
Vary: X-Auth-Token
Content-Type: application/json
Content-Length: 93
X-Openstack-Request-Id: req-063b7068-1efd-474b-8183-3ec7745a6843
Date: Thu, 21 May 2015 19:19:49 GMT
Connection: keep-alive

Error: File not found
Not found. 404 Not Found. Please contact your web hosting service provider for assistance.

Profile Keystone v3 HTTP(/v3/auth/tokens)

Note: We may need a Placeholder to support domain

<plist version="1.0">
<dict>
<key>Protocol</key>
<string>swift</string>
<key>Vendor</key>
<string>cyberduck</string>
<key>Context</key>
<string>v3/auth/tokens</string>
<key>Description</key>
<string>Openstack Swift (Keystone HTTP)</string>
<key>Username Placeholder</key>
<string>Tenant ID:Access Key</string>
<key>Password Placeholder</key>
<string>Secret Key</string>
<key>Scheme</key>
<string>http</string>
<key>Default Port</key>
<string>35357</string>
</dict>
</plist>

Log output (/v3/auth/tokens)

POST /v3/auth/tokens HTTP/1.1
Accept: application/json
Content-Type: application/json
Content-Length: 197
Host: 9.18.76.136:35357
Connection: Keep-Alive
User-Agent: Cyberduck/4.8 (17513).17513 (Windows 7/6.1) (x86)
HTTP/1.1 201 Created
X-Subject-Token: 83b0c04a331e4cc3abde981ae15c5c27
Vary: X-Auth-Token
Content-Type: application/json
Content-Length: 1691
X-Openstack-Request-Id: req-c378121d-a748-4f99-b4d6-53c169f05b29
Date: Thu, 21 May 2015 19:21:19 GMT
Connection: keep-alive

Error: Connection failed
Created. 201 Created.

Fixed context path in r17603.

Fix expecting 201 response code in r17604.

TestResult with Cyberduck/4.8 (17608)

@dkocher we are still not able to connect to keystone v3 using cyberduck (17608). Any suggestions on how to debug further?

Replying to bill_az:

@dkocher we are still not able to connect to keystone v3 using cyberduck (17608). Any suggestions on how to debug further?

Please again post the log output in the Transcript.

@dkocher I am receiving a "201 Created" Response with Connection Failed (please see Attachment 17608.jpg)

Log output:

POST /v3/auth/tokens HTTP/1.1
Accept: application/json
Content-Type: application/json
Content-Length: 196
Host: 9.18.76.126:35357
Connection: Keep-Alive
User-Agent: '''Cyberduck/4.8.17696''' (Windows 7/6.1) (x86)
HTTP/1.1 201 Created
X-Subject-Token: b6634117e4f341babcff31ac406a1623
Vary: X-Auth-Token
Content-Type: application/json
Content-Length: 1691
X-Openstack-Request-Id: req-550bfd38-d585-4ac6-8d9e-1ac075a9de67
Date: Wed, 03 Jun 2015 15:18:56 GMT
Connection: keep-alive

Fix upstream in ​3db505511c1bc100baeb2ee1d862fa4d89c860e1.

In r17750.

I would like to try the fix, but I am not able to find the build for this change. The latest nightly build for windows is Cyberduck-Installer-4.8.17726.exe, and by trying Preference ->Update ->Snapshot Builds it says "You're up to date!" with User-Agent: Cyberduck/4.8.17726 (Windows 7/6.1) (x86).

Replying to ariday:

I would like to try the fix, but I am not able to find the build for this change. The latest nightly build for windows is Cyberduck-Installer-4.8.17726.exe, and by trying Preference ->Update ->Snapshot Builds it says "You're up to date!" with User-Agent: Cyberduck/4.8.17726 (Windows 7/6.1) (x86).

A new build is now available. Thanks for testing!

Replying to dkocher:

Replying to ariday:

I would like to try the fix, but I am not able to find the build for this change. The latest nightly build for windows is Cyberduck-Installer-4.8.17726.exe, and by trying Preference ->Update ->Snapshot Builds it says "You're up to date!" with User-Agent: Cyberduck/4.8.17726 (Windows 7/6.1) (x86).

A new build is now available. Thanks for testing!

Tested Build 4.7.1.17798. Now I do not see any Error Message, but the connection is not established.

Log output:

POST /v3/auth/tokens HTTP/1.1
Accept: application/json
Content-Type: application/json
Content-Length: 194
Host: 9.18.76.17:35357
Connection: Keep-Alive
User-Agent: Cyberduck/4.7.1.17798 (Windows 7/6.1) (x86)
HTTP/1.1 201 Created
X-Subject-Token: 0abf709c013449ca963c38bfb64a5d73
Vary: X-Auth-Token
Content-Type: application/json
Content-Length: 1606
X-Openstack-Request-Id: req-1d7e869b-9c90-4900-8190-76a1cd256ffe
Date: Tue, 23 Jun 2015 16:24:52 GMT
Connection: keep-alive

Log file for Swift Keystonev3 Connection not established 4.7.1.17798

1428	Caused by: java.lang.UnsupportedOperationException: JsonNull
1429	        at com.google.gson.JsonElement.getAsString(JsonElement.java:191)
1430	        at ch.iterate.openstack.swift.handler.AuthenticationJson3ResponseHandler.handleResponse(AuthenticationJson3ResponseHandler.java:81)
1431	        at ch.iterate.openstack.swift.handler.AuthenticationJson3ResponseHandler.handleResponse(AuthenticationJson3ResponseHandler.java:29)
1432	        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:222)
1433	        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:164)
1434	        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:139)
1435	        at ch.iterate.openstack.swift.Client.authenticate(Client.java:211)
1436	        at ch.iterate.openstack.swift.Client.authenticate(Client.java:204)
1437	        at ch.cyberduck.core.openstack.SwiftSession.login(SwiftSession.java:132)
1438	        at ch.cyberduck.core.KeychainLoginService.authenticate(KeychainLoginService.java:71)
1439	        at ch.cyberduck.core.LoginConnectionService.authenticate(LoginConnectionService.java:201)
1440	        at ch.cyberduck.core.LoginConnectionService.authenticate(LoginConnectionService.java:213)
1441	        at ch.cyberduck.core.LoginConnectionService.connect(LoginConnectionService.java:191)
1442	        at ch.cyberduck.core.LoginConnectionService.check(LoginConnectionService.java:129)
1443	        at ch.cyberduck.core.LoginConnectionService.check(LoginConnectionService.java:136)
1444	        at ch.cyberduck.core.threading.SessionBackgroundAction.connect(SessionBackgroundAction.java:220)
1445	        at ch.cyberduck.core.threading.BrowserBackgroundAction.connect(BrowserBackgroundAction.java:108)
1446	        at ch.cyberduck.core.threading.SessionBackgroundAction.call(SessionBackgroundAction.java:186)
1447	        at ch.cyberduck.core.AbstractController$BackgroundCallable.call(AbstractController.java:174)
1448	        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
1449	        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
1450	        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:618)
1451	        at ch.cyberduck.core.threading.NamedThreadFactory$1.run(NamedThreadFactory.java:58)
1452	        at java.lang.Thread.run(Thread.java:961)

There seems to be a confusion with naming the region in the response XML. We have taken an example from ​https://github.com/openstack/python-keystoneclient/blob/master/examples/pki/cms/auth_v3_token_scoped.json for a testcase where the field is named regionwhereas in the documentation at ​http://developer.openstack.org/api-ref-identity-v3.html the field is named region_id. The downside of all these JSON blobs with no validation that we would have for free with XML Schema.

We can find no example at ​http://docs.openstack.org/developer/keystone/api_curl_examples.html with a object-store type in the service catalog result set.

Revert support for v3 authentication and postpone to 4.8 in r17846.

Can you provide a temporary integration testing environment?

@dkocher - can we schedule a joint debug session? We can share screen/conference call to debug. If this works, let me know a day / time that would be convenient.

@dkocher hi, still want to see if we can set up a meeting with shared screen to debug. Would that work? What time zone are you in?

Replying to bill_az:

@dkocher hi, still want to see if we can set up a meeting with shared screen to debug. Would that work? What time zone are you in?

We are in UTC+1.

Please try build r17986 or later.

Is it possible for you to provide the HTTP response body (JSON formatted) the server sends possibly using the Swift command line tools?

Log file for Swift Keystonev3 Connection not established 4.7.2 despite HTTP/1.1 200 OK

DEBUG org.apache.http.impl.conn.DefaultHttpClientConnectionOperator - Connecting to /<IPADDR> [[BR]]
DEBUG org.apache.http.impl.conn.DefaultHttpClientConnectionOperator - Connection established <IPADDR>:63814<-><IPADDR>:35357  [[BR]]
DEBUG org.apache.http.impl.execchain.MainClientExec - Executing request POST /v2.0/tokens HTTP/1.1  [[BR]]
DEBUG org.apache.http.impl.execchain.MainClientExec - Target auth state: UNCHALLENGED  [[BR]]
DEBUG org.apache.http.impl.execchain.MainClientExec - Proxy auth state: UNCHALLENGED  [[BR]]
INFO  ch.cyberduck.core.Session - POST /v2.0/tokens HTTP/1.1  [[BR]]
INFO  ch.cyberduck.core.Session - Accept: application/json  [[BR]]
INFO  ch.cyberduck.core.Session - Content-Type: application/json  [[BR]]
INFO  ch.cyberduck.core.Session - Content-Length: 96  [[BR]]
INFO  ch.cyberduck.core.Session - Host: <IPADDR>0:35357 [[BR]]
INFO  ch.cyberduck.core.Session - Connection: Keep-Alive [[BR]]
INFO  ch.cyberduck.core.Session - User-Agent: Cyberduck/4.7.2.18004 (Windows 7/6.1) (x86) [[BR]]
DEBUG org.apache.http.headers - http-outgoing-0 >> POST /v2.0/tokens HTTP/1.1 [[BR]]
DEBUG org.apache.http.headers - http-outgoing-0 >> Accept: application/json [[BR]]
DEBUG org.apache.http.headers - http-outgoing-0 >> Content-Type: application/json [[BR]]
DEBUG org.apache.http.headers - http-outgoing-0 >> Content-Length: 96 [[BR]]
DEBUG org.apache.http.headers - http-outgoing-0 >> Host: <IPADDR>:35357 [[BR]]
DEBUG org.apache.http.headers - http-outgoing-0 >> Connection: Keep-Alive [[BR]]
DEBUG org.apache.http.headers - http-outgoing-0 >> User-Agent: Cyberduck/4.7.2.18004 (Windows 7/6.1) (x86) [[BR]]
DEBUG org.apache.http.headers - http-outgoing-0 << HTTP/1.1 200 OK [[BR]]
DEBUG org.apache.http.headers - http-outgoing-0 << Vary: X-Auth-Token [[BR]]
DEBUG org.apache.http.headers - http-outgoing-0 << Content-Type: application/json [[BR]]
DEBUG org.apache.http.headers - http-outgoing-0 << Content-Length: 1284 [[BR]]
DEBUG org.apache.http.headers - http-outgoing-0 << X-Openstack-Request-Id: req-dab2ae62-b226-4c52-bfca-73d3995f42a1 [[BR]]
DEBUG org.apache.http.headers - http-outgoing-0 << Date: Mon, 10 Aug 2015 19:36:08 GMT [[BR]]
DEBUG org.apache.http.headers - http-outgoing-0 << Connection: keep-alive [[BR]]
INFO  ch.cyberduck.core.Session - HTTP/1.1 200 OK [[BR]]
INFO  ch.cyberduck.core.Session - Vary: X-Auth-Token [[BR]]
INFO  ch.cyberduck.core.Session - Content-Type: application/json [[BR]]
INFO  ch.cyberduck.core.Session - Content-Length: 1284 [[BR]]
INFO  ch.cyberduck.core.Session - X-Openstack-Request-Id: req-dab2ae62-b226-4c52-bfca-73d3995f42a1 [[BR]]
INFO  ch.cyberduck.core.Session - Date: Mon, 10 Aug 2015 19:36:08 GMT [[BR]]
INFO  ch.cyberduck.core.Session - Connection: keep-alive [[BR]]
DEBUG org.apache.http.impl.execchain.MainClientExec - Connection can be kept alive indefinitely [[BR]]
DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection [id: 0][route: {}->http://<IPADDR>:35357] can be kept alive indefinitely [[BR]]
DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager - Connection released: [id: 0][route: {}->http://<IPADDR>:35357][total kept alive: 1; route allocated: 1 of 10; total allocated: 1 of 2147483647] [[BR]]
ERROR ch.cyberduck.core.AbstractController - Unhandled exception running background task JsonNull [[BR]]

Replying to dkocher:

Revert support for v3 authentication and postpone to 4.8 in r17846.

Added again in r18010.

Connection successful with V3 but when trying to create containers a 401 response is shown
(region should be specified in profile to connect successfully)
401 might be a domain issue

POST /v3/auth/tokens HTTP/1.1\\
Accept: application/json\\
Content-Type: application/json\\
Content-Length: 194\\
Host: <ipaddress>:35357\\
Connection: Keep-Alive\\
User-Agent: Cyberduck/4.8.18022 (Windows 7/6.1) (x86)\\
HTTP/1.1 201 Created\\
X-Subject-Token: 685e0d061b764c33a97e7bed342b77dc\\
Vary: X-Auth-Token\\
Content-Type: application/json\\
Content-Length: 1666\\
X-Openstack-Request-Id: req-71ce95ec-6faa-4dfd-8f4b-3a5e7fc50c5b\\
Date: Thu, 20 Aug 2015 21:18:34 GMT\\
Connection: keep-alive\\
HEAD /v1/AUTH_02c0745ea4e044d1bd672f5be8b327f9 HTTP/1.1\\
X-Auth-Token: ee447665eaaf41cc9a88ee91330ef6d9\\
Host: <ipaddress>:8080\\
Connection: Keep-Alive\\
User-Agent: Cyberduck/4.8.18022 (Windows 7/6.1) (x86)\\
HTTP/1.1 401 Unauthorized\\
Content-Type: text/html; charset=UTF-8\\
Www-Authenticate: Swift realm="AUTH_02c0745ea4e044d1bd672f5be8b327f9"\\
WWW-Authenticate: Keystone uri='http://127.0.0.1:35357'\\
X-Trans-Id: tx74d8aae64d014a54a22d5-0055d6442a\\
Content-Length: 0\\
Date: Thu, 20 Aug 2015 21:18:34 GMT\\
Connection: keep-alive\\
POST /v3/auth/tokens HTTP/1.1\\
Accept: application/json\\
Content-Type: application/json\\
Content-Length: 194\\
Host: 9.18.76.100:35357\\
Connection: Keep-Alive\\
User-Agent: Cyberduck/4.8.18022 (Windows 7/6.1) (x86)\\
HTTP/1.1 201 Created\\
X-Subject-Token: 18e4a4d879d2489d822059b12f28e63d\\
Vary: X-Auth-Token\\
Content-Type: application/json\\
Content-Length: 1666\\
X-Openstack-Request-Id: req-b874ec71-2070-490b-a061-78de597b701f\\
Date: Thu, 20 Aug 2015 21:18:34 GMT\\
Connection: keep-alive\\
HEAD /v1/AUTH_02c0745ea4e044d1bd672f5be8b327f9 HTTP/1.1\\
X-Auth-Token: ee447665eaaf41cc9a88ee91330ef6d9\\
Host: <ipaddress>:8080\\
Connection: Keep-Alive\\
User-Agent: Cyberduck/4.8.18022 (Windows 7/6.1) (x86)\\
HTTP/1.1 401 Unauthorized\\
Content-Type: text/html; charset=UTF-8\\
Www-Authenticate: Swift realm="AUTH_02c0745ea4e044d1bd672f5be8b327f9"\\
WWW-Authenticate: Keystone uri='http://127.0.0.1:35357'\\
X-Trans-Id: tx6698aeea5bf8497fb42a4-0055d6442a\\
Content-Length: 0\\
Date: Thu, 20 Aug 2015 21:18:34 GMT\\
Connection: keep-alive\\

Profile:

<plist version="1.0">\\
<dict>\\
<key>Protocol</key>\\
<string>swift</string>\\
<key>Vendor</key>\\
<string>cyberduck</string>\\
<key>Context</key>\\
<string>/v3/auth/tokens</string>\\
<key>Region</key>\\
<string>regionOne</string>\\
<key>Scheme</key>\\
<string>http</string>\\
<key>Description</key>\\
<string>Openstack Swift 3(HTTP)</string>\\
</dict>\\
</plist>\\

Replying to danizar:

Connection successful with V3 but when trying to create containers a 401 response is shown
(region should be specified in profile to connect successfully)
401 might be a domain issue

Do other operations such as listing containers and keys succeed?

Replying to dkocher:

Replying to danizar:

Connection successful with V3 but when trying to create containers a 401 response is shown
(region should be specified in profile to connect successfully)
401 might be a domain issue

Do other operations such as listing containers and keys succeed?

No other operation succeed, it is only possible to establish the connection.

Hi together,

Not sure about the current state of this. But besides the region, the profile should also contain an option to specify the used domain (ideally separating user domain and project domain).
The domain should be forwarded to ch.cyberduck.core.openstack.SwiftAuthenticationService were ch.iterate.openstack.swift.method.Authentication3UsernamePasswordProjectRequest is used without specifying the domain. So Authentication3UsernamePasswordProjectRequest is already able to handle domains I would assume.
This leads to the fact, that only projects in default domain are accessible. Having customers isolated by domain in keystone v3, is one benefit someone would use v3. Would you agree?

Many Thanks in advance,
Falk

I would be more than happy to provide you with a test account for a public cloud with Openstack Swift and Identity v3 with domains and projects. Who should I send it to?

Replying to stenstad:

I would be more than happy to provide you with a test account for a public cloud with Openstack Swift and Identity v3 with domains and projects. Who should I send it to?

Please write to ​feedback@cyberduck.io.

We have been sending requests with a domain identified by id instead of name.

{
    "auth": {
        "identity": {
            "methods": [
                "password"
            ],
            "password": {
                "user": {
                    "domain": {
                        "id": "cyberduck.io"
                    },
                    "name": "cyberduck",
                    "password": "-----------------"
                }
            }
        },
        "scope": {
            "project": {
                "domain": {
                    "id": "cyberduck.io"
                },
                "name": "Production"
            }
        }
    }
}

should instead be

{
    "auth": {
        "identity": {
            "methods": [
                "password"
            ],
            "password": {
                "user": {
                    "domain": {
                        "name": "cyberduck.io"
                    },
                    "name": "cyberduck",
                    "password": "-----------------"
                }
            }
        },
        "scope": {
            "project": {
                "domain": {
                    "name": "cyberduck.io"
                },
                "name": "Production"
            }
        }
    }
}

Reference ​upstream changeset.

A sample ​profile for zetta.io.

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
    <dict>
        <key>Protocol</key>
        <string>swift</string>
        <key>Vendor</key>
        <string>zetta.io</string>
        <key>Default Hostname</key>
       	<string>identity.api.zetta.io</string>
        <key>Context</key>
        <string>/v3/auth/tokens</string>
        <key>Description</key>
        <string>Zetta.IO (OpenStack Swift Keystone v3)</string>
        <key>Username Placeholder</key>
        <string>Project Name:Project Domain:Username</string>
        <key>Password Placeholder</key>
        <string>Password</string>
    </dict>
</plist>

Fix in ​da4aa75f81b.

Hi. Great news. Would it be possible to have a nightly build with this fix included?

Replying to Falk Reimann:

Hi. Great news. Would it be possible to have a nightly build with this fix included?

New snapshot builds will become available this week.

Version 4.8.0.18437 is now available as a snapshot build.

Hi. I did a quick check with Cyberduck connecting to a OpenStack devstack with keystone v3 enabled. It is still not working for me. I think the main issue is, that Cyberduck uses the UserID as Auth Token instead of the X-Subject-Token after issuing an token from keystone and therefore Swift will not authorize the request. I attached the main information from the log drawer. Can you please revisit this?
Thanks in advance, Falk

Token issued from keystone:

X-Subject-Token: 4c29d27f7c2e46b6bc82708d6060311b

Token provided to Swift:

X-Auth-Token: e239b9d1a16b4d6ea37770beabe06fea

User Id:

id: e239b9d1a16b4d6ea37770beabe06fea

Log Drawer output (truncated):

POST /v3/auth/tokens HTTP/1.1
Host: devstack:5000
User-Agent: Cyberduck/4.8.0.18437 (Mac OS X/10.11.2) (x86_64)
HTTP/1.1 201 Created
X-Subject-Token: 4c29d27f7c2e46b6bc82708d6060311b
Vary: X-Auth-Token
x-openstack-request-id: req-4cc53034-b79f-4545-ac6f-302c4d160d45

HEAD /v1/AUTH_0a9834e6208948fabd35f1497d71bcd6 HTTP/1.1
X-Auth-Token: e239b9d1a16b4d6ea37770beabe06fea
Host: devstack:8090
User-Agent: Cyberduck/4.8.0.18437 (Mac OS X/10.11.2) (x86_64)
HTTP/1.1 401 Unauthorized

And the user details I used for connection:

$ openstack user show swift
+-----------+----------------------------------+
| Field     | Value                            |
+-----------+----------------------------------+
| domain_id | default                          |
| email     | None                             |
| enabled   | True                             |
| id        | e239b9d1a16b4d6ea37770beabe06fea |
| name      | swift                            |
+-----------+----------------------------------+

The documentation ​documentation has to say

In v3, your token is returned to you in an X-Subject-Token header, instead of as part of the request body. You should still authenticate yourself to other services using the X-Auth-Token header.

Fix in ​upstream.

In r18867.

Replying to dkocher:

In r18867.

Many thanks for the quick response.
Just tested version 4.8.0 (18464) and keystone v3 does now work with the issued token against swift.
Great, Thanks, Falk