Temporary tokens (credentials from AWS STS) do not work with AWS GovCloud S3

[cduser]

The S3(Credentials from AWS Security Token Service) profile (​​https://svn.cyberduck.io/trunk/profiles/S3%20(Credentials%20from%20AWS%20Security%20Token%20Service).cyberduckprofile) does not work with AWS GovCloud accounts. Cyberduck gets into a loop where it says "Authenticating as publish_profile" followed by "Login failed". I also tried using the AWS GovCloud profile (​https://svn.cyberduck.io/trunk/profiles/S3%20Gov%20Cloud.cyberduckprofile), but it doesn't support temporary tokens. I also tried creating my own profile merging 'S3(Credentials from AWS Security Token Service) and AWS GovCloud but that didn't work either. Here is the custom profile I tried out:

<?xml version="1.0" encoding="UTF-8"?>
<!--
  ~ Copyright (c) 2002-2018 iterate GmbH. All rights reserved.
  ~ https://cyberduck.io/
  ~
  ~ This program is free software; you can redistribute it and/or modify
  ~ it under the terms of the GNU General Public License as published by
  ~ the Free Software Foundation, either version 3 of the License, or
  ~ (at your option) any later version.
  ~
  ~ This program is distributed in the hope that it will be useful,
  ~ but WITHOUT ANY WARRANTY; without even the implied warranty of
  ~ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
  ~ GNU General Public License for more details.
  -->

<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
    <dict>
        <key>Protocol</key>
        <string>s3</string>
        <key>Vendor</key>
        <string>s3-token</string>
        <key>Scheme</key>
        <string>https</string>
        <key>Description</key>
        <string>AWS GovCloud S3</string>
        <key>Default Port</key>
        <string>443</string>
        <key>Default Nickname</key>
        <string>AWS GovCloud S3</string>
        <key>Default Hostname</key>
        <string>s3-us-gov-west-1.amazonaws.com</string>
        <key>Username Placeholder</key>
        <string>Profile Name</string>
        <key>Password Configurable</key>
        <false/>
        <key>Token Configurable</key>
        <false/>
        <key>Anonymous Configurable</key>
        <false/>
        <key>Region</key>
        <string>us-gov-west-1</string>
    </dict>
</plist>

This is the AWS credentials file I'm using:

[publish_profile]
output = json
region = us-gov-west-1
aws_access_key_id = AAAAAAAAAAAAAAAAAAAA
aws_secret_access_key = KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKK
aws_session_token = SSSSSSSSSSS//////////SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS

Is there a way to support both AWS GovCloud and S3 (Credentials from AWS Security Token Service) at the same time?

Thanks!

[dkocher]

The auto configuration from ~/aws/credentials is currently only triggered when the hostname defaults to s3.amazonaws.com.

[cduser]

Thanks for checking! I also want to mention that there are now 2 GovCloud regions with different endpoints:

s3-us-gov-west-1.amazonaws.com
s3.us-gov-east-1.amazonaws.com

[yla]

Fixed in r46275.

[cduser]

Replying to yla:

Fixed in r46275.

Thanks a lot! Two questions:

1. When will the update make it to a snapshot release?
2. Is there a way to configure a default value for the Profile Name in ~/.aws/credentials in the .cyberduckprofile file?

Thanks!

[dkocher]

Replying to cduser:

Thanks for checking! I also want to mention that there are now 2 GovCloud regions with different endpoints:

s3-us-gov-west-1.amazonaws.com
s3.us-gov-east-1.amazonaws.com

Added profile in r46283.

[dkocher]

A new snapshot build has now been published. Please comment on this ticket if the issue is resolved (or reopen) as we cannot fully test this ourselves.

[cduser]

Replying to dkocher:

A new snapshot build has now been published. Please comment on this ticket if the issue is resolved (or reopen) as we cannot fully test this ourselves.

Hi dkocher,

I can help with the testing. Unfortunately I'm still having issues, but its looking better.

This is what I'm doing:

I'm using the following profile:

<?xml version="1.0" encoding="UTF-8"?>
<!--
  ~ Copyright (c) 2002-2018 iterate GmbH. All rights reserved.
  ~ https://cyberduck.io/
  ~
  ~ This program is free software; you can redistribute it and/or modify
  ~ it under the terms of the GNU General Public License as published by
  ~ the Free Software Foundation, either version 3 of the License, or
  ~ (at your option) any later version.
  ~
  ~ This program is distributed in the hope that it will be useful,
  ~ but WITHOUT ANY WARRANTY; without even the implied warranty of
  ~ MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
  ~ GNU General Public License for more details.
  -->

<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
    <dict>
        <key>Protocol</key>
        <string>s3</string>
        <key>Vendor</key>
        <string>s3-token</string>
        <key>Description</key>
        <string>S3 (Credentials from AWS Security Token Service)</string>
        <key>Default Nickname</key>
        <string>S3 (Credentials from AWS Security Token Service)</string>
        <key>Username Placeholder</key>
        <string>Profile Name in ~/.aws/credentials</string>
        <key>Password Configurable</key>
        <false/>
        <key>Token Configurable</key>
        <false/>
        <key>Anonymous Configurable</key>
        <false/>
        <key>Region</key>
        <string>us-gov-west-1</string>
    </dict>
</plist>

When adding the profile to cyberduck I setServer to s3-us-gov-west-1.amazonaws.com and Profile Name in ~/.aws/credentials to cyberduck. I then get new temporary credentials from AWS and put them in my ~/.aws/credentials file like this:

[cyberduck]
output = json
region = us-gov-west-1
aws_access_key_id = AAAAAAAAAAAAAAAAAAAA
aws_secret_access_key = KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKK
aws_session_token = SSSSSSSSSSS//////////SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS

When I double click on the the shortcut in cyberduck I see Authenticating as cyberduck in the lower left corner with a spinning icon, but it never connects. If I try to close cyberduck it gets locked up and I have to force quit. Is there a way to enable debug logs?

Thank you for all your help and prompt response!

[dkocher]

The error I can reproduce here is

<?xml version="1.0" encoding="UTF-8"?><Error><Code>AuthorizationHeaderMalformed</Code><Message>The authorization header is malformed; the region 'us-gov-west-1' is wrong; expecting 'us-east-1'</Message><Region>us-east-1</Region><RequestId>6DC92B83F187052E</RequestId><HostId>/i/KpB+I0jY/luCGm6wHoW5YJjdxMYTknIMe9rYtCMInebV+rJBtiI8b9sK7NXfOzS+n0wuMUTQ=</HostId></Error>

[dkocher]

Fix regression in r46287.

[cduser]

Replying to dkocher:

Fix regression in r46287.

Great! The AWS GovCloud S3 login issues are fixed. I can now log in (using the configuration in the credentials file) and see the directory listing. Unfortunately I run into an error when trying to download files or directories. Since I get the same error using the normal access keys, I created a new ticket #10596.

Thanks!