Cyberduck Mountain Duck CLI

Opened 3 years ago

Closed 2 months ago

Last modified 8 weeks ago

#8880 closed feature (fixed)

Authentication using AWS AssumeRole and GetSessionToken with AWS STS

Reported by: tigris Owned by: dkocher
Priority: high Milestone: 6.7.0
Component: s3 Version: 4.7
Severity: normal Keywords: s3 iam sts mfa
Cc: Architecture: Intel
Platform: Mac OS X 10.10

Description (last modified by dkocher)

I am using amazon AssumeRole function to assume a role that can access an S3 bucket. This means to connect to S3, it needs more than just SecretKey and AccessKey, it also need SecurityToken or SessionToken which is an extremely large string.

If you set these 3 things in your environment, you can use tools like awscli etc from command line. I would like to use cyberduck instead as it can thread nicely. But unfortunately cyberduck only supports IAM users and not roles.

It does support roles from an EC2 instance, so I think it should be very easy to support from my own OSX laptop? I was thinking of just running a local proxy for 169.254.169.254 to fake the fact I am not running on EC2, but it seemed like overkill.

I notice a few people are suggesting entry of the security token - but isn't that short-lived? Don't see how that's a stable configuration solution. When configuring AWS CLI for this, I'd have an entry for the master account, and then one entry for each assumed role, such as:

[profile master]
region = us-east-1
output = json

[profile security]
role_arn = arn:aws:iam::999999999999:role/MyAccessRole
source_profile = master
region=eu-west-1
output=json

I think you need a way to collect and use this information, mainly the role_arn and reference to the source_profile.

Change History (54)

comment:1 Changed 3 years ago by dkocher

Referencing Support S3 authentication via IAM Role credentials #8610.

comment:2 Changed 3 years ago by dkocher

  • Component changed from core to s3
  • Owner set to dkocher
  • Summary changed from S3 via an amazon assumed role to Authentication using AWS AssumeRole and GetSessionToken

comment:3 follow-up: Changed 3 years ago by limitusus

  • Keywords sts added

I also would like this fixed. STS provides safer access privileges.

Does it take time to fix?

comment:4 in reply to: ↑ 3 Changed 3 years ago by dkocher

Replying to limitusus:

I also would like this fixed. STS provides safer access privileges.

AWS Security Token Service (STS)

comment:5 Changed 3 years ago by dkocher

Can you post the transcript from the log drawer (⌘-L) for the authentication failure that we get when trying to authenticate with the AccessKeyId and SecretAccessKey only with the token missing.

comment:6 Changed 3 years ago by dkocher

  • Milestone set to 4.8
  • Status changed from new to assigned

comment:7 Changed 3 years ago by limitusus

Sure. I set ACCESS_KEY_ID as user name and SECRET_ACCESS_KEY as password, without setting SESSION_TOKEN, and got the below transcript log. Please check it out.

GET / HTTP/1.1
Date: Thu, 10 Sep 2015 15:34:51 GMT
x-amz-request-payer: requester
Authorization: AWS ASIAJMHJ3CDT6TZFE5VQ:n6tV+L8WNQJPzE8U9bLJupZwEgg=
Host: s3.amazonaws.com:443
Connection: Keep-Alive
User-Agent: Cyberduck/4.7.2.18004 (Mac OS X/10.10.5) (x86_64)
HTTP/1.1 403 Forbidden
x-amz-request-id: 18855270603EBB20
x-amz-id-2: MLFktIVbM4nJRyzpHZhEdMTBODjIdcFGbPfzJXIPbavXlyzFfYDbW75Hx7fNk5wFMC/RU3PXHig=
Content-Type: application/xml
Transfer-Encoding: chunked
Date: Thu, 10 Sep 2015 15:34:51 GMT
Server: AmazonS3

comment:8 Changed 2 years ago by e-man-calle

I also need this!

comment:9 Changed 2 years ago by dkocher

  • Milestone changed from 5.0 to 5.1

comment:10 Changed 2 years ago by dkocher

  • Summary changed from Authentication using AWS AssumeRole and GetSessionToken to Authentication using AWS AssumeRole and GetSessionToken with AWS STS

comment:11 Changed 2 years ago by dkocher

If you are running Cyberduck within EC2, please refer to Connecting with temporary access credentials from EC2.

comment:12 Changed 2 years ago by dkocher

Can you clarify the use case and if we should use the AWS STS API to obtain the session token from https://sts.amazonaws.com.

comment:13 Changed 2 years ago by dkocher

AssumeRole

Returns a set of temporary security credentials (consisting of an access key ID, a secret access key, and a security token) that you can use to access AWS resources that you might not normally have access to. Typically, you use AssumeRole for cross-account access or federation.

comment:14 Changed 2 years ago by dkocher

Do you require multi-factor authentication (MFA) information for AssumeRole?

comment:15 Changed 2 years ago by dkocher

  • Milestone changed from 5.1 to 5.2

comment:16 Changed 2 years ago by dkocher

  • Milestone changed from 5.2 to 5.1.3

Milestone renamed

comment:17 Changed 2 years ago by dkocher

  • Milestone changed from 5.1.3 to 6.0

Ticket retargeted after milestone closed

comment:18 Changed 2 years ago by dkocher

  • Milestone 6.0 deleted

comment:19 Changed 15 months ago by dkocher

#9992 closed as duplicate.

comment:20 Changed 15 months ago by dkocher

#10013 closed as duplicate.

comment:21 Changed 15 months ago by dkocher

We possibly just need to add support to allow the input of a SessionToken in the authentication prompt to set the X-Amz-Security-Token header.

comment:22 Changed 10 months ago by dkocher

  • Milestone set to 6.3.1

Related ticket #10009.

comment:23 Changed 10 months ago by dkocher

  • Milestone 6.3.1 deleted

Ticket retargeted after milestone closed

comment:24 Changed 10 months ago by kuhnboy

Last edited 10 months ago by kuhnboy (previous) (diff)

comment:25 Changed 10 months ago by mjcsb

  • Description modified (diff)
  • Type changed from enhancement to feature

I also need this.

What would be AWESOME: instead of collecting the access_key_id and secret_key, you instead would collect the $AWS_DEFAULT_PROFILE or "--profile" as used in aws s3 CLI commands, so if we have configured awscli to use roles via lines in ~/.aws/config, this would simply work without having to double-enter the data in two locations.

Happy if you provide an option to both store inside Cyberduck AND if not stored internally, attempt to lookup the profile inside ~/.aws/* too.

But, AWS best practice for the last few years has been to use role assumption in any multi-account scenario, and they've been pushing multi-account at the enterprise level also for a few years, so I think you need to prioritize this - it seems to have been a request for years now.

comment:26 Changed 6 months ago by jibi-waba

What is the status of this feature request? It has been delayed and/or removed from the schedule multiple times over the past two years. As mentioned by someone else a few months ago, please make this a priority and add support for session tokens soon.

comment:28 Changed 6 months ago by jibi-waba

If Cyberduck supported the use of aws_session_token from the credentials file, then this would definitely be the route to take. However, using only the aws_access_key_id and aws_secret_access_key from that file does not allow authentication to the service. It needs a combination of all three values. You may want to add support for the legacy variable name - aws_security_token - which shares the same value as aws_session_token (at least in our environment). The token information is generated via the STS service when authenticating via SAML-based identity provider (whether that is Okta or ADFS or Auth0 or other provider).

Here's a truncated profile in my credentials file:

[aws-account-name]
aws_access_key_id = ASIA...
aws_secret_access_key = ++LQV7...
aws_session_token = FQoDY...
aws_security_token = FQoDY...
last_updated = 2018-03-26T18:14:35Z

More information: https://aws.amazon.com/blogs/security/a-new-and-standardized-way-to-manage-credentials-in-the-aws-sdks/

Last edited 6 months ago by jibi-waba (previous) (diff)

comment:29 Changed 6 months ago by mjcsb

aws_session_token and aws_security_token are, I think, the wrong way to fix this. That's not how AWS recommends you configure cross-account roles in AWS CLI.

The original ticket description remains the correct approach, IMHO. The IAM access code should look up a profile in ~/.aws/config - NOT - specify secret/access keys explicitly. This profile may contain either the secret/access keys needed, or it may contain a role_arn combined with a reference to a source_profile. It is the combination of the source_profile to get the secret/access key, with the role_arn to assume that role in another account, which is needed to access the S3 bucket in the other account.

I'm pretty sure all the code necessary to make this work is open source and visible in the AWS CLI GitHub project, someone just needs to refactor it to work here. Not sure why this is taking so long...

comment:30 Changed 6 months ago by jibi-waba

Both scenarios could/should be supported. Cyberduck should not define best practice or method. If the SDK or boto support a specific credentials configuration (or cross-account access configuration) for authentication, then clients like Cyberduck should also strive to support this authentication method(s). We have no issues with other clients, such as the AWSCLI or Terraform, supporting aws_session_token, so the expectation is that Cyberduck would also support it.

As you said, this should be a rather simple change since this is standard support within the SDK. Perhaps this needs to be split into two separate requests, but they’re closely related.

comment:31 Changed 6 months ago by dkocher

  • Milestone set to 7.0

comment:32 Changed 6 months ago by mcnicr

  • Keywords mfa added

I would also suggest that allowing for MFA is/should be closely tied to this feature. S3 security is at the forefront of many highlevel breaches in the news recently and permitting temporary credentials via sts:AssumeRole coupled with sts:GetSessionToken and allowing a client to use MFA significantly improves safety of data.

We have 100's of Cyberduck users who can no longer use the product due to this missing feature. I am very much looking forward to seeing this implemented so my user community can return to using CyberDuck

comment:33 Changed 5 months ago by ekent

  • Priority changed from low to high

Are there any updates on when this will be available? I've increased the priority to high as mentioned above, hundreds of Cyberduck users are no longer able to use the product due to this missing feature. This means that potential customers will also look elsewhere. Many companies are starting to use the AssumeRole function, and so the need for this is increasing by the day. Please add the feature in soon!

My suggestion is to pull all the credentials from the .aws/credentials file, upon every connect. The contents of the file will change, and so having the ability to pull the information every time will be useful and enable users not to have to re-enter the details (for if their credentials have a timeout/expiry of 60 mins).

comment:34 Changed 5 months ago by dkocher

Thanks everyone for the input provided!

As we are not accustomed ourselves to using session tokens, we would love to get some feedback if requirements are met when we implement this with the following constraints:

  • Read AWS credential profiles file from the default location (~/.aws/credentials). We already do this – note the AWS access key being set when you add a new bookmark and set the protocol to S3.
  • If the AWS Access Key set in the bookmark matches, read the session token.
  • We use the default profile "default" unless a custom profile name is set in the connection profile using the key Context. Refer to Profiles.
  • When Contextis set to an URL, the session token is retrieved from the EC2 instance metadata service (already supported).

comment:36 Changed 5 months ago by jibi-waba

@dkocher, I will be more than willing to test our use case when the above constraints are accounted for. For us, the big one is the "read the session token" portion. I believe what you have stated in the above criteria will address our scenario, which is passing the session token for authentication when a named profile is referenced. Thanks for your work and research on this ticket.

comment:37 Changed 5 months ago by mcnicr

A typical use case we have is switching roles between accounts that require MFA for the assume role to succeed. A sample of the type of config file most users are using is adding the mfa_serial to the config default profile and then referencing this in other profiles. This setup is using a single sign-on account '00000000000' for user management for passwords/access keys and MFA. Then the users will assume role into a different account to access S3.

When accessing S3 the UI should allow the user to input the MFA token to retrieve an sts:SessionToken which will carry the MFA characteristics along to be used to get sts:AssumeRole credentials.

User Credentials -> Session Credentials with MFA -> Assume Role into accounts with S3 data.

~user/.aws/config

[default]
region=us-east-1
output=json
mfa_serial=arn:aws:iam::000000000000:mfa/user@domain.com 
[profile assumerole]
role_arn=arn:aws:iam::11111111111111:role/role1account1
source_profile=default
[profile assumerole2]
role_arn=arn:aws:iam::22222222222222:role/role2account2
source_profile=default

comment:38 Changed 5 months ago by jibi-waba

@mcnicr MFA support with session tokens might be better served as a different ticket/feature request that is dependent on this one for session token support. Just my opinion.

comment:39 Changed 5 months ago by dkocher

  • Description modified (diff)

comment:40 follow-up: Changed 5 months ago by dt1001

I already have a script that gets me this far in ~/.aws/credentials:

[publish_profile]
output = json
region = us-west-1
aws_access_key_id = AAAAAAAAAAAAAAAAAAAA
aws_secret_access_key = KKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKKK
aws_session_token = SSSSSSSSSSS//////////SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS=

I want to configure Cyberduck with just profile = "publish_profile" and have it use those three values. It would also be nice to catch any expiration error so you could remind the user that their session has expired and they need to run through their external SSO tool again to refresh the aws_session_token.

comment:41 Changed 4 months ago by dkocher

Related issue #10222 asking for generic support of MFA.

comment:42 Changed 2 months ago by dkocher

Replying to tigris:

It does support roles from an EC2 instance, so I think it should be very easy to support from my own OSX laptop? I was thinking of just running a local proxy for 169.254.169.254 to fake the fact I am not running on EC2, but it seemed like overkill.

AWS Vault looks like an interesting project which supports exposing credentials running a local EC2 Instance Metadata server which should work together with the profile for Connecting with temporary access credentials (Token) from EC2.

comment:43 Changed 2 months ago by dkocher

  • Milestone changed from 7.0 to 6.7.0

comment:45 Changed 2 months ago by dkocher

We used the following steps to test our implementation.

  1. Create an IAM user testuser
  2. Create an IAM policy userpolicy with the following policy document:
          {
            "Version": "2012­10­17",
            "Statement": [
              {
                "Effect": "Allow",
                "Action": "sts:AssumeRole",
                "Resource": "arn:aws:iam::123456789012:role/testrole",
                "Condition": {
                  "Bool": {"aws:MultiFactorAuthPresent": true}
                }
    } ]
    }
    
  3. Attach the “userpolicy” policy to the “testuser” user.
  4. Create an IAM role testrole, specifying 123456789012 as the account and electing to

require MFA.

  1. Create an IAM policy rolepolicy with the following policy document:
          {
            "Version": "2012­10­17",
            "Statement": [{
              "Effect": "Allow",
              "Action": "s3:*",
              "Resource": "*"
    }] }
    
  2. Attach the rolepolicy policy to the testrole role.
  3. Generate an access key and secret for testuser
  4. Configure an MFA device for testuser

  1. Create a file ~/.aws/credentials with the following contents (substituting where indicated):
   [testuser]
   aws_access_key_id=<access key for testuser>
   aws_secret_access_key=<secret key for testuser>
   [testrole]
   role_arn=arn:aws:iam::123456789012:role/testrole
   source_profile=testuser
   mfa_serial=arn:aws:iam::123456789012:mfa/testuser
  1. Install the S3 (Credentials from AWS Security Token Service) profile and configure a bookmark using the testrole profile by entering testrole in ‘’Profile Name’’.

comment:46 Changed 2 months ago by dkocher

In r44703.

comment:47 Changed 2 months ago by dkocher

In r44721.

comment:48 Changed 2 months ago by dkocher

Skip saving temporary credentials in keychain in r44759.

comment:49 Changed 2 months ago by dkocher

  • Resolution set to fixed
  • Status changed from assigned to closed

comment:50 Changed 2 months ago by dkocher

Please update to the latest snapshot build available to test this integration.

comment:51 Changed 2 months ago by jibi-waba

I tested this for our method of accessing AWS via a SAML provider, where the security tokens are written directly into our credentials file via a tool we use to generate those tokens via STS (post-SAML auth). I have confirmed this method is now working to access S3 buckets. Thanks again!

comment:52 follow-up: Changed 2 months ago by jibi-waba

That said, it seems like the latest snapshot builds have broken our ability to use the traditional S3 profile with access/secret key auth. When prompted for secret key, clicking Login just beeps at us (multiple computers, multiple users, existing and new keys, keys work in other programs and CLIs, etc.).

comment:53 in reply to: ↑ 52 Changed 2 months ago by dkocher

Replying to jibi-waba:

That said, it seems like the latest snapshot builds have broken our ability to use the traditional S3 profile with access/secret key auth. When prompted for secret key, clicking Login just beeps at us (multiple computers, multiple users, existing and new keys, keys work in other programs and CLIs, etc.).

Fix in r44807.

comment:54 in reply to: ↑ 40 Changed 8 weeks ago by dt1001

Replying to dt1001:

I already have a script that gets me this far in ~/.aws/credentials ... I want to configure Cyberduck with just profile = "publish_profile" and have it use those three values ...

I just tested this with nightly build 28570 and it worked perfectly. Thank you so much! Really makes this enterprise friendly. Going to get my company to buy several licenses now.

Note: See TracTickets for help on using tickets.
swiss made software