Cyberduck Mountain Duck CLI

#8610 closed enhancement (fixed)

Support S3 authentication via IAM Role credentials

Reported by: ebekker Owned by: dkocher
Priority: normal Milestone: 4.7
Component: s3 Version:
Severity: normal Keywords:
Cc: Architecture:


Would it be possible to add support for authentication to AWS S3 using Access Key credentials that are derived from an IAM Role on an EC2 instance?

IAM Roles allow you to assign a set of permissions to a resource that is actually deployed in the AWS environment. The way this is implemented is that a set of credentials (Access Key + Secret Key + Session Token) are dynamically assigned and rotated to a particular AWS resources, such as an EC2 Instance.

The Access Key credentials can be retrieved on an EC2 instance by accessing its own instance meta data via the URL:

The last component of that URL path (THIS_IS_THE_ROLE_NAME) is actually the name of the role assigned, but it is the only entry in that path, so to get to it programmatically, you would need to call its parent URL and find the only returned value in the response, and then retrieve the target URL which contains the actual credentials.

The actual credentials returned look like the following JSON fragment:

  "Code" : "Success",
  "LastUpdated" : "2015-02-24T21:10:36Z",
  "Type" : "AWS-HMAC",
  "SecretAccessKey" : "wli/Bu889nQjdRxpgF6QR3Hoqjz8Lou7pnoxBU/r",
  "Token" : "AQoDYXdzEN7//////////wEa0AMBFkIFLfF7oMkGW9MnnPwNuRNikTvwPiHCk+DOrQgIZ9/vas0tUoe35TPBbnMB6keqO0IZaFvwICaA4k83X+clJ3aRC+E26K6oC8H3LDTfEFWofnoxuPGCq8MKPdMJOz2URnwn0Mv5p4ecuxXmNJqD2pdh65xH7jtA4slK3ZyD6ggXvSSMzlq9VeVSTz39V57FZNRQ6u4JcjWv3gBqfrJyTY10uLP8N4xMO0M7uEU9hnXlwxbMKkm3arjjl81IVYTh4OIaju3NcCsBMnWqxetsZtCWG4+SZbT6RWKOZMD7r8joIsjomRIkuJDjSpVL/f3Xa1iphF+qOFVsYUav2XNQukvHcborOWy2CIDBnOsMrA67z8BVByZG6qLQBywsxTzYv/w+nEWkY1avVWAhcWeMxDrx3UYVtHLnObZUdhnEbeXrTgwbjXqYRFi4Pe8cnMNRQvcmh08MPGEiXLtqc7G0zIr8yWdl0Im5CK4OBSkPAzujkcnu6hjMAaVkuXu+OPU9Q/qV9mLx6C8+VhZ/udPY2537qCGL3H9/2LWaLO9uCrRYpx+f1es+idg12P+bO68aN0G6mzKDbGZsDjJqONt5622CYDZBWJumAlFQYmTcmSCX0bOnBQ==",
  "Expiration" : "2015-02-25T03:16:33Z"

From this response you can get the following 3 components which are needed to authenticate requests to the AWS S3 API:

  • AccessKeyId - the Access Key
  • SecretAccessKey - the Secret Key
  • Token - the Session Token

You can also obtain the LastUpdated and Expiration components which indicate when the credentials were last generated, as well as when they will expire and be rotated once again.

For CyberDuck, the request I'm making is to add an option flag to the S3 connection settings to indicate the use of IAM Role credentials (similar to the existing "Anonymous" flag). When set, CyberDuck would obtain the current Role credentials as described above, and store them in the current user session, and with every API call to S3, verify the creds are still valid (by comparing current time against the expected Expiration time), and use those creds to authenticate/authorize each API request.

This will allow CyberDuck to be used on an EC2 instance within AWS assuming the IAM Role access policies.

Change History (13)

comment:1 Changed on Feb 24, 2015 at 10:59:02 PM by ebekker

After taking another look at the "Open Connection" and "Add Bookmark" dialog boxes, I see that the actual structure of these dialogs is pretty much fixed across all the different providers, and the only thing that varies, is the labeling of a few fields (cred-related) and also the options that are available in the "Connect Mode" drop-down.

To keep consistent with this approach, this feature could be implemented either (1) as a completely new provider type (S3 IAM Role) or (2) as a drop-down selection for the "Connect Mode" option under the S3 provider.

comment:2 Changed on Feb 25, 2015 at 9:50:12 AM by dkocher

  • Summary changed from Support S3 authentication via IAM ROLE credentials to Support S3 authentication via IAM Role credentials

comment:3 follow-up: Changed on Feb 25, 2015 at 9:51:06 AM by dkocher

Do you run Cyberduck on a Windows EC2 instance?

comment:4 Changed on Feb 25, 2015 at 9:51:40 AM by dkocher

This feature would also be very suitable when running Cyberduck CLI on a Linux instance.

comment:5 Changed on Feb 25, 2015 at 9:52:19 AM by dkocher

We would set this up with a Profile required to be installed.

comment:6 in reply to: ↑ 3 Changed on Feb 25, 2015 at 11:15:19 AM by ebekker

Replying to dkocher:

Do you run Cyberduck on a Windows EC2 instance?

Yes, we're rolling out a few intances in AWS right now and each one is assigned an IAM Role specific to their function or service class. Those roles determine permissions the instances inherit to support their functions wrt/ calls and access to different AWS services and resources, including S3 where the different instances have access to different buckets and actions.

comment:7 Changed on Feb 25, 2015 at 11:19:17 AM by dkocher

  • Milestone set to 4.7
  • Status changed from new to assigned

comment:8 Changed on Feb 25, 2015 at 11:26:57 AM by ebekker

I was reviewing some of the relevant code to implement this request, and I thought of another way to add the functionality with minimal disruption to the existing UI or codebase as starting point.

Instead of touching any of the UI elements at all, a special sentinel value, such as %IAM_ROLE%, that when provided for the "Access Token" of an S3 connection (i.e. the Username), triggers slightly different behavior in the S3 connection building.

Looking at the latest version of the code base in trunk for (source/ch/cyberduck/core/s3/ @ r17003), at line 114 where you handle switching between an anonymous connection (null AWSCredentials) or constucting a set of credentials based on user-provided Access Key + Secret Key, you could add a third option when IAM_ROLE_SENTINEL.equals(host.GetCredentials().GetUsername()) to construct an AWSSessionCredentials instance.

The Access Key, Secret Key and Session Token would all be derived from the running context as described above.

Last edited on Feb 25, 2015 at 11:33:07 AM by ebekker (previous) (diff)

comment:9 Changed on Feb 25, 2015 at 3:17:36 PM by dkocher

  • Resolution set to fixed
  • Status changed from assigned to closed

In r17007. Untested besides unit tests. Please install the S3 (Temporary Credentials).cyberduckprofile profile and give it a try with the latest snapshot build.

comment:11 Changed on Feb 26, 2015 at 10:43:40 AM by dkocher

Add option to override username and password requirements in profile to make it skip the login prompt in r17010.

comment:12 Changed on Feb 27, 2015 at 10:15:06 PM by dkocher

We have verified this works with Cyberduck CLI on EC2. Change the default role name s3access in the Context URI in the profile accordingly. Defaults to

[ec2-user@ip-172-30-0-79 ~]$ duck --list s3-role://cyberduck-sandbox/ -v
Authenticating as anonymous…
> GET /latest/meta-data/iam/security-credentials/aws-ec2-role HTTP/1.1
> Host:
> Connection: Keep-Alive
> User-Agent: Cyberduck/4.7 (Linux/3.14.27-25.47.amzn1.x86_64) (amd64)
> Accept-Encoding: gzip,deflate
< HTTP/1.0 200 OK
< Content-Type: text/plain
< Accept-Ranges: bytes
< ETag: "1762412530"
< Last-Modified: Fri, 27 Feb 2015 22:01:56 GMT
< Content-Length: 898
< Connection: keep-alive
< Date: Fri, 27 Feb 2015 22:09:28 GMT
Listing directory cyberduck-sandbox…

comment:13 Changed on Mar 5, 2015 at 1:36:49 PM by ebekker

We verified this works as designed, thank you!

Sorry for the delay, we did not expect you would have addressed this ticket so fast and made it available in a short-term release, so it took us a bit to be able to test it.

Note: See TracTickets for help on using tickets.