Cyberduck Mountain Duck CLI

#11735 closed enhancement (fixed)

Support S3 interface endpoints (AWSPrivateLink for Amazon S3)

Reported by: malaval Owned by: dkocher
Priority: normal Milestone: 7.10.1
Component: s3 Version: 7.9.2
Severity: normal Keywords:
Cc: Architecture:
Platform:

Description (last modified by dkocher)

S3 interface endpoints enable to connect to Amazon S3 using a private IP address:

I am unable to connect to Amazon S3 using the interface endpoint URL (e.g. vpce-0971cacd1f2xxxxxxxxx.s3.eu-west-1.vpce.amazonaws.com) as the server hostname. Cyberduck continously tries to authenticate (I see thousands of packets in Wireshark) and fails a few minutes later. The issue comes from how Cyberduck generates the SigV4 signature, because it considers that "vpce" is the region (e.g. HTTP header Authorization is AWS4-HMAC-SHA256 Credential=AKIASFI36Y5VXXXXXXX/20210702/vpce/s3/aws4_request which fails).

I think that two things should be corrected in Cyberduck:

  • Fetch the region differently from the server endpoint URL
  • Consider S3 interface endpoint URL as "special URL" and use this URL only (don't use dualstack or North Virginia as the default region to list existing S3 buckets)

As a workaround, I was able to connect to a S3 interface endpoint by:

  • Resolving s3.eu-west-1.amazonaws.com to one of the private IP addresses of the interface endpoint (added an entry in the hosts file)
  • Applying the default parameters:
s3.bucket.virtualhost.disable=true
s3.endpoint.dualstack.enable=false
s3.endpoint.format.ipv4=s3.eu-west-1.amazonaws.com

However, it would be great if Cyberduck could natively support S3 interface endpoints, without all these tricks.

Change History (18)

comment:1 Changed on Jul 18, 2021 at 9:50:20 PM by dkocher

  • Description modified (diff)

comment:2 Changed on Jul 18, 2021 at 9:50:38 PM by dkocher

  • Description modified (diff)

comment:3 Changed on Jul 18, 2021 at 9:51:39 PM by dkocher

  • Milestone set to 8.0
  • Owner set to dkocher
  • Status changed from new to assigned
  • Summary changed from Unable to use S3 interface endpoints to Support S3 interface endpoints (AWSPrivateLink for Amazon S3)

comment:4 Changed on Jul 18, 2021 at 9:51:46 PM by dkocher

  • Type changed from defect to enhancement

comment:5 Changed on Jul 18, 2021 at 10:18:46 PM by dkocher

  • Description modified (diff)

comment:6 Changed on Jul 30, 2021 at 1:47:10 PM by dkocher

We already check the hostname using the regular expression ([a-z0-9\-]+\.)?s3(\.dualstack)?(\.[a-z0-9\-]+)?\.amazonaws\.com and thus no usage of the dual stack endpoints should apply.

comment:7 Changed on Jul 30, 2021 at 1:47:38 PM by dkocher

Dependent on upstream changeset to detect region.

comment:8 Changed on Jul 30, 2021 at 1:49:25 PM by dkocher

Existing documentation to disable the use of virtual host style requests in connection profile.

comment:9 Changed on Jul 30, 2021 at 1:51:47 PM by dkocher

  • Description modified (diff)

comment:10 Changed on Jul 30, 2021 at 2:13:46 PM by dkocher

Regional DNS names include a unique VPC endpoint ID, a service identifier, the AWS Region, and vpce.amazonaws.com in its name. For example, for VPC endpoint ID vpce-1a2b3c4d, the DNS name generated might be similar to vpce-1a2b3c4d-5e6f.s3.us-east-1.vpce.amazonaws.com.

comment:11 Changed on Aug 2, 2021 at 9:20:09 AM by dkocher

  • Milestone changed from 8.0 to 7.10.1

comment:12 Changed on Aug 3, 2021 at 1:25:41 PM by dkocher

  • Resolution set to fixed
  • Status changed from assigned to closed

In r51610 with upstream fix. Please try the latest snapshot build with this connection profile.

Last edited on Aug 5, 2021 at 8:41:42 AM by dkocher (previous) (diff)

comment:13 follow-up: Changed on Aug 5, 2021 at 7:47:12 AM by malaval

Hello,

Many thanks for the quick update! I tested with Access Key and Secret and it works. However I tried to create a mix profile with STS credentials and Private Link, and it does not work. The profile is:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
    <dict>
        <key>Protocol</key>
        <string>s3</string>
        <key>Vendor</key>
        <string>s3-sts-privatelink</string>
        <key>Scheme</key>
        <string>https</string>
        <key>Description</key>
        <string>S3 (STS and Private Link)</string>
        <key>Default Nickname</key>
        <string>S3 (STS and Private Link)</string>
        <key>Username Placeholder</key>
        <string>Profile Name in ~/.aws/credentials</string>
        <key>Password Configurable</key>
        <false/>
        <key>Token Configurable</key>
        <false/>
        <key>Anonymous Configurable</key>
        <false/>
        <key>Properties</key>
        <array>
            <string>s3service.disable-dns-buckets=true</string>
            <string>s3.endpoint.dualstack.enable=false</string>
        </array>
    </dict>
</plist>

It works when the server is s3.amazonaws.com, but I get an error message "The AWS Access Key Id you provided does not exist in our records". It seems like it does not use STS credentials.

Is that the expected behavior? Thanks!

Last edited on Aug 5, 2021 at 8:18:22 AM by dkocher (previous) (diff)

comment:14 in reply to: ↑ 13 Changed on Aug 5, 2021 at 8:39:31 AM by dkocher

Replying to malaval:

It works when the server is s3.amazonaws.com, but I get an error message "The AWS Access Key Id you provided does not exist in our records". It seems like it does not use STS credentials.

Is that the expected behavior? Thanks!

This requires an additional fix as we currently handle the VPC endpoints as non AWS S3.

comment:15 Changed on Aug 5, 2021 at 8:42:03 AM by dkocher

  • Resolution fixed deleted
  • Status changed from closed to reopened

comment:16 Changed on Aug 6, 2021 at 12:43:07 PM by dkocher

  • Resolution set to fixed
  • Status changed from reopened to closed

Additional fix in r51618.

comment:17 Changed on Aug 10, 2021 at 7:29:27 AM by malaval

Thank you! I don't manage to get the last release work. With the latest snapshot build (35295), I tried to both approaches and none of them worked for me. What should be the cyberduckprofile to use STS credentials with VPC endpoints?

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
    <dict>
        <key>Protocol</key>
        <string>s3</string>
        <key>Vendor</key>
        <string>s3-privatelink</string>
        <key>Scheme</key>
        <string>https</string>
        <key>Description</key>
        <string>AWS PrivateLink for Amazon S3 (VPC endpoint)</string>
        <key>Hostname Configurable</key>
        <true/>
        <key>Port Configurable</key>
        <true/>
        <key>Username Placeholder</key>
        <string>Profile Name in ~/.aws/credentials</string>
        <key>Password Configurable</key>
        <false/>
        <key>Token Configurable</key>
        <false/>
        <key>Anonymous Configurable</key>
        <false/>
        <key>Properties</key>
        <array>
            <string>s3service.disable-dns-buckets=true</string>
            <string>s3.endpoint.dualstack.enable=false</string>
        </array>
    </dict>
</plist>

Or

<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
    <dict>
        <key>Protocol</key>
        <string>s3</string>
        <key>Vendor</key>
        <string>s3-token</string>
        <key>Description</key>
        <string>S3 (Credentials from AWS Security Token Service)</string>
        <key>Default Nickname</key>
        <string>S3 (Credentials from AWS Security Token Service)</string>
        <key>Username Placeholder</key>
        <string>Profile Name in ~/.aws/credentials</string>
        <key>Password Configurable</key>
        <false/>
        <key>Token Configurable</key>
        <false/>
        <key>Anonymous Configurable</key>
        <false/>
        <key>Properties</key>
        <array>
            <string>s3service.disable-dns-buckets=true</string>
            <string>s3.endpoint.dualstack.enable=false</string>
        </array>
    </dict>
</plist>

comment:18 Changed on Aug 12, 2021 at 12:55:25 PM by dkocher

Can you please attach the debug log output for the connection attempt. To enable debug logging open a Terminal.app window and enter

defaults write ~/Library/Preferences/ch.sudo.cyberduck.plist logging debug

Log output can be found in the cyberduck.log file in~/Library/Logs/Cyberduck. You can easily reach this file in Console.app (Open from /Applications/Utilities) under Reports → Log Reports → cyberduck.log.

Note: See TracTickets for help on using tickets.