Cyberduck Mountain Duck CLI

Opened 10 months ago

Closed 8 months ago

Last modified 8 months ago

#10170 closed enhancement (fixed)

Expect header included in the list of the SignedHeaders for AWS4 signature

Reported by: zioproto Owned by: dkocher
Priority: normal Milestone: 6.4.0
Component: s3 Version: 6.3.1
Severity: normal Keywords:
Cc: Architecture:
Platform:

Description (last modified by dkocher)

I have a problem with Cyberduck 6.3.1 connecting to my Rados Gateway with the S3 protocol.

Cyberduck does not send the Expect header, but adds it in the list of the SignedHeaders.

The AWS4 signature is not working because Cyberduck sends this Authentication header:

Authorization: AWS4-HMAC-SHA256 Credential=[..stripped..out..]/20171206/us-east-1/s3/aws4_request,SignedHeaders=content-type;date;expect;host;x-amz-content-sha256;x-amz-date,Signature=ec83ddc9de6b379c929a8307ff5cb918a2ffad533e11408c079607ad8004cf3f
User-Agent: Cyberduck/6.3.1.27228 (Mac OS X/10.12.6) (x86_64)

The SignedHeaders contains the "expect" header, however Cyberduck is not sending this header out in the HTTP request.

I tried adding with Nginx in the middle the missing header:

proxy_set_header Expect 100-continue;

and this fixed it, now the backend can verify the signature because the header is there.

to reproduce just try to create a new bucket

I use this profile:

Change History (16)

comment:1 Changed 10 months ago by dkocher

  • Component changed from core to s3
  • Milestone set to 7.0
  • Owner set to dkocher
  • Status changed from new to assigned
  • Summary changed from Cyberduck does not send the Expect header, but adds it in the list of the SignedHeaders for AWS4 signature to Does not send the Expect header, but adds it in the list of the SignedHeaders for AWS4 signature

comment:2 Changed 10 months ago by dkocher

  • Description modified (diff)

comment:3 Changed 10 months ago by dkocher

I cannot reproduce the issue. We add a Expect: 100-continue to all PUT requests and include it in the AWS4-HMAC-SHA256 signature. I have added a test case in e451ce03b1e6eb4393f84a3400f5ac76eb647c57.

comment:4 Changed 10 months ago by yla

Can you please post the transcript from the log drawer (⌘-L) when trying to create a bucket?

comment:5 Changed 10 months ago by zioproto

I did not know about ( CMD-L ) ! You are right, it looks like the header is there. Is it possible for some reason that NGINX is not forwarding the header ? By the way, why are you including this header in the AWS4 signature ? It is not a mandatory header to put in the signed headers.

thank you

PUT /luminous/ HTTP/1.1
Date: Tue, 12 Dec 2017 09:37:12 GMT
Expect: 100-continue
Content-Type: application/octet-stream
x-amz-acl: private
x-amz-content-sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Host: os.unil.cloud.switch.ch
x-amz-date: 20171212T093712Z
Authorization: ********
Content-Length: 0
Connection: Keep-Alive
User-Agent: Cyberduck/6.3.1.27228 (Mac OS X/10.12.6) (x86_64)
HTTP/1.1 403 Forbidden
Server: nginx
Date: Tue, 12 Dec 2017 09:37:12 GMT
Content-Type: application/xml
Content-Length: 200
Connection: keep-alive
x-amz-request-id: tx000000000000000000026-005a2fa348-6645540-default
Accept-Ranges: bytes

comment:6 Changed 10 months ago by zioproto

I start to think this could be a nginx bug.

In nginx if I add the directive:

proxy_pass_header Expect;

the header is not forwarded, even if it is logged in the Nginx log file.

If I add the directive

 proxy_set_header Expect 100-continue;

then the header is added and the signature verifies correctly.

Note that all the others SignedHeaders are forwarded without a special config, is just this Expect header that has problems.

Nginx version 1.12.1-0+xenial0

comment:7 Changed 10 months ago by zioproto

I found a workaround in my Nginx config In my location config section I added the following:

        if ( $http_expect ) {
          set $expect $http_expect;
        }
        proxy_set_header Expect $expect;

comment:8 Changed 10 months ago by dkocher

  • Milestone changed from 7.0 to 6.3.2
  • Resolution set to thirdparty
  • Status changed from assigned to closed

Added test in r43544. Thanks for the background infromation about nginx configuration issues.

comment:9 Changed 9 months ago by tschop

  • Resolution thirdparty deleted
  • Status changed from closed to reopened

Unfortunately the workaround in the Nginx configuration doesn't work as expected. We use Nginx as a pure proxy, and because Nginx try to minimize the communication with the backend, it will not forward the "Expect: 100-continue". It is the default behaviour of Nginx, and it is not possible to disable it.

We add a Expect: 100-continue to all PUT requests and include it in the AWS4-HMAC-SHA256 signature.


Why do you include the Expect header in the AWS4 SignedHeaders signature? This header is not relevant for the content of the request, and does not carry any useful information.

Last edited 9 months ago by tschop (previous) (diff)

comment:10 Changed 9 months ago by zioproto

I would also strongly suggest not to include in the signature the Expect header. To make S3-like storage solutions scale horizontally, there is often a reverse proxy in front of the storage backend, that might not forward the Expect header. This header has always the same content string '100-continue', and this makes including in the signature this header nonsense.

Thank you

comment:11 Changed 9 months ago by dkocher

  • Milestone changed from 6.3.2 to 7.0

comment:12 Changed 8 months ago by dkocher

  • Summary changed from Does not send the Expect header, but adds it in the list of the SignedHeaders for AWS4 signature to Expect header included in the list of the SignedHeaders for AWS4 signature

comment:13 Changed 8 months ago by dkocher

  • Type changed from defect to enhancement

comment:14 Changed 8 months ago by dkocher

  • Resolution set to fixed
  • Status changed from reopened to closed

comment:15 Changed 8 months ago by dkocher

  • Milestone changed from 7.0 to 6.3.6

comment:16 Changed 8 months ago by dkocher

  • Milestone changed from 6.3.6 to 6.4.0

Milestone renamed

Note: See TracTickets for help on using tickets.
swiss made software