Interoperability with Ceph S3 (Radosgw)

[ben hines]

This is using a non-https Ceph S3 (Radosgw) server.

0. Set up a s3 bucket
1. Create a file: ie -

curl -X PUT -d @test.txt -i -H "x-amz-acl: public-read-write"  http://radosgw/test3/subdir/foo.txt
HTTP/1.1 200 OK

2. Verify curl can download it:

curl http://radosgw/test3/subdir/foo.txt
foo

• 3. Connect to the bucket with cyberduck and attempt to download it, fails.

According to tcpdump, Cybderduck is encoding the slash in the key-name:

GET /test3/subdir%2Ffoo.txt HTTP/1.1

The bucket index looks like:

<Contents><Key>subdir/foo.txt</Key><LastModified>2015-04-30T19:12:38.000Z</LastModified><ETag>"acbd18db4cc2f85cedef654fccc4a4d8"</ETag><Size>3</Size><StorageClass>STANDARD</StorageClass><Owner><ID/><DisplayName/></Owner></Contents>

The forward slash in the key name should not be encoded by cyberduck before it tries to download it.

I have not tested this with Amazon S3, so i'm not sure if this only affects Ceph/insecure s3.

[dkocher]

We URI encode all keys because they may contain characters outside the URI spec and also include the separator / character replacing with %2F. Any S3 interoperable provider should always URI decode request URIs. Please open an issue with Ceph Support.

[ben hines]

I'm kinda skeptical about that. Since it's the directory separator, and S3 is a flat structure, slashes are just assumed to be slashes. (ie, since there is no concept of a directory, a slash in a key name will always be just that, both a slash in the keyname as well as a fake 'path separator'. There's no case where you'd want an encoded forward-slash)

Its specifically included as 'safe' to include in key names. In fact, forward slash is specifically NOT included in the list of characters that you need to URL-Encode when passing to the S3 service, in this document:

​http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html

Cyberduck should not be URL-encoding it.

-Ben

[ben hines]

(Cyberduck SHOULD be url-encoding every other character listed in that document -- just not forward slash)

[ben hines]

I'll test out cyberduck + ceph + files with other special characters, as a test, Monday.

[dkocher]

Fix in ​78e1dbb.

[dkocher]

In r17494.