Opened on Apr 30, 2015 at 7:18:42 PM
Closed on May 6, 2015 at 2:13:10 PM
#8779 closed defect (fixed)
Interoperability with Ceph S3 (Radosgw)
Reported by: | ben hines | Owned by: | dkocher |
---|---|---|---|
Priority: | normal | Milestone: | 4.7.1 |
Component: | s3 | Version: | 4.7 |
Severity: | normal | Keywords: | s3 aws |
Cc: | Architecture: | Intel | |
Platform: | Windows 7 |
Description (last modified by dkocher)
This is using a non-https Ceph S3 (Radosgw) server.
- Set up a s3 bucket
- Create a file: ie -
curl -X PUT -d @test.txt -i -H "x-amz-acl: public-read-write" http://radosgw/test3/subdir/foo.txt HTTP/1.1 200 OK
- Verify curl can download it:
curl http://radosgw/test3/subdir/foo.txt foo
- 3. Connect to the bucket with cyberduck and attempt to download it, fails.
According to tcpdump, Cybderduck is encoding the slash in the key-name:
GET /test3/subdir%2Ffoo.txt HTTP/1.1
The bucket index looks like:
<Contents><Key>subdir/foo.txt</Key><LastModified>2015-04-30T19:12:38.000Z</LastModified><ETag>"acbd18db4cc2f85cedef654fccc4a4d8"</ETag><Size>3</Size><StorageClass>STANDARD</StorageClass><Owner><ID/><DisplayName/></Owner></Contents>
The forward slash in the key name should not be encoded by cyberduck before it tries to download it.
I have not tested this with Amazon S3, so i'm not sure if this only affects Ceph/insecure s3.
Change History (10)
comment:1 Changed on May 1, 2015 at 6:54:13 AM by dkocher
- Milestone set to 4.8
- Summary changed from Files in subfolders on non-AWS S3 server can not be downloaded to Interoperability with Ceph S3 (Radosgw)
comment:2 Changed on May 1, 2015 at 6:55:01 AM by dkocher
- Description modified (diff)
comment:3 Changed on May 2, 2015 at 6:54:09 AM by dkocher
- Resolution set to thirdparty
- Status changed from new to closed
comment:4 Changed on May 2, 2015 at 10:02:05 AM by ben hines
I'm kinda skeptical about that. Since it's the directory separator, and S3 is a flat structure, slashes are just assumed to be slashes. (ie, since there is no concept of a directory, a slash in a key name will always be just that, both a slash in the keyname as well as a fake 'path separator'. There's no case where you'd want an encoded forward-slash)
Its specifically included as 'safe' to include in key names. In fact, forward slash is specifically NOT included in the list of characters that you need to URL-Encode when passing to the S3 service, in this document:
http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html
Cyberduck should not be URL-encoding it.
-Ben
comment:5 Changed on May 2, 2015 at 10:02:35 AM by ben hines
(Cyberduck SHOULD be url-encoding every other character listed in that document -- just not forward slash)
comment:6 Changed on May 2, 2015 at 10:04:02 AM by ben hines
- Resolution thirdparty deleted
- Status changed from closed to reopened
comment:7 Changed on May 2, 2015 at 10:05:10 AM by ben hines
I'll test out cyberduck + ceph + files with other special characters, as a test, Monday.
comment:8 Changed on May 2, 2015 at 5:48:54 PM by dkocher
- Status changed from reopened to new
comment:9 Changed on May 6, 2015 at 9:36:25 AM by dkocher
Fix in 78e1dbb.
comment:10 Changed on May 6, 2015 at 2:13:10 PM by dkocher
- Resolution set to fixed
- Status changed from new to closed
In r17494.
We URI encode all keys because they may contain characters outside the URI spec and also include the separator / character replacing with %2F. Any S3 interoperable provider should always URI decode request URIs. Please open an issue with Ceph Support.