Cyberduck Mountain Duck CLI

Opened on Apr 30, 2015 at 7:18:42 PM

Closed on May 6, 2015 at 2:13:10 PM

#8779 closed defect (fixed)

Interoperability with Ceph S3 (Radosgw)

Reported by: ben hines Owned by: dkocher
Priority: normal Milestone: 4.7.1
Component: s3 Version: 4.7
Severity: normal Keywords: s3 aws
Cc: Architecture: Intel
Platform: Windows 7

Description (last modified by dkocher)

This is using a non-https Ceph S3 (Radosgw) server.

  1. Set up a s3 bucket
  2. Create a file: ie - 

curl -X PUT -d @test.txt -i -H "x-amz-acl: public-read-write"  http://radosgw/test3/subdir/foo.txt
HTTP/1.1 200 OK
  1. Verify curl can download it: 
curl http://radosgw/test3/subdir/foo.txt
foo
  • 3. Connect to the bucket with cyberduck and attempt to download it, fails.

According to tcpdump, Cybderduck is encoding the slash in the key-name: 

GET /test3/subdir%2Ffoo.txt HTTP/1.1

The bucket index looks like: 

<Contents><Key>subdir/foo.txt</Key><LastModified>2015-04-30T19:12:38.000Z</LastModified><ETag>"acbd18db4cc2f85cedef654fccc4a4d8"</ETag><Size>3</Size><StorageClass>STANDARD</StorageClass><Owner><ID/><DisplayName/></Owner></Contents>

The forward slash in the key name should not be encoded by cyberduck before it tries to download it.

I have not tested this with Amazon S3, so i'm not sure if this only affects Ceph/insecure s3.

Change History (10)

comment:1 Changed on May 1, 2015 at 6:54:13 AM by dkocher

  • Milestone set to 4.8
  • Summary changed from Files in subfolders on non-AWS S3 server can not be downloaded to Interoperability with Ceph S3 (Radosgw)

comment:2 Changed on May 1, 2015 at 6:55:01 AM by dkocher

  • Description modified (diff)

comment:3 Changed on May 2, 2015 at 6:54:09 AM by dkocher

  • Resolution set to thirdparty
  • Status changed from new to closed

We URI encode all keys because they may contain characters outside the URI spec and also include the separator / character replacing with %2F. Any S3 interoperable provider should always URI decode request URIs. Please open an issue with Ceph Support.

comment:4 Changed on May 2, 2015 at 10:02:05 AM by ben hines

I'm kinda skeptical about that. Since it's the directory separator, and S3 is a flat structure, slashes are just assumed to be slashes. (ie, since there is no concept of a directory, a slash in a key name will always be just that, both a slash in the keyname as well as a fake 'path separator'. There's no case where you'd want an encoded forward-slash)

Its specifically included as 'safe' to include in key names. In fact, forward slash is specifically NOT included in the list of characters that you need to URL-Encode when passing to the S3 service, in this document:

​http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html

Cyberduck should not be URL-encoding it.

-Ben

Last edited on May 2, 2015 at 10:09:22 AM by ben hines (previous) (diff)

comment:5 Changed on May 2, 2015 at 10:02:35 AM by ben hines

(Cyberduck SHOULD be url-encoding every other character listed in that document -- just not forward slash)

comment:6 Changed on May 2, 2015 at 10:04:02 AM by ben hines

  • Resolution thirdparty deleted
  • Status changed from closed to reopened

comment:7 Changed on May 2, 2015 at 10:05:10 AM by ben hines

I'll test out cyberduck + ceph + files with other special characters, as a test, Monday.

comment:8 Changed on May 2, 2015 at 5:48:54 PM by dkocher

  • Status changed from reopened to new

comment:10 Changed on May 6, 2015 at 2:13:10 PM by dkocher

  • Resolution set to fixed
  • Status changed from new to closed

In r17494.

Note: See TracTickets for help on using tickets.