Cyberduck Mountain Duck CLI

#8779 closed defect (fixed)

Interoperability with Ceph S3 (Radosgw)

Reported by: ben hines Owned by: dkocher
Priority: normal Milestone: 4.7.1
Component: s3 Version: 4.7
Severity: normal Keywords: s3 aws
Cc: Architecture: Intel
Platform: Windows 7

Description (last modified by dkocher)

This is using a non-https Ceph S3 (Radosgw) server.

  1. Set up a s3 bucket
  2. Create a file: ie -

curl -X PUT -d @test.txt -i -H "x-amz-acl: public-read-write"  http://radosgw/test3/subdir/foo.txt
HTTP/1.1 200 OK
  1. Verify curl can download it:
curl http://radosgw/test3/subdir/foo.txt
foo
  • 3. Connect to the bucket with cyberduck and attempt to download it, fails.

According to tcpdump, Cybderduck is encoding the slash in the key-name:

GET /test3/subdir%2Ffoo.txt HTTP/1.1

The bucket index looks like:

<Contents><Key>subdir/foo.txt</Key><LastModified>2015-04-30T19:12:38.000Z</LastModified><ETag>"acbd18db4cc2f85cedef654fccc4a4d8"</ETag><Size>3</Size><StorageClass>STANDARD</StorageClass><Owner><ID/><DisplayName/></Owner></Contents>

The forward slash in the key name should not be encoded by cyberduck before it tries to download it.

I have not tested this with Amazon S3, so i'm not sure if this only affects Ceph/insecure s3.

Change History (10)

comment:1 Changed on May 1, 2015 at 6:54:13 AM by dkocher

  • Milestone set to 4.8
  • Summary changed from Files in subfolders on non-AWS S3 server can not be downloaded to Interoperability with Ceph S3 (Radosgw)

comment:2 Changed on May 1, 2015 at 6:55:01 AM by dkocher

  • Description modified (diff)

comment:3 Changed on May 2, 2015 at 6:54:09 AM by dkocher

  • Resolution set to thirdparty
  • Status changed from new to closed

We URI encode all keys because they may contain characters outside the URI spec and also include the separator / character replacing with %2F. Any S3 interoperable provider should always URI decode request URIs. Please open an issue with Ceph Support.

comment:4 Changed on May 2, 2015 at 10:02:05 AM by ben hines

I'm kinda skeptical about that. Since it's the directory separator, and S3 is a flat structure, slashes are just assumed to be slashes. Its specifically included as 'safe' to include in key names. In fact, forward slash is specifically NOT included in the list of characters that you need to URL-Encode when passing to the S3 service, in this document:

http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html

Cyberduck should not be URL-encoding it.

-Ben

Version 0, edited on May 2, 2015 at 10:02:05 AM by ben hines (next)

comment:5 Changed on May 2, 2015 at 10:02:35 AM by ben hines

(Cyberduck SHOULD be url-encoding every other character listed in that document -- just not forward slash)

comment:6 Changed on May 2, 2015 at 10:04:02 AM by ben hines

  • Resolution thirdparty deleted
  • Status changed from closed to reopened

comment:7 Changed on May 2, 2015 at 10:05:10 AM by ben hines

I'll test out cyberduck + ceph + files with other special characters, as a test, Monday.

comment:8 Changed on May 2, 2015 at 5:48:54 PM by dkocher

  • Status changed from reopened to new

comment:10 Changed on May 6, 2015 at 2:13:10 PM by dkocher

  • Resolution set to fixed
  • Status changed from new to closed

In r17494.

Note: See TracTickets for help on using tickets.
swiss made software