Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Interoperability with Ceph S3 (Radosgw) #8779

Closed
cyberduck opened this issue Apr 30, 2015 · 6 comments
Closed

Interoperability with Ceph S3 (Radosgw) #8779

cyberduck opened this issue Apr 30, 2015 · 6 comments
Assignees
Labels
bug fixed s3 AWS S3 Protocol Implementation
Milestone

Comments

@cyberduck
Copy link
Collaborator

bb26ca3 created the issue

This is using a non-https Ceph S3 (Radosgw) server.

  1. Set up a s3 bucket
  2. Create a file: ie -
curl -X PUT -d @test.txt -i -H "x-amz-acl: public-read-write"  http://radosgw/test3/subdir/foo.txt
HTTP/1.1 200 OK
  1. Verify curl can download it:
curl http://radosgw/test3/subdir/foo.txt
foo
    1. Connect to the bucket with cyberduck and attempt to download it, fails.

According to tcpdump, Cybderduck is encoding the slash in the key-name:

GET /test3/subdir%2Ffoo.txt HTTP/1.1

The bucket index looks like:

<Contents><Key>subdir/foo.txt</Key><LastModified>2015-04-30T19:12:38.000Z</LastModified><ETag>"acbd18db4cc2f85cedef654fccc4a4d8"</ETag><Size>3</Size><StorageClass>STANDARD</StorageClass><Owner><ID/><DisplayName/></Owner></Contents>

The forward slash in the key name should not be encoded by cyberduck before it tries to download it.

I have not tested this with Amazon S3, so i'm not sure if this only affects Ceph/insecure s3.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

We URI encode all keys because they may contain characters outside the URI spec and also include the separator / character replacing with %2F. Any S3 interoperable provider should always URI decode request URIs. Please open an issue with Ceph Support.

@cyberduck
Copy link
Collaborator Author

bb26ca3 commented

I'm kinda skeptical about that. Since it's the directory separator, and S3 is a flat structure, slashes are just assumed to be slashes. (ie, since there is no concept of a directory, a slash in a key name will always be just that, both a slash in the keyname as well as a fake 'path separator'. There's no case where you'd want an encoded forward-slash)

Its specifically included as 'safe' to include in key names. In fact, forward slash is specifically NOT included in the list of characters that you need to URL-Encode when passing to the S3 service, in this document:

http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingMetadata.html

Cyberduck should not be URL-encoding it.

-Ben

@cyberduck
Copy link
Collaborator Author

bb26ca3 commented

(Cyberduck SHOULD be url-encoding every other character listed in that document -- just not forward slash)

@cyberduck
Copy link
Collaborator Author

bb26ca3 commented

I'll test out cyberduck + ceph + files with other special characters, as a test, Monday.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Fix in 78e1dbb.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

In 8b477b8.

@iterate-ch iterate-ch locked as resolved and limited conversation to collaborators Nov 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug fixed s3 AWS S3 Protocol Implementation
Projects
None yet
Development

No branches or pull requests

2 participants