Cyberduck Mountain Duck CLI

Opened 5 months ago

Closed 5 months ago

Last modified 5 months ago

#10194 closed enhancement (fixed)

Support for OCI Object Storage

Reported by: ilker Owned by: dkocher
Priority: normal Milestone: 6.4.0
Component: s3 Version: 6.3.4
Severity: normal Keywords:
Cc: Architecture:
Platform:

Description (last modified by ilker)

Oracle Cloud Infrastructure (OCI) Object Storage provides an S3 Compatible API. It would really be great if Cyberduck could also support it. Looks like it almost works but fails since OCI Object Storage uses different endpoints than S3 does. Cyberduck tries to use us-east-1 region identifier, which results in failure in authentication.

More information on OCI S3 Compatible API is available here: https://cloud.oracle.com/infrastructure/storage/object-storage/faq

As a side note, Cyberduck already supports OCI Object Storage Classic via Swift, which is different than OCI Object Storage.

Change History (20)

comment:1 Changed 5 months ago by ilker

Please see & review PR that addresses this issue:

https://bitbucket.org/dkocher/jets3t/pull-requests/13/master/diff

comment:2 Changed 5 months ago by dkocher

  • Component changed from core to s3
  • Milestone set to 7.0
  • Owner set to dkocher

The hack is not required. Please create connection profile with a customRegion setting.

comment:3 Changed 5 months ago by ilker

  • Description modified (diff)

Thank you dkocher. I had tried it but it does not work. I use the following profile:

        <Key>Region</Key>
        <String>eu-frankfurt-1</String>
        <Key>Regions</Key>
        <array>
                <string>us-phoenix-1</string>
                <string>us-ashburn-1</string>
                <string>eu-frankfurt-1</string>
        </array>

When I try to authenticate, I get the following error message: https://www.ilkertemir.com/cyberduck.png

Last edited 5 months ago by ilker (previous) (diff)

comment:4 Changed 5 months ago by ilker

BTW, I think you meant Region setting as there doesn't seem to be a customRegion setting? Regardless, I also tried customRegion setting and the result is the same.

Last edited 5 months ago by ilker (previous) (diff)

comment:5 Changed 5 months ago by ilker

@dkocher, did you have a chance to look into this? Whether via the submitted PR or a configuration change, it will be great if Cyberduck can support OCI Object Storage.

Please kindly let me know what you think and if it is possible to put this into 7.0 release.

comment:6 Changed 5 months ago by dkocher

Yes, this is planned. I will want to have a look into this to find a generic solution for any S3 provider.

comment:7 Changed 5 months ago by dkocher

As a workaround it should be possible to use the OpenStack Swift endpoints.

comment:8 Changed 5 months ago by dkocher

  • Resolution set to fixed
  • Status changed from new to closed

Fix in r43740 (upstream https://bitbucket.org/dkocher/jets3t/commits/5b3b2d1b2a8199d6d6a676482e57e05ab2ecc5c4?at=bugfix/issue-47). The Region setting in the connection profile should now be used as the default when signing requests with AWS4-HMAC-SHA256. I am unable to verify with OCI Object Storage – horrendous user interface to setup services I am unable to follow. Verified against objects-us-west-1.dream.io.

comment:9 Changed 5 months ago by dkocher

@ilker It would be awesome if you could update our wiki page about Oracle Cloud Storage at https://trac.cyberduck.io/wiki/help/en/howto/oraclecloud and provide connection profiles. Please update to the latest snapshot build available for testing.

comment:10 Changed 5 months ago by ilker

@dkocher Thank you. Unfortunately this didn't work but looks like we are pretty close. jets3t changes seem to do the job, but Region value from the configuration does not make it to jets3t.

This part of the code (https://github.com/iterate-ch/cyberduck/blob/394248296c53308211e28581ec6175fd15300552/s3/src/main/java/ch/cyberduck/core/s3/S3Session.java#L203) seems to set the storage-service.default-region to null. Not entirely sure how host.getRegion() should get the Region out of the config but somewhere along the way it seems to be getting lost.

For reference here is the configuration I use: https://github.com/itemir/cyberduck/blob/master/profiles/OCI%20Object%20Storage%20eu-frankfurt-1.cyberduckprofile

P.S. Happy to update the wiki and upload the profiles once we get this running. I will also send a PR to merge the profiles to the main repo.

P.S. II A similar issue seems to exist for Regions Key Values as well. Regions as specified in the configuration do not seem to be parsed out. It might be caused by the same underlying issue so fixing this may as well fix it too. If not, I will file a separate ticket for it once we get this sorted out.

comment:11 Changed 5 months ago by ilker

  • Resolution fixed deleted
  • Status changed from closed to reopened

comment:12 Changed 5 months ago by dkocher

Please make elements key and string in the connection profile lowercase.

comment:13 Changed 5 months ago by dkocher

Please make sure the Vendor value is unique for all profiles.

comment:14 Changed 5 months ago by dkocher

It looks like error messages in the XML response from the service include JSON in the Message element of the error response. This should be plain text instead.

<?xml version='1.0' encoding='UTF-8'?><Error><Message>AuthDataPlaneServerException{statusCode=404, body='{  "code" : "NotFound",  "message" : "Not Found"}'}</Message><Code>SignatureDoesNotMatch</Code></Error>

comment:15 Changed 5 months ago by dkocher

  • Resolution set to fixed
  • Status changed from reopened to closed

comment:16 Changed 5 months ago by dkocher

  • Resolution fixed deleted
  • Status changed from closed to reopened

Reopen awaiting feedback from testing.

comment:17 Changed 5 months ago by ilker

  • Resolution set to fixed
  • Status changed from reopened to closed

@dkocher, thank you. This did work. I will update the wiki page and link to updated connection profiles. I sent a PR for adding profiles to the main repo. Also ACK on json vs text error responses.

One last question, do you have any tentative timeline for the 7.0 release?

Thanks again and greetings to Switzerland.

comment:18 Changed 5 months ago by dkocher

We will retarget this to a minor update – we release about once a month.

comment:19 Changed 5 months ago by dkocher

  • Milestone changed from 7.0 to 6.3.6

comment:20 Changed 5 months ago by dkocher

  • Milestone changed from 6.3.6 to 6.4.0

Milestone renamed

Note: See TracTickets for help on using tickets.
swiss made software