Cyberduck Mountain Duck CLI

#9711 closed feature (fixed)

Support Manta Object Store

Reported by: elijah Owned by: dkocher
Priority: low Milestone: 6.3
Component: core Version: 5.1
Severity: normal Keywords:
Cc: Architecture:
Platform:

Description

Manta is an open source object store like S3 or Swift. It is widely in use and will be quite ubiquitous now that Samsung has purchased Joyent (the creators of Manta).

This ticket is to request that support for Manta be added to CyberDuck.

There is a Java SDK available for Manta.

Attachments (4)

Info.plist.debug (15.5 KB) - added by dkocher on May 17, 2017 at 7:38:57 AM.
Debug configuration
windows-build-failure.log (43.2 KB) - added by tjcelaya on Oct 26, 2017 at 1:35:53 AM.
windows-build-failure.log
windows-build-failure-2017-09-26.log (194.2 KB) - added by tjcelaya on Oct 26, 2017 at 7:59:25 PM.
Build errors after installing msbuildtasks
windows-build-failure-2017-09-27.log (15.1 KB) - added by tjcelaya on Oct 27, 2017 at 5:30:53 PM.

Download all attachments as: .zip

Change History (43)

comment:1 Changed on Sep 24, 2016 at 7:36:12 AM by dkocher

This is certainly possible but we would need some more users to vote this up to justify the effort.

comment:2 Changed on Jan 16, 2017 at 1:05:52 PM by dkocher

  • Summary changed from Add support for the open source object store Manta to Support Manta Object Store

comment:3 Changed on Jan 16, 2017 at 1:31:49 PM by dkocher

  • Priority changed from normal to low

comment:4 Changed on May 8, 2017 at 9:06:11 PM by tjcelaya

Hey there, Tomas from Joyent here, I've been using Cyberduck for years now and want to contribute to the project. I'm researching what it would take to implement this and had a look at ticket #9799 along with the associated changeset which added support for Microsoft OneDrive. After reviewing the changes and getting project building locally, I'm interested in implementing a storage provider for Manta within Cyberduck. Following the example of r38915, it looks like the steps for adding a new provider is as follows:

  • Create a new module in the the top-level cyberduck project and add it to the root project pom.xml
  • extend AbstractProtocol and register the new protocol in ch.cyberduck.cli.Terminal and ch.cyberduck.ui.cocoa.MainApplication
  • add defaults to ch.cyberduck.core.preferences.Preferences
  • implement interfaces under ch.cyberduck.core.features.* within the new module
  • create a cyberduckprofile describing the connection information required (server, manta user/subuser, manta key id) and the field labels
  • potentially, add new params to AbstractProtocol to reflect the usage of an SSH Key Fingerprint instead of a user password or client secret

Please let me know if I've missed any critical details or there are any points in need of clarfication. The divergence in terminology for the authentication secret stems from the fact that authentication for Manta relies on a combination of username and an SSH Key fingerprint for a key that has previously been uploaded to the Manta instance in use (see Setting Up Your Environment in the Manta documentation for details). Since Cyberduck provides a dropdown field for supplying an SSH Private Key in addition to text input for passwords there is some ambiguity regarding the best approach to take, please see Question 1 for some proposals about how to handle this.

Questions:

  1. The Open Connection dialog includes fields for both Password/AuthCode and an SSH private key. This gives us some options for authenticating users (since the labels are configurable, Password will refer to the text input and SSH Private Key refers to the dropdown that is enabled when the SFTP protocol is chosen):
    1. disabling the Password field and expecting users to select a SSH Private Key. This seems to be the most intuitive to handle key selection.
    2. disabling the SSH Private Key field and expecting users to supply the key fingerprint in the Password field, similarly to the way the Manta documentation asks users to set the MANTA_KEY_ID environment variable. This aligns with Manta CLI configuration but is not user-friendly.
  2. Sonar is mentioned on the main page of the issue tracker but is inaccessible through https://sonar.cyberduck.io/ and I couldn't find any style-checking plugin in the project, is there a different code style tool in use?
  3. The modifications in https://trac.cyberduck.io/changeset/38915/trunk/Cyberduck.xcodeproj/project.pbxproj refer to resources being added to the Xcode Project. Are there any special considerations around adding to this file or would I be able to make this change myself?
  4. The java-manta library allows for optional Client-Side Encryption while the Cyberduck roadmap indicates support for Cryptomator is being added. Handling encryption through Manta's client library could lead to issues and confusion, in addition to requiring us to either collect more configuration options or force certain settings from Cyberduck. How the Manta provider in Cyberduck will support encryption is still an open question I'd like to discuss further once these questions are ironed out.

comment:5 Changed on May 9, 2017 at 8:55:39 AM by dkocher

Fixed the sonar link to point to https://sonar.iterate.ch/.

comment:6 Changed on May 9, 2017 at 8:57:07 AM by dkocher

The codestyle is part of the project in the .idea directory and picked up automatically when opening the project. Obviously this only works when working with IntelliJ IDEA which is our preferred IDE.

comment:7 Changed on May 9, 2017 at 8:59:11 AM by dkocher

I would suggest to move the client-side encryption support in Manta to a different ticket. Our Cryptomator integration is now finished and will be part of 6.0 available later today.

comment:8 Changed on May 9, 2017 at 9:01:33 AM by dkocher

You will need XCode installed to make changes to the project.pbxproj project file. Adding resources there is not a priority and can be done at the very last to include resources by default. You can install your new connection profile by double clicking – see Profiles.

comment:9 Changed on May 9, 2017 at 9:03:49 AM by dkocher

For the ambiguity for the SSH key fingerprint required you will want to at least override getPasswordPlaceholder from the Protocol interface. This will adjust the label and placeholder text when editing the bookmark. Or you can disable the password field with isPasswordConfigurable. Currently the private key popup is only enabled for protocols of type sftp (see getType in Protocol).

comment:10 Changed on May 9, 2017 at 9:12:15 AM by dkocher

You analysis about what is to implement is right. Don't hesitate to ping me for any further clarifications.

comment:11 Changed on May 17, 2017 at 12:25:11 AM by tjcelaya

  • Owner set to tjcelaya

comment:12 Changed on May 17, 2017 at 12:30:12 AM by tjcelaya

Work on this is progressing though I've got two questions thus far:

When attempting to build and run the project in Xcode (using Cyberduck.xcodeproj to open the project) the application crashes immediately with the error pasted below. I haven't had a chance to debug it further but noticed there was no output in the Xcode console and am not familiar with logging in Objective-C. Any suggestions?

Error loading runtime bundle.
The application failed to launch with a unrecoverable exception.

When attempting to turn on debugging using the debug.sh script I get an error about paths, though there is a file at that path it's missing the .debug suffix:

  [cyberduck](master) PDT 17:13 UTC 00:13
./debug.sh -e
Enabling debug configuration...
/Users/tomascelaya/repo/cyberduck
cp: /Users/tomascelaya/repo/cyberduck/setup/app/Info.plist.debug: No such file or directory

Not sure how to resolve the first issue further without getting past the second.

Last edited on May 17, 2017 at 12:31:33 AM by tjcelaya (previous) (diff)

comment:13 Changed on May 17, 2017 at 7:38:41 AM by dkocher

Create a debug configuration Info.plist.debug yourself by duplicating the Info.plist file and prepend -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005 to the value for VMOptions. I will attach a sample.

Changed on May 17, 2017 at 7:38:57 AM by dkocher

Debug configuration

comment:14 Changed on May 17, 2017 at 7:41:13 AM by dkocher

I never run the project from XCode – the build in XCode is partial and only covers the native libraries. You will have to run mvn package -DskipTests -pl osx -am.

You can then execute the Cyberduck.app binary in osx/target or run with the i18n.sh script using i18n/src/main/resources/i18n.sh -l en.lproj -r.

comment:15 Changed on May 17, 2017 at 7:41:58 AM by dkocher

We are now mirroring to https://github.com/iterate-ch/cyberduck to make contributions easier.

comment:16 Changed on May 17, 2017 at 10:24:39 PM by tjcelaya

Thanks for all the help so far! Definitely getting closer sincemvn package succeeds without any errors. Running the bundle at osx/target/Cyberduck.app closes as soon as it's launched though and invoking open osx/target/Cyberduck.app yields the following error:

LSOpenURLsWithRole() failed with error -10810 for the file /Users/tomascelaya/repo/cyberduck/osx/target/Cyberduck.app.

Thankfully i18n.sh yield a more obvious error message:

  [cyberduck](master) PDT 14:39 UTC 21:39
i18n/src/main/resources/i18n.sh -l en.lproj -r
Using Language:en.lproj
Running app using en...
Error: Could not find or load main class ch.cyberduck.ui.cocoa.MainApplication
*** DONE. ***

The error specifically occurs on the last line in run() but attempting to run the Cyberduck.app/Contents/Cyberduck binary prints out the same message, so it seems like I'm missing something related to classpath configuration. I'm attempting to track down the issue further by inspecting file open syscalls but haven't found anything promising yet.

comment:17 follow-up: Changed on May 17, 2017 at 10:31:36 PM by dkocher

Please try running the install phase with mvn install -DskipTests -pl osx -am.

comment:18 in reply to: ↑ 17 Changed on May 17, 2017 at 11:49:18 PM by tjcelaya

That did the trick! A warning came up about "App Transport Security" but everything seems to work and the manta endpoints are all https so I don't think it'll cause any issues.

Thanks!

Replying to dkocher:

Please try running the install phase with mvn install -DskipTests -pl osx -am.

comment:19 follow-up: Changed on May 24, 2017 at 11:58:51 PM by tjcelaya

I'm nearing completion and will be ready to submit my changes as soon a I resolve an issue with Manta subusers, would a pull request on Github work or is there some other process for submitting contributions?

comment:20 in reply to: ↑ 19 Changed on May 25, 2017 at 5:24:21 AM by dkocher

Replying to tjcelaya:

I'm nearing completion and will be ready to submit my changes as soon a I resolve an issue with Manta subusers, would a pull request on Github work or is there some other process for submitting contributions?

A pull request on GitHub works just great.

comment:21 Changed on May 25, 2017 at 5:27:23 AM by dkocher

  • Milestone set to 7.0

comment:24 Changed on Oct 26, 2017 at 1:25:12 AM by tjcelaya

java-manta 3.7.1 3.1.7 (or 3.8.0 3.2.0) is undergoing performance testing and will be released very soon. In the meantime I wanted to try building Cyberduck on Windows and have been running into an issue with Maven/Ant/MSBuild. I've installed the following software:

  • Java 1.8u152 JDK
  • Maven 3.5.2
  • Nuget 4.3.0.4406
  • .NET Framework 4.7.1

I've included the full Maven output when running mvn install -DskipTests -am -pl windows since I'm not sure if this is caused by a separate module from the one that's failing. The machine in use is a Windows Server 2016 Standard VM running on Joyent's Public Cloud: [ moved to file ]

Last edited on Nov 8, 2017 at 1:49:10 AM by tjcelaya (previous) (diff)

Changed on Oct 26, 2017 at 1:35:53 AM by tjcelaya

windows-build-failure.log

comment:26 Changed on Oct 26, 2017 at 7:58:25 PM by tjcelaya

Thanks, I installed the latest msbuildtasks MSI and am getting further into the build but have encountered some confusing errors. I've attached them as windows-build-failure-2017-09-26.log.

Which version of the .NET Framework and Windows SDK is used for builds? I discovered that I needed to install the Windows 8 SDK (https://developer.microsoft.com/en-us/windows/downloads/windows-8-sdk) based on the error message that occurred after adding msbuildtasks but the new errors seem to be related to compiler versions? In order to install as few conflicting version of .NET as possible I am using the MSBuild executable from the .NET Framework installed by Windows 8 SDK ( added C:\Windows\Microsoft.NET\Framework\v4.0.30319 to my PATH since it contains MSBuild.exe). These errors are the same on both Windows Server 2012 and Windows Server 2016.

Changed on Oct 26, 2017 at 7:59:25 PM by tjcelaya

Build errors after installing msbuildtasks

comment:27 Changed on Oct 27, 2017 at 6:20:32 AM by jmalek

Using MSBuild from C:\Windows\Microsoft.NET\Framework\v4.0.30319 will not work as we are using features not present in this version.

Please install Visual Studio 2017 from https://www.visualstudio.com/en/vs/ and include following "additional items" upon installation. "Workloads" required to build Cyberduck for Windows:

  • .NET Desktop development

Additional items required to build Cyberduck for Windows or Cyberduck for Windows Store:

  • Windows 10 SDK (10.0.14393.0)

For more information about workloads and additional items in visual studios new installer go to https://docs.microsoft.com/en-us/visualstudio/install/modify-visual-studio

You will, however, get additional errors after building as we create an installer just after building. You can ignore those errors.

comment:28 follow-up: Changed on Oct 27, 2017 at 5:32:29 PM by tjcelaya

The Visual Studio 2017 installation (w/ Windows 10 SDK (10.0.14393.0)) seems to have addressed some compiler issues. Unfortunately I've encountered a new error. Apologies for stumbling on what are most likely very simple issues, I really appreciate the assistance!

The latest errors have been attached as windows-build-failure-2017-09-27.log.

Last edited on Oct 27, 2017 at 5:54:24 PM by tjcelaya (previous) (diff)

comment:29 in reply to: ↑ 28 Changed on Oct 27, 2017 at 5:53:47 PM by dkocher

Replying to tjcelaya:

The Visual Studio 2017 installation (w/ Windows 10 SDK (10.0.14393.0)) seems to have addressed some compiler issues. Unfortunately I've encountered a new error. Apologies for stumbling on what are most likely very simple issues, I really appreciate the assistance!

The latest errors have been attached as {{windows-build-failure-2017-09-27.log}}}.

We have updated the build instructions at https://github.com/iterate-ch/cyberduck/blob/master/README.md. Please install Bonjour SDK for Windows or exclude the bonjourmodule from the build.

comment:30 Changed on Oct 29, 2017 at 11:32:39 PM by tjcelaya

Thanks for all the help! It looks like I'm down to one last issue, actually loading the new cyberduckprofile... I've added a new MantaProtocol() with the appropriate using statement at the top of MainController.cs but the profile doesn't actually show up when invoked. Confirmed profiles could be added using Microsoft OneDrive Business.cyberduckprofile. Other steps taken include adding the new profile into profiles/default, including the <Bundled/> property set to true, including an image with the protocol name in ResourcesBundle.resx using Visual Studio 2017 on the top-level .sln project. I know the issue is related to loading the protocol because repurposing a different profile which is confirmed to work and changing its protocol leads to the same behavior.

In search of a commonality between the protocols which did load I tried grepping for protocol names in groups (i.e. grepping individually for each protocol) and combining the lists with comm -12 but alas the only results were the git index and compiled files like .dll, .exe. It would help if I could figure out how to see the logs printed to stdout but I haven't figured out how to do that (or if it's possible) yet.

Basically, how do I make sure the Windows build can use the new protocol?

comment:31 Changed on Oct 30, 2017 at 4:30:21 PM by dkocher

Please move the connection profile to profiles/default and install a snapshot build of the profiles module prior building if you want to have the profile bundled by default.

I have fixed the missing dependency in the feature brach.

comment:32 Changed on Oct 31, 2017 at 9:53:31 PM by tjcelaya

Thanks again for all the assistance! I'm doing a final pass of acceptance testing now that Windows builds are picking up our cyberduckprofile and will be submitting one more PR against feature/TRAC-9711 that should make the manta protocol implementation ready for release.

comment:33 Changed on Nov 1, 2017 at 10:25:01 AM by dkocher

  • Milestone changed from 7.0 to 6.3
  • Owner changed from tjcelaya to dkocher
  • Status changed from new to assigned

comment:34 Changed on Nov 6, 2017 at 8:55:47 AM by dkocher

  • Resolution set to fixed
  • Status changed from assigned to closed

In r43133.

comment:35 follow-ups: Changed on Nov 7, 2017 at 7:44:55 PM by tjcelaya

I'm not sure why this behavior wasn't occurring on the feature/TRAC-9711 branch but it seems like the password entry field is now disabled when trying to log in with a passphrase-protected key. Still bisecting the commits between master and the previous working version to figure out where the bug was introduced so I can provide more detail about the issue.

comment:36 in reply to: ↑ 35 Changed on Nov 7, 2017 at 8:28:27 PM by dkocher

Replying to tjcelaya:

I'm not sure why this behavior wasn't occurring on the feature/TRAC-9711 branch but it seems like the password entry field is now disabled when trying to log in with a passphrase-protected key. Still bisecting the commits between master and the previous working version to figure out where the bug was introduced so I can provide more detail about the issue.

I will have a look. Possibly a regression that also applies to SFTP.

comment:37 in reply to: ↑ 35 Changed on Nov 7, 2017 at 8:33:50 PM by dkocher

Replying to tjcelaya:

I'm not sure why this behavior wasn't occurring on the feature/TRAC-9711 branch but it seems like the password entry field is now disabled when trying to log in with a passphrase-protected key. Still bisecting the commits between master and the previous working version to figure out where the bug was introduced so I can provide more detail about the issue.

In r43273.

comment:38 follow-up: Changed on Nov 7, 2017 at 10:07:08 PM by tjcelaya

Thanks for the quick resolution. I opened a PR to address the fact that the private key file dropdown is enabled and might be confusing (plus a URL encoding issue)

comment:39 in reply to: ↑ 38 Changed on Nov 8, 2017 at 7:34:22 PM by dkocher

Replying to tjcelaya:

Thanks for the quick resolution. I opened a PR to address the fact that the private key file dropdown is enabled and might be confusing (plus a URL encoding issue)

Changed to password prompt for private key passphrase in r43304.

Note: See TracTickets for help on using tickets.
swiss made software