Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Manta Object Store #9711

Closed
cyberduck opened this issue Sep 24, 2016 · 34 comments
Closed

Support Manta Object Store #9711

cyberduck opened this issue Sep 24, 2016 · 34 comments

Comments

@cyberduck
Copy link
Collaborator

f6042bc created the issue

Manta is an open source object store like S3 or Swift. It is widely in use and will be quite ubiquitous now that Samsung has purchased Joyent (the creators of Manta).

This ticket is to request that support for Manta be added to CyberDuck.

There is a Java SDK available for Manta.


Attachments

@cyberduck
Copy link
Collaborator Author

@dkocher commented

This is certainly possible but we would need some more users to vote this up to justify the effort.

@cyberduck
Copy link
Collaborator Author

3d620ce commented

Hey there, Tomas from Joyent here, I've been using Cyberduck for years now and want to contribute to the project. I'm researching what it would take to implement this and had a look at ticket #9799 along with the associated changeset which added support for Microsoft OneDrive. After reviewing the changes and getting project building locally, I'm interested in implementing a storage provider for Manta within Cyberduck. Following the example of 71fd632, it looks like the steps for adding a new provider is as follows:

  • Create a new module in the the top-level cyberduck project and add it to the root project pom.xml
  • extend AbstractProtocol and register the new protocol in ch.cyberduck.cli.Terminal and ch.cyberduck.ui.cocoa.MainApplication
  • add defaults to ch.cyberduck.core.preferences.Preferences
  • implement interfaces under ch.cyberduck.core.features.* within the new module
  • create a cyberduckprofile describing the connection information required (server, manta user/subuser, manta key id) and the field labels
  • potentially, add new params to AbstractProtocol to reflect the usage of an SSH Key Fingerprint instead of a user password or client secret

Please let me know if I've missed any critical details or there are any points in need of clarfication. The divergence in terminology for the authentication secret stems from the fact that authentication for Manta relies on a combination of username and an SSH Key fingerprint for a key that has previously been uploaded to the Manta instance in use (see Setting Up Your Environment in the Manta documentation for details). Since Cyberduck provides a dropdown field for supplying an SSH Private Key in addition to text input for passwords there is some ambiguity regarding the best approach to take, please see Question 1 for some proposals about how to handle this.

Questions:

  1. The Open Connection dialog includes fields for both Password/AuthCode and an SSH private key. This gives us some options for authenticating users (since the labels are configurable, Password will refer to the text input and SSH Private Key refers to the dropdown that is enabled when the SFTP protocol is chosen):
    1. disabling the Password field and expecting users to select a SSH Private Key. This seems to be the most intuitive to handle key selection.
    2. disabling the SSH Private Key field and expecting users to supply the key fingerprint in the Password field, similarly to the way the Manta documentation asks users to set the MANTA_KEY_ID environment variable. This aligns with Manta CLI configuration but is not user-friendly.
  2. Sonar is mentioned on the main page of the issue tracker but is inaccessible through https://sonar.cyberduck.io/ and I couldn't find any style-checking plugin in the project, is there a different code style tool in use?
  3. The modifications in https://trac.cyberduck.io/changeset/38915/trunk/Cyberduck.xcodeproj/project.pbxproj refer to resources being added to the Xcode Project. Are there any special considerations around adding to this file or would I be able to make this change myself?
  4. The java-manta library allows for optional Client-Side Encryption while the Cyberduck roadmap indicates support for Cryptomator is being added. Handling encryption through Manta's client library could lead to issues and confusion, in addition to requiring us to either collect more configuration options or force certain settings from Cyberduck. How the Manta provider in Cyberduck will support encryption is still an open question I'd like to discuss further once these questions are ironed out.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Fixed the sonar link to point to (https://sonar.iterate.ch/).

@cyberduck
Copy link
Collaborator Author

@dkocher commented

The codestyle is part of the project in the .idea directory and picked up automatically when opening the project. Obviously this only works when working with IntelliJ IDEA which is our preferred IDE.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

I would suggest to move the client-side encryption support in Manta to a different ticket. Our Cryptomator integration is now finished and will be part of 6.0 available later today.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

You will need XCode installed to make changes to the project.pbxproj project file. Adding resources there is not a priority and can be done at the very last to include resources by default. You can install your new connection profile by double clicking – see Profiles.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

For the ambiguity for the SSH key fingerprint required you will want to at least override getPasswordPlaceholder from the Protocol interface. This will adjust the label and placeholder text when editing the bookmark. Or you can disable the password field with isPasswordConfigurable. Currently the private key popup is only enabled for protocols of type sftp (see getType in Protocol).

@cyberduck
Copy link
Collaborator Author

@dkocher commented

You analysis about what is to implement is right. Don't hesitate to ping me for any further clarifications.

@cyberduck
Copy link
Collaborator Author

3d620ce commented

Work on this is progressing though I've got two questions thus far:

When attempting to build and run the project in Xcode (using Cyberduck.xcodeproj to open the project) the application crashes immediately with the error pasted below. I haven't had a chance to debug it further but noticed there was no output in the Xcode console and am not familiar with logging in Objective-C. Any suggestions?

Error loading runtime bundle.
The application failed to launch with a unrecoverable exception.

When attempting to turn on debugging using the debug.sh script I get an error about paths, though there is a file at that path it's missing the .debug suffix:

  [cyberduck](master) PDT 17:13 UTC 00:13
./debug.sh -e
Enabling debug configuration...
/Users/tomascelaya/repo/cyberduck
cp: /Users/tomascelaya/repo/cyberduck/setup/app/Info.plist.debug: No such file or directory

Not sure how to resolve the first issue further without getting past the second.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Create a debug configuration Info.plist.debug yourself by duplicating the Info.plist file and prepend -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5005 to the value for VMOptions. I will attach a sample.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

I never run the project from XCode – the build in XCode is partial and only covers the native libraries. You will have to run mvn package -DskipTests -pl osx -am.

You can then execute the Cyberduck.app binary in osx/target or run with the i18n.sh script using i18n/src/main/resources/i18n.sh -l en.lproj -r.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

We are now mirroring to https://github.com/iterate-ch/cyberduck to make contributions easier.

@cyberduck
Copy link
Collaborator Author

3d620ce commented

Thanks for all the help so far! Definitely getting closer sincemvn package succeeds without any errors. Running the bundle at osx/target/Cyberduck.app closes as soon as it's launched though and invoking open osx/target/Cyberduck.app yields the following error:

LSOpenURLsWithRole() failed with error -10810 for the file /Users/tomascelaya/repo/cyberduck/osx/target/Cyberduck.app.

Thankfully i18n.sh yield a more obvious error message:

  [cyberduck](master) PDT 14:39 UTC 21:39
i18n/src/main/resources/i18n.sh -l en.lproj -r
Using Language:en.lproj
Running app using en...
Error: Could not find or load main class ch.cyberduck.ui.cocoa.MainApplication
*** DONE. ***

The error specifically occurs on the last line in run() but attempting to run the Cyberduck.app/Contents/Cyberduck binary prints out the same message, so it seems like I'm missing something related to classpath configuration. I'm attempting to track down the issue further by inspecting file open syscalls but haven't found anything promising yet.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Please try running the install phase with mvn install -DskipTests -pl osx -am.

@cyberduck
Copy link
Collaborator Author

3d620ce commented

That did the trick! A warning came up about "App Transport Security" but everything seems to work and the manta endpoints are all https so I don't think it'll cause any issues.

Thanks!

Replying to [comment:17 dkocher]:

Please try running the install phase with mvn install -DskipTests -pl osx -am.

@cyberduck
Copy link
Collaborator Author

3d620ce commented

I'm nearing completion and will be ready to submit my changes as soon a I resolve an issue with Manta subusers, would a pull request on Github work or is there some other process for submitting contributions?

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Replying to [comment:19 tjcelaya]:

I'm nearing completion and will be ready to submit my changes as soon a I resolve an issue with Manta subusers, would a pull request on Github work or is there some other process for submitting contributions?

A pull request on GitHub works just great.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Now in feature/TRAC-9711.

@cyberduck
Copy link
Collaborator Author

3d620ce commented

java-manta 3.7.1 3.1.7 (or 3.8.0 3.2.0) is undergoing performance testing and will be released very soon. In the meantime I wanted to try building Cyberduck on Windows and have been running into an issue with Maven/Ant/MSBuild. I've installed the following software:

  • Java 1.8u152 JDK
  • Maven 3.5.2
  • Nuget 4.3.0.4406
  • .NET Framework 4.7.1
    I've included the full Maven output when running mvn install -DskipTests -am -pl windows since I'm not sure if this is caused by a separate module from the one that's failing. The machine in use is a Windows Server 2016 Standard VM running on Joyent's Public Cloud:
    [ moved to file ]

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Please try the branch (https://github.com/iterate-ch/cyberduck/tree/feature/TRAC-9711) and install the build tasks from (https://github.com/loresoft/msbuildtasks).

@cyberduck
Copy link
Collaborator Author

3d620ce commented

Thanks, I installed the latest msbuildtasks MSI and am getting further into the build but have encountered some confusing errors. I've attached them as windows-build-failure-2017-09-26.log.

Which version of the .NET Framework and Windows SDK is used for builds? I discovered that I needed to install the Windows 8 SDK (https://developer.microsoft.com/en-us/windows/downloads/windows-8-sdk) based on the error message that occurred after adding msbuildtasks but the new errors seem to be related to compiler versions? In order to install as few conflicting version of .NET as possible I am using the MSBuild executable from the .NET Framework installed by Windows 8 SDK ( added C:\Windows\Microsoft.NET\Framework\v4.0.30319 to my PATH since it contains MSBuild.exe). These errors are the same on both Windows Server 2012 and Windows Server 2016.

@cyberduck
Copy link
Collaborator Author

@AliveDevil commented

Using MSBuild from C:\Windows\Microsoft.NET\Framework\v4.0.30319 will not work as we are using features not present in this version.

Please install Visual Studio 2017 from https://www.visualstudio.com/en/vs/ and include following "additional items" upon installation.
"Workloads" required to build Cyberduck for Windows:

  • .NET Desktop development

Additional items required to build Cyberduck for Windows or Cyberduck for Windows Store:

  • Windows 10 SDK (10.0.14393.0)

For more information about workloads and additional items in visual studios new installer go to https://docs.microsoft.com/en-us/visualstudio/install/modify-visual-studio

You will, however, get additional errors after building as we create an installer just after building. You can ignore those errors.

@cyberduck
Copy link
Collaborator Author

3d620ce commented

The Visual Studio 2017 installation (w/ Windows 10 SDK (10.0.14393.0)) seems to have addressed some compiler issues. Unfortunately I've encountered a new error. Apologies for stumbling on what are most likely very simple issues, I really appreciate the assistance!

The latest errors have been attached as windows-build-failure-2017-09-27.log.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Replying to [comment:28 tjcelaya]:

The Visual Studio 2017 installation (w/ Windows 10 SDK (10.0.14393.0)) seems to have addressed some compiler issues. Unfortunately I've encountered a new error. Apologies for stumbling on what are most likely very simple issues, I really appreciate the assistance!

The latest errors have been attached as {{windows-build-failure-2017-09-27.log}}}.

We have updated the build instructions at (https://github.com/iterate-ch/cyberduck/blob/master/README.md). Please install Bonjour SDK for Windows or exclude the bonjourmodule from the build.

@cyberduck
Copy link
Collaborator Author

3d620ce commented

Thanks for all the help! It looks like I'm down to one last issue, actually loading the new cyberduckprofile... I've added a new MantaProtocol() with the appropriate using statement at the top of MainController.cs but the profile doesn't actually show up when invoked. Confirmed profiles could be added using Microsoft OneDrive Business.cyberduckprofile. Other steps taken include adding the new profile into profiles/default, including the <Bundled/> property set to true, including an image with the protocol name in ResourcesBundle.resx using Visual Studio 2017 on the top-level .sln project. I know the issue is related to loading the protocol because repurposing a different profile which is confirmed to work and changing its protocol leads to the same behavior.

In search of a commonality between the protocols which did load I tried grepping for protocol names in groups (i.e. grepping individually for each protocol) and combining the lists with comm -12 but alas the only results were the git index and compiled files like .dll, .exe. It would help if I could figure out how to see the logs printed to stdout but I haven't figured out how to do that (or if it's possible) yet.

Basically, how do I make sure the Windows build can use the new protocol?

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Please move the connection profile to profiles/default and install a snapshot build of the profiles module prior building if you want to have the profile bundled by default.

I have fixed the missing dependency in the feature brach.

@cyberduck
Copy link
Collaborator Author

3d620ce commented

Thanks again for all the assistance! I'm doing a final pass of acceptance testing now that Windows builds are picking up our cyberduckprofile and will be submitting one more PR against feature/TRAC-9711 that should make the manta protocol implementation ready for release.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

In d52c5d8.

@cyberduck
Copy link
Collaborator Author

3d620ce commented

I'm not sure why this behavior wasn't occurring on the feature/TRAC-9711 branch but it seems like the password entry field is now disabled when trying to log in with a passphrase-protected key. Still bisecting the commits between master and the previous working version to figure out where the bug was introduced so I can provide more detail about the issue.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Replying to [comment:35 tjcelaya]:

I'm not sure why this behavior wasn't occurring on the feature/TRAC-9711 branch but it seems like the password entry field is now disabled when trying to log in with a passphrase-protected key. Still bisecting the commits between master and the previous working version to figure out where the bug was introduced so I can provide more detail about the issue.
I will have a look. Possibly a regression that also applies to SFTP.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Replying to [comment:35 tjcelaya]:

I'm not sure why this behavior wasn't occurring on the feature/TRAC-9711 branch but it seems like the password entry field is now disabled when trying to log in with a passphrase-protected key. Still bisecting the commits between master and the previous working version to figure out where the bug was introduced so I can provide more detail about the issue.
In 4f8182b.

@cyberduck
Copy link
Collaborator Author

3d620ce commented

Thanks for the quick resolution. I opened a PR to address the fact that the private key file dropdown is enabled and might be confusing (plus a URL encoding issue)

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Replying to [comment:38 tjcelaya]:

Thanks for the quick resolution. I opened a PR to address the fact that the private key file dropdown is enabled and might be confusing (plus a URL encoding issue)

Changed to password prompt for private key passphrase in 6a15da9.

@iterate-ch iterate-ch locked as resolved and limited conversation to collaborators Nov 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

2 participants