Cyberduck Mountain Duck CLI

#9662 closed enhancement (fixed)

Support SHA256 checksums and use checksum verification

Reported by: carsten.jahn Owned by: dkocher
Priority: normal Milestone: 5.1.3
Component: irods Version: 5.0.11
Severity: normal Keywords:
Cc: Architecture:
Platform:

Description

cyberduck.log says:

WARN ch.cyberduck.core.io.Checksum - Failure to detect algorithm for checksum sha2:47DEQpj8HBSa+/TImW+5JCeuQeRkm5NMpJWZG3hSuFU=

I've seen that core/src/main/java/ch/cyberduck/core/io/Checksum.java guesses different checksum algorithms, and its likely confused by the 'sha2:' prefix it gets from iRODS.

Can you add support for this, please?

Change History (14)

comment:1 Changed on Aug 24, 2016 at 1:41:56 PM by dkocher

It looks like this checksum is additionally Base64 encoded.

comment:2 Changed on Aug 24, 2016 at 1:42:31 PM by dkocher

  • Milestone set to 5.1
  • Owner set to dkocher
  • Status changed from new to assigned

comment:3 Changed on Aug 24, 2016 at 1:52:46 PM by dkocher

  • Summary changed from support iRODS sha256 checksum to Support SHA256 checksums

comment:4 Changed on Aug 24, 2016 at 1:55:26 PM by dkocher

  • Milestone 5.1 deleted

Dependent on upstream issue.

comment:5 Changed on Aug 24, 2016 at 1:55:31 PM by dkocher

  • Type changed from defect to enhancement

comment:6 Changed on Aug 25, 2016 at 5:12:54 AM by carsten.jahn

Thanks for looking into this. I also wondered if the custom checksum comparison implementation in Cyberduck (as it currently exists) is even necessary here. Jargon should compute and transmit a client checksum and have the iRODS server validate it, if this feature is not turned off. Currently it is turned off with setComputeAndVerifyChecksumAfterTransfer in the IRODSSesson class.

BTW, having Jargon handle the client checksumming would have the added benefit of that checksum being stored in the iRODS database for future comparisons (I'm not 100% sure about that, but the C client has a similar behavior).

comment:7 Changed on Aug 26, 2016 at 5:19:38 AM by carsten.jahn

(I should add that the Jargon checksumming does not work if you upload a file with the streaming API. I found out in a similar thread for a different Jargon client: https://github.com/DICE-UNC/irods-cloud-browser/issues/182 )

comment:8 Changed on Sep 7, 2016 at 12:51:10 PM by carsten.jahn

  • Summary changed from Support SHA256 checksums to Support SHA256 checksums and use checksum verification built into Jargon/iRODS protocol

I wanted to add that I tested Cyberduck 5.1.0 upload with iRODS 4.1.9 today in more depth, with Cyberduck debug log enabled.

Cyberduck does this call: org.irods.jargon.core.pub.CollectionAndDataObjectListAndSearchAOImpl - ObjStat ... for retrieving the file size and checksum from iRODS after transfer.

The iRODS/Jargon response will contain a checksum only if an iRODS server rule for calculating the checksum on upload is defined (acPostProcForPut {msiSysChksumDataObj; }) - otherwise, the returned checksum is empty (...dataId=321827, checksum=, ownerName=...). The usage of msiSysChksumDataObj is not necessary if the iRODS client submits a checksum during upload - the client-calculated checksum will be transferred along with the data and the server will check it upon receipt. However, because Cyberduck does setComputeAndVerifyChecksumAfterTransfer(false), the Jargon default behavior (as defined in jargon.properties) is overridden and Jargon does not send a checksum. Thus, when Cyberduck asks for the object checksum, it does not receive any.

The iRODS default rule file does not contain the msiSysChksumDataObj call, but Cyberduck seems to depend on it, as well as on the MD5 checksum algorithm setting, which is SHA256 on modern iRODS servers. Switching back to MD5 and including the msiSysChksumDataObj rule, I can verify that Cyberduck does not complain about the checksum any more.

There is another dependency on server configuration built in: the ObjStat information contains an empty iRODS checksum even if msiSysChksumDataObj is present, in the case that Cyberduck writes to a 'compound' iRODS resource. This may be an iRODS or Jargon bug.

I think the ideal implementation would be to remove setComputeAndVerifyChecksumAfterTransfer(false) and setComputeChecksumAfterTransfer(false), so that the default transfer.computeandvalidate.checksum=true from jargon.properties is in effect. Do not attempt to parse iRODS checksums and compare it to a self-computed checksum. Rather, have Jargon handle this internally. Thus, Cyberduck would automatically support all known iRODS checksum algorithms and wouldn't depend on a particular server-side rule being present.

Maybe the Jargon verify feature was turned off for performance reasons - then it would be good to have this configurable for people who would like to use the Jargon default, maybe as a "hidden option" somewhere.

comment:9 Changed on Sep 7, 2016 at 1:07:26 PM by dkocher

  • Milestone set to 5.2
  • Summary changed from Support SHA256 checksums and use checksum verification built into Jargon/iRODS protocol to Support SHA256 checksums and use checksum verification

comment:10 Changed on Sep 8, 2016 at 7:23:17 AM by dkocher

In r21443.

Last edited on Sep 8, 2016 at 7:23:33 AM by dkocher (previous) (diff)

comment:12 Changed on Sep 8, 2016 at 7:23:36 AM by dkocher

  • Resolution set to fixed
  • Status changed from assigned to closed

comment:13 Changed on Sep 8, 2016 at 8:11:58 AM by carsten.jahn

I've had a look at the changes - looks good in general, but I wanted to point out that there are still two types of checksums that iRODS may return, depending on iRODS server configuration and/or how the client uploaded a file.

If you want to work with the checksum obtained from iRODS, the checksum string may

  • start with "sha2:" - then remove the prefix and treat it as base64 encoded with SHA256 algorithm (looks like the prefix is currently not removed)
  • does not start with "sha2:" - then treat it as base16 encoded with MD5 algorithm (just as the earlier implementation did)

Supporting both cases would cover many possible iRODS setups (IMHO).

comment:14 Changed on Oct 5, 2016 at 1:39:23 PM by dkocher

  • Milestone changed from 5.2 to 5.1.3

Milestone renamed

Note: See TracTickets for help on using tickets.
swiss made software