Cyberduck Mountain Duck CLI

Opened on Jul 3, 2018 at 8:15:03 AM

Closed on Feb 26, 2019 at 8:44:10 AM

Last modified on Aug 22, 2019 at 1:33:53 PM

#10385 closed defect (fixed)

Missing and duplicated files in list

Reported by: gavinband Owned by: dkocher
Priority: normal Milestone: 6.9.4
Component: sftp Version: 6.6.2
Severity: major Keywords:
Cc: Architecture: Intel
Platform: macOS 10.13

Description

I'm using Cyberduck v6.6.2 (28219) on Mac OS X 10.13.4 High Sierra. On the sites that I have tried to use Cyberduck with, it fails to list some files and will fail to download them properly. (It also has other strange behaviour, such as failing to list files alphabetically when I sort by filename). The attached screenshot shows an example - here I've sorted by filename, but the files with chr19 strangely appear at the top of the list, and the file Tanzania_GWAS-2.5M_b37_chr0X_aligned.vcf.gzis missing from the view. (It is missing from the whole listing although only the top is shown in the screenshot). The second screenshot shows the output of ls done in sftp` in the same folder, correctly listing the contents of the folder.

Digging a bit further, it appears both programs correctly list 104 files, but Cyberduck has duplicated information for some files. Specifically, the top 4 files in the Cyberduck listing (Tanzania_GWAS-2.5M_b37_chr19_aligned.vcf.gz, Tanzania_GWAS-2.5M_b37_chr19_aligned.vcf.gz.tbi, Tanzania_GWAS-2.5M_b37_chr19.vcf.gz and Tanzania_GWAS-2.5M_b37_chr0X.vcf.gz.tbi) all appear again further down the list. The associated information e.g. file size is also repeated. Presumably these duplicate entries replace those for the missing files.

Attachments (4)

cyberduck listing.png​ (868.0 KB) - added by gavinband on Jul 3, 2018 at 8:17:51 AM.
File listing in Cyberduck
sftp listing.png​ (326.7 KB) - added by gavinband on Jul 3, 2018 at 8:18:09 AM.
File listing in sftp
log.txt​ (881 bytes) - added by gavinband on Jul 10, 2018 at 12:33:04 PM.
Log
2019-02-15 cyberduck grab.png​ (380.7 KB) - added by gavinband on Feb 15, 2019 at 4:41:04 PM.
Screenshot showing issue in v6.9.3

Download all attachments as: .zip

Change History (24)

Changed on Jul 3, 2018 at 8:17:51 AM by gavinband

File listing in Cyberduck

Changed on Jul 3, 2018 at 8:18:09 AM by gavinband

File listing in sftp

comment:1 Changed on Jul 9, 2018 at 7:04:27 PM by dkocher

  • Component changed from core to ftp
  • Owner set to dkocher
  • Status changed from new to assigned

Please post the transcript from the log drawer (⌘-L).

Changed on Jul 10, 2018 at 12:33:04 PM by gavinband

Log

comment:2 Changed on Jul 10, 2018 at 12:33:52 PM by gavinband

I've attached the log drawer output (log.txt) from this sequence of operations: 1: connect to the server 2: navigate to the relevant folder 3: double-click on it.

comment:3 Changed on Jul 10, 2018 at 1:56:42 PM by dkocher

  • Component changed from ftp to sftp

comment:4 Changed on Jul 10, 2018 at 2:14:58 PM by dkocher

  • Summary changed from Missing files in Cyberduck to Missing and duplicated files in list

comment:5 Changed on Feb 14, 2019 at 1:17:48 PM by dkocher

Can you let us know if you still have this bug with the current release.

comment:6 Changed on Feb 15, 2019 at 1:45:55 PM by dkocher

  • Resolution set to worksforme
  • Status changed from assigned to closed

comment:7 Changed on Feb 15, 2019 at 4:40:22 PM by gavinband

  • Resolution worksforme deleted
  • Status changed from closed to reopened

This is still an issue in the updated version of Cyberduck, as shown in the attached screenshot. Best, Gavin.

Changed on Feb 15, 2019 at 4:41:04 PM by gavinband

Screenshot showing issue in v6.9.3

comment:8 Changed on Feb 15, 2019 at 5:35:55 PM by dkocher

I suppose it is not possible to get a temporary account on the server to reproduce the issue?

comment:9 follow-up: Changed on Feb 15, 2019 at 9:21:10 PM by gavinband

I can ask about this - can you send your public key? Thanks, g.

comment:10 in reply to: ↑ 9 Changed on Feb 15, 2019 at 9:42:44 PM by dkocher

Replying to gavinband:

I can ask about this - can you send your public key? Thanks, g. 

-----BEGIN PGP PUBLIC KEY BLOCK-----
mQGiBD0Xeq8RBAD4ckqCbmcOFOFw7JIWBJB9yxPNzYgCkJY9SKgTmmDYQQa04MzQ
WCoQsIzYE0Xcpni/JdQYHqfrgn6bpPMyjFwGCgYgJA0qsUz5PAc01WGuaUHkBy+d
TYoO1z9XwDcEitukYalPErWiEGE0mYssH4j11ZOqviSU1OQF7eTGSOQuZwCgi4Jw
jQKlANMZhm7Gc3g1gAeDcw8EAMzDeT8bTVtnq4z9OBLBZFryMtsfK6ccyuetLXzY
l+Klnp7TU3qdwfkDOFFKWOZYQUkq3/H9KOQQNuYCxEHPgjEqTv1RaWzh7y6avqNC
NhZdY5dAEYm5XjZzkcv2y76WdXdeFVnHYYqxuuZp7lhXQYbQquVgUxGqZ5DyRbRS
iDIAA/4kYypTf1EJSjEgyPKzzPCkbfr/GNzadLt8SoV0rtg2EyJpSKe5Mt67E26d
LLTmnS8FlIS74dN5x9fU73q010ibHzs3bH/bgTqC13tnBSMhoRdvZKCjx74BhiH/
KTRLt56e2pNz9WzbWGS2wfomlL+g5mNC/qjz89KB2VJGsmDNFLQsRGF2aWQgS29j
aGVyIChTd2l0emVybGFuZCkgPGRrb2NoZXJAc3Vkby5jaD6IXwQTEQIAHwIbAwQL
BwMCAxUCAwMWAgECHgECF4AFAkNLAnECGQEACgkQdBH4SdTFvT5TKACfXt/oHGxb
YtEMOZq+aMHZz/EgSdgAnibONVRQAe1inP4e7GWpJC7L50z9iFwEExECABwFAj7N
SccCGwMECwcDAgMVAgMDFgIBAh4BAheAAAoJEHQR+EnUxb0+EDcAnAqlyFXoSULy
vRNVousjKUxxR772AJ92xIijVS1XoyVDly7nvABJOcFDBLQxRGF2aWQgS29jaGVy
IChTd2l0emVybGFuZCkgPGRrb2NoZXJAY3liZXJkdWNrLmNoPohcBBMRAgAcBQI+
zUnkAhsDBAsHAwIDFQIDAxYCAQIeAQIXgAAKCRB0EfhJ1MW9PkHsAJ4uETk/Z5MM
XEyZBIMebNPGhl9/rQCfVj00Ve1/CIkCcaaYAXNPrVdtXeC0L0RhdmlkIEtvY2hl
ciAoU3dpdHplcmxhbmQpIDxka29jaGVyQGl0ZXJhdGUuY2g+iGIEExECACIFAlIB
ZyUCGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJEHQR+EnUxb0+DjoAn0OM
bncv7y/OloIZZ5+LmE/o2c0sAJ9Axb+uei4jksGwSyarZrgyIIL0FbQna2V5YmFz
ZS5pby9ka29jaGVyIDxka29jaGVyQGtleWJhc2UuaW8+iFcEExEKABcFAj0Xeq8C
GwMDCwkHAxUKCAIeAQIXgAAKCRB0EfhJ1MW9Pl+FAJ9OH0fS/JXr9hr5OtGj8qtU
lOMpJwCeIioLBwulISb/QR+mHhU28td3uAa5AQ0EPRd6uRAEAKnRk+3fOic8Z++r
NvxK8zI4slJYsRAv0nywPsArkKRZOqqTIPOTG/YtVFFyBSIFciCZ8nFYUf87Shzz
VOYQvrftrHU8tDhvxejDOuh/Wr1aixZqEXxIgZ1+87P+UEmoH4q/uXiNiIZk6JKr
bVAPstBwBNikbxYi850kLh51pi7fAAMFA/0WiT3ca4LxboMZNGhKvYtQmdUSFwDZ
T2o5C0Vt7wvxYIUynp7lm++yJRsXQHiDVN8SE5hYKE9P6MK7UXt2V7Kn4BHxD532
M0Nzt36HrID0QQ5SPSa2Mz1beE2JWUozOby+dejYtTuN8gmAeAxWr+iqiJa2p14g
XJdgdgW/3qjlJ4hGBBgRAgAGBQI9F3q5AAoJEHQR+EnUxb0+LuQAnj2ZXnImA6nD
LZZJAQrc1Bo1nIQRAJ4+DX+v1pr2iGn3BbnZhsh+j1heWQ==
=w3zG
-----END PGP PUBLIC KEY BLOCK-----
Last edited on Feb 15, 2019 at 9:43:03 PM by dkocher (previous) (diff)

comment:11 follow-up: Changed on Feb 18, 2019 at 3:37:05 PM by gavinband

Hi, do you have an ssh public key for this, rather than a gpg one? Thanks, g.

comment:12 in reply to: ↑ 11 Changed on Feb 23, 2019 at 3:47:27 PM by dkocher

Replying to gavinband:

Hi, do you have an ssh public key for this, rather than a gpg one? Thanks, g. 

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDplFjFCWXZmp9/Dy30JsZdP8QcKBqhjVReiBrBEolpbywdbqqZyHFcVFyyPb92d2+4p6++M0cddMVPQ1MEZjXd8PTpm9XR09wrHq0DrHvkArhI+2lpsZ3wm2DxeGl3mhv8U8jXgcMLjjGF+mDN2t0p58+Qawb7qfWUMjUQN2BhmKCT5fgzKu0dKUtZi7FSfU6U1223pwB2k8iGdYeLftNvEe0aZiTsa695IqjL1rUMRpiTlnn/ip2CiJZlZjem2rnnPZFNsfFQLuLGGDYWboTxz9nHbcsbvAMzckiqpXM1GdojMFSMPV8sS7mjLRhmOs6bG6nrsygNu26O4wdyULoL dkocher@mellifera.local

comment:13 Changed on Feb 25, 2019 at 11:12:12 AM by gavinband

Hi, Thanks - please try now: ​tanzania-gwas@ftp.malariagen.net, accessed by sftp using the ssh key you've sent. The problem can be seen in:

/home/datarelease/data/callsets/genotyped/Tanzania_GWAS-2.5M_b37/master/

As reported the problems are

1: the list of files is incorrect (e.g. files with 'Tanzania_GWAS-2.5M_b37_chr19.vcf.gz' in appear twice in the list, including at the top after sorting by filename), while other files including the file 'Tanzania_GWAS-2.5M_b37_chr0X.vcf.gz' is missing.

2: There is equally a problem when downloading the whole folder of files - I think, but am not sure, that this has to do with trying to download Tanzania_GWAS-2.5M_b37_chr0X.vcf.gz but actually downloading Tanzania_GWAS-2.5M_b37_chr19.vcf.gz. (Sorry I realise this is vague, I can re-download to be more specific about this aspect of the problem, but maybe solving problem #1 will fix this.)

I am using Cyberduck 6.9.3 on a 2014 Macbook Pro running macOS High Sierra (10.13.6).

(Please note this data is restricted access - fine to download/access for testing purposes but please delete afterwards.)

Many thanks for your help with this, Gavin.

comment:14 follow-up: Changed on Feb 25, 2019 at 11:14:36 AM by gavinband

FYI that should be:

tanzania-gwas AT ftp DOT malariagen DOT net .

Best wishes, Gavin.

comment:15 Changed on Feb 25, 2019 at 12:25:21 PM by dkocher

  • Milestone set to 6.9.4

comment:16 Changed on Feb 25, 2019 at 1:28:50 PM by dkocher

The list view (View → as List) is not affected by this bug.

comment:17 Changed on Feb 25, 2019 at 5:12:50 PM by dkocher

We can reproduce the issue which is caused by a hashing collision for 19 and 0X.

comment:18 in reply to: ↑ 14 Changed on Feb 25, 2019 at 8:20:24 PM by dkocher

Replying to gavinband:

FYI that should be:

tanzania-gwas AT ftp DOT malariagen DOT net .

Best wishes, Gavin.

Thanks for providing the test account which was very helpful to find the cause of this bug. We can reproduce the issue and you can revoke access.

comment:19 Changed on Feb 26, 2019 at 8:44:10 AM by yla

  • Resolution set to fixed
  • Status changed from reopened to closed

In r46441. Thanks for reporting this.

comment:20 Changed on Aug 22, 2019 at 1:33:53 PM by dkocher

Revised in r47576.

Note: See TracTickets for help on using tickets.