Missing and duplicated files in list

[gavinband]

I'm using Cyberduck v6.6.2 (28219) on Mac OS X 10.13.4 High Sierra. On the sites that I have tried to use Cyberduck with, it fails to list some files and will fail to download them properly. (It also has other strange behaviour, such as failing to list files alphabetically when I sort by filename). The attached screenshot shows an example - here I've sorted by filename, but the files with chr19 strangely appear at the top of the list, and the file Tanzania_GWAS-2.5M_b37_chr0X_aligned.vcf.gzis missing from the view. (It is missing from the whole listing although only the top is shown in the screenshot). The second screenshot shows the output of ls done in sftp` in the same folder, correctly listing the contents of the folder.

Digging a bit further, it appears both programs correctly list 104 files, but Cyberduck has duplicated information for some files. Specifically, the top 4 files in the Cyberduck listing (Tanzania_GWAS-2.5M_b37_chr19_aligned.vcf.gz, Tanzania_GWAS-2.5M_b37_chr19_aligned.vcf.gz.tbi, Tanzania_GWAS-2.5M_b37_chr19.vcf.gz and Tanzania_GWAS-2.5M_b37_chr0X.vcf.gz.tbi) all appear again further down the list. The associated information e.g. file size is also repeated. Presumably these duplicate entries replace those for the missing files.

[gavinband]

File listing in Cyberduck

[gavinband]

File listing in sftp

[dkocher]

Please post the transcript from the log drawer (⌘-L).

[gavinband]

Log

[gavinband]

I've attached the log drawer output (log.txt) from this sequence of operations: 1: connect to the server 2: navigate to the relevant folder 3: double-click on it.

[dkocher]

Can you let us know if you still have this bug with the current release.

[gavinband]

This is still an issue in the updated version of Cyberduck, as shown in the attached screenshot. Best, Gavin.

[gavinband]

Screenshot showing issue in v6.9.3

[dkocher]

I suppose it is not possible to get a temporary account on the server to reproduce the issue?

[gavinband]

I can ask about this - can you send your public key? Thanks, g.

[dkocher]

Replying to gavinband:

I can ask about this - can you send your public key? Thanks, g.

-----BEGIN PGP PUBLIC KEY BLOCK-----
mQGiBD0Xeq8RBAD4ckqCbmcOFOFw7JIWBJB9yxPNzYgCkJY9SKgTmmDYQQa04MzQ
WCoQsIzYE0Xcpni/JdQYHqfrgn6bpPMyjFwGCgYgJA0qsUz5PAc01WGuaUHkBy+d
TYoO1z9XwDcEitukYalPErWiEGE0mYssH4j11ZOqviSU1OQF7eTGSOQuZwCgi4Jw
jQKlANMZhm7Gc3g1gAeDcw8EAMzDeT8bTVtnq4z9OBLBZFryMtsfK6ccyuetLXzY
l+Klnp7TU3qdwfkDOFFKWOZYQUkq3/H9KOQQNuYCxEHPgjEqTv1RaWzh7y6avqNC
NhZdY5dAEYm5XjZzkcv2y76WdXdeFVnHYYqxuuZp7lhXQYbQquVgUxGqZ5DyRbRS
iDIAA/4kYypTf1EJSjEgyPKzzPCkbfr/GNzadLt8SoV0rtg2EyJpSKe5Mt67E26d
LLTmnS8FlIS74dN5x9fU73q010ibHzs3bH/bgTqC13tnBSMhoRdvZKCjx74BhiH/
KTRLt56e2pNz9WzbWGS2wfomlL+g5mNC/qjz89KB2VJGsmDNFLQsRGF2aWQgS29j
aGVyIChTd2l0emVybGFuZCkgPGRrb2NoZXJAc3Vkby5jaD6IXwQTEQIAHwIbAwQL
BwMCAxUCAwMWAgECHgECF4AFAkNLAnECGQEACgkQdBH4SdTFvT5TKACfXt/oHGxb
YtEMOZq+aMHZz/EgSdgAnibONVRQAe1inP4e7GWpJC7L50z9iFwEExECABwFAj7N
SccCGwMECwcDAgMVAgMDFgIBAh4BAheAAAoJEHQR+EnUxb0+EDcAnAqlyFXoSULy
vRNVousjKUxxR772AJ92xIijVS1XoyVDly7nvABJOcFDBLQxRGF2aWQgS29jaGVy
IChTd2l0emVybGFuZCkgPGRrb2NoZXJAY3liZXJkdWNrLmNoPohcBBMRAgAcBQI+
zUnkAhsDBAsHAwIDFQIDAxYCAQIeAQIXgAAKCRB0EfhJ1MW9PkHsAJ4uETk/Z5MM
XEyZBIMebNPGhl9/rQCfVj00Ve1/CIkCcaaYAXNPrVdtXeC0L0RhdmlkIEtvY2hl
ciAoU3dpdHplcmxhbmQpIDxka29jaGVyQGl0ZXJhdGUuY2g+iGIEExECACIFAlIB
ZyUCGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJEHQR+EnUxb0+DjoAn0OM
bncv7y/OloIZZ5+LmE/o2c0sAJ9Axb+uei4jksGwSyarZrgyIIL0FbQna2V5YmFz
ZS5pby9ka29jaGVyIDxka29jaGVyQGtleWJhc2UuaW8+iFcEExEKABcFAj0Xeq8C
GwMDCwkHAxUKCAIeAQIXgAAKCRB0EfhJ1MW9Pl+FAJ9OH0fS/JXr9hr5OtGj8qtU
lOMpJwCeIioLBwulISb/QR+mHhU28td3uAa5AQ0EPRd6uRAEAKnRk+3fOic8Z++r
NvxK8zI4slJYsRAv0nywPsArkKRZOqqTIPOTG/YtVFFyBSIFciCZ8nFYUf87Shzz
VOYQvrftrHU8tDhvxejDOuh/Wr1aixZqEXxIgZ1+87P+UEmoH4q/uXiNiIZk6JKr
bVAPstBwBNikbxYi850kLh51pi7fAAMFA/0WiT3ca4LxboMZNGhKvYtQmdUSFwDZ
T2o5C0Vt7wvxYIUynp7lm++yJRsXQHiDVN8SE5hYKE9P6MK7UXt2V7Kn4BHxD532
M0Nzt36HrID0QQ5SPSa2Mz1beE2JWUozOby+dejYtTuN8gmAeAxWr+iqiJa2p14g
XJdgdgW/3qjlJ4hGBBgRAgAGBQI9F3q5AAoJEHQR+EnUxb0+LuQAnj2ZXnImA6nD
LZZJAQrc1Bo1nIQRAJ4+DX+v1pr2iGn3BbnZhsh+j1heWQ==
=w3zG
-----END PGP PUBLIC KEY BLOCK-----

[gavinband]

Hi, do you have an ssh public key for this, rather than a gpg one? Thanks, g.

[dkocher]

Replying to gavinband:

Hi, do you have an ssh public key for this, rather than a gpg one? Thanks, g.

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDplFjFCWXZmp9/Dy30JsZdP8QcKBqhjVReiBrBEolpbywdbqqZyHFcVFyyPb92d2+4p6++M0cddMVPQ1MEZjXd8PTpm9XR09wrHq0DrHvkArhI+2lpsZ3wm2DxeGl3mhv8U8jXgcMLjjGF+mDN2t0p58+Qawb7qfWUMjUQN2BhmKCT5fgzKu0dKUtZi7FSfU6U1223pwB2k8iGdYeLftNvEe0aZiTsa695IqjL1rUMRpiTlnn/ip2CiJZlZjem2rnnPZFNsfFQLuLGGDYWboTxz9nHbcsbvAMzckiqpXM1GdojMFSMPV8sS7mjLRhmOs6bG6nrsygNu26O4wdyULoL dkocher@mellifera.local

[gavinband]

Hi, Thanks - please try now: tanzania-gwas@…, accessed by sftp using the ssh key you've sent. The problem can be seen in:

/home/datarelease/data/callsets/genotyped/Tanzania_GWAS-2.5M_b37/master/

As reported the problems are

1: the list of files is incorrect (e.g. files with 'Tanzania_GWAS-2.5M_b37_chr19.vcf.gz' in appear twice in the list, including at the top after sorting by filename), while other files including the file 'Tanzania_GWAS-2.5M_b37_chr0X.vcf.gz' is missing.

2: There is equally a problem when downloading the whole folder of files - I think, but am not sure, that this has to do with trying to download Tanzania_GWAS-2.5M_b37_chr0X.vcf.gz but actually downloading Tanzania_GWAS-2.5M_b37_chr19.vcf.gz. (Sorry I realise this is vague, I can re-download to be more specific about this aspect of the problem, but maybe solving problem #1 will fix this.)

I am using Cyberduck 6.9.3 on a 2014 Macbook Pro running macOS High Sierra (10.13.6).

(Please note this data is restricted access - fine to download/access for testing purposes but please delete afterwards.)

Many thanks for your help with this, Gavin.

[gavinband]

FYI that should be:

tanzania-gwas AT ftp DOT malariagen DOT net .

Best wishes, Gavin.

[dkocher]

The list view (View → as List) is not affected by this bug.

[dkocher]

We can reproduce the issue which is caused by a hashing collision for 19 and 0X.

[dkocher]

Replying to gavinband:

FYI that should be:

tanzania-gwas AT ftp DOT malariagen DOT net .

Best wishes, Gavin.

Thanks for providing the test account which was very helpful to find the cause of this bug. We can reproduce the issue and you can revoke access.

[yla]

In r46441. Thanks for reporting this.

[dkocher]

Revised in r47576.