Opened on Jul 3, 2018 at 8:15:03 AM
Closed on Feb 26, 2019 at 8:44:10 AM
Last modified on Aug 22, 2019 at 1:33:53 PM
#10385 closed defect (fixed)
Missing and duplicated files in list
Reported by: | gavinband | Owned by: | dkocher |
---|---|---|---|
Priority: | normal | Milestone: | 6.9.4 |
Component: | sftp | Version: | 6.6.2 |
Severity: | major | Keywords: | |
Cc: | Architecture: | Intel | |
Platform: | macOS 10.13 |
Description
I'm using Cyberduck v6.6.2 (28219) on Mac OS X 10.13.4 High Sierra. On the sites that I have tried to use Cyberduck with, it fails to list some files and will fail to download them properly. (It also has other strange behaviour, such as failing to list files alphabetically when I sort by filename). The attached screenshot shows an example - here I've sorted by filename, but the files with chr19 strangely appear at the top of the list, and the file Tanzania_GWAS-2.5M_b37_chr0X_aligned.vcf.gzis missing from the view. (It is missing from the whole listing although only the top is shown in the screenshot). The second screenshot shows the output of ls done in sftp` in the same folder, correctly listing the contents of the folder.
Digging a bit further, it appears both programs correctly list 104 files, but Cyberduck has duplicated information for some files. Specifically, the top 4 files in the Cyberduck listing (Tanzania_GWAS-2.5M_b37_chr19_aligned.vcf.gz, Tanzania_GWAS-2.5M_b37_chr19_aligned.vcf.gz.tbi, Tanzania_GWAS-2.5M_b37_chr19.vcf.gz and Tanzania_GWAS-2.5M_b37_chr0X.vcf.gz.tbi) all appear again further down the list. The associated information e.g. file size is also repeated. Presumably these duplicate entries replace those for the missing files.
Attachments (4)
Change History (24)
Changed on Jul 3, 2018 at 8:17:51 AM by gavinband
comment:1 Changed on Jul 9, 2018 at 7:04:27 PM by dkocher
- Component changed from core to ftp
- Owner set to dkocher
- Status changed from new to assigned
Please post the transcript from the log drawer (⌘-L).
comment:2 Changed on Jul 10, 2018 at 12:33:52 PM by gavinband
I've attached the log drawer output (log.txt) from this sequence of operations: 1: connect to the server 2: navigate to the relevant folder 3: double-click on it.
comment:3 Changed on Jul 10, 2018 at 1:56:42 PM by dkocher
- Component changed from ftp to sftp
comment:4 Changed on Jul 10, 2018 at 2:14:58 PM by dkocher
- Summary changed from Missing files in Cyberduck to Missing and duplicated files in list
comment:5 Changed on Feb 14, 2019 at 1:17:48 PM by dkocher
Can you let us know if you still have this bug with the current release.
comment:6 Changed on Feb 15, 2019 at 1:45:55 PM by dkocher
- Resolution set to worksforme
- Status changed from assigned to closed
comment:7 Changed on Feb 15, 2019 at 4:40:22 PM by gavinband
- Resolution worksforme deleted
- Status changed from closed to reopened
This is still an issue in the updated version of Cyberduck, as shown in the attached screenshot. Best, Gavin.
comment:8 Changed on Feb 15, 2019 at 5:35:55 PM by dkocher
I suppose it is not possible to get a temporary account on the server to reproduce the issue?
comment:9 follow-up: ↓ 10 Changed on Feb 15, 2019 at 9:21:10 PM by gavinband
I can ask about this - can you send your public key? Thanks, g.
comment:10 in reply to: ↑ 9 Changed on Feb 15, 2019 at 9:42:44 PM by dkocher
Replying to gavinband:
I can ask about this - can you send your public key? Thanks, g.
-----BEGIN PGP PUBLIC KEY BLOCK----- mQGiBD0Xeq8RBAD4ckqCbmcOFOFw7JIWBJB9yxPNzYgCkJY9SKgTmmDYQQa04MzQ WCoQsIzYE0Xcpni/JdQYHqfrgn6bpPMyjFwGCgYgJA0qsUz5PAc01WGuaUHkBy+d TYoO1z9XwDcEitukYalPErWiEGE0mYssH4j11ZOqviSU1OQF7eTGSOQuZwCgi4Jw jQKlANMZhm7Gc3g1gAeDcw8EAMzDeT8bTVtnq4z9OBLBZFryMtsfK6ccyuetLXzY l+Klnp7TU3qdwfkDOFFKWOZYQUkq3/H9KOQQNuYCxEHPgjEqTv1RaWzh7y6avqNC NhZdY5dAEYm5XjZzkcv2y76WdXdeFVnHYYqxuuZp7lhXQYbQquVgUxGqZ5DyRbRS iDIAA/4kYypTf1EJSjEgyPKzzPCkbfr/GNzadLt8SoV0rtg2EyJpSKe5Mt67E26d LLTmnS8FlIS74dN5x9fU73q010ibHzs3bH/bgTqC13tnBSMhoRdvZKCjx74BhiH/ KTRLt56e2pNz9WzbWGS2wfomlL+g5mNC/qjz89KB2VJGsmDNFLQsRGF2aWQgS29j aGVyIChTd2l0emVybGFuZCkgPGRrb2NoZXJAc3Vkby5jaD6IXwQTEQIAHwIbAwQL BwMCAxUCAwMWAgECHgECF4AFAkNLAnECGQEACgkQdBH4SdTFvT5TKACfXt/oHGxb YtEMOZq+aMHZz/EgSdgAnibONVRQAe1inP4e7GWpJC7L50z9iFwEExECABwFAj7N SccCGwMECwcDAgMVAgMDFgIBAh4BAheAAAoJEHQR+EnUxb0+EDcAnAqlyFXoSULy vRNVousjKUxxR772AJ92xIijVS1XoyVDly7nvABJOcFDBLQxRGF2aWQgS29jaGVy IChTd2l0emVybGFuZCkgPGRrb2NoZXJAY3liZXJkdWNrLmNoPohcBBMRAgAcBQI+ zUnkAhsDBAsHAwIDFQIDAxYCAQIeAQIXgAAKCRB0EfhJ1MW9PkHsAJ4uETk/Z5MM XEyZBIMebNPGhl9/rQCfVj00Ve1/CIkCcaaYAXNPrVdtXeC0L0RhdmlkIEtvY2hl ciAoU3dpdHplcmxhbmQpIDxka29jaGVyQGl0ZXJhdGUuY2g+iGIEExECACIFAlIB ZyUCGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJEHQR+EnUxb0+DjoAn0OM bncv7y/OloIZZ5+LmE/o2c0sAJ9Axb+uei4jksGwSyarZrgyIIL0FbQna2V5YmFz ZS5pby9ka29jaGVyIDxka29jaGVyQGtleWJhc2UuaW8+iFcEExEKABcFAj0Xeq8C GwMDCwkHAxUKCAIeAQIXgAAKCRB0EfhJ1MW9Pl+FAJ9OH0fS/JXr9hr5OtGj8qtU lOMpJwCeIioLBwulISb/QR+mHhU28td3uAa5AQ0EPRd6uRAEAKnRk+3fOic8Z++r NvxK8zI4slJYsRAv0nywPsArkKRZOqqTIPOTG/YtVFFyBSIFciCZ8nFYUf87Shzz VOYQvrftrHU8tDhvxejDOuh/Wr1aixZqEXxIgZ1+87P+UEmoH4q/uXiNiIZk6JKr bVAPstBwBNikbxYi850kLh51pi7fAAMFA/0WiT3ca4LxboMZNGhKvYtQmdUSFwDZ T2o5C0Vt7wvxYIUynp7lm++yJRsXQHiDVN8SE5hYKE9P6MK7UXt2V7Kn4BHxD532 M0Nzt36HrID0QQ5SPSa2Mz1beE2JWUozOby+dejYtTuN8gmAeAxWr+iqiJa2p14g XJdgdgW/3qjlJ4hGBBgRAgAGBQI9F3q5AAoJEHQR+EnUxb0+LuQAnj2ZXnImA6nD LZZJAQrc1Bo1nIQRAJ4+DX+v1pr2iGn3BbnZhsh+j1heWQ== =w3zG -----END PGP PUBLIC KEY BLOCK-----
comment:11 follow-up: ↓ 12 Changed on Feb 18, 2019 at 3:37:05 PM by gavinband
Hi, do you have an ssh public key for this, rather than a gpg one? Thanks, g.
comment:12 in reply to: ↑ 11 Changed on Feb 23, 2019 at 3:47:27 PM by dkocher
Replying to gavinband:
Hi, do you have an ssh public key for this, rather than a gpg one? Thanks, g.
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDplFjFCWXZmp9/Dy30JsZdP8QcKBqhjVReiBrBEolpbywdbqqZyHFcVFyyPb92d2+4p6++M0cddMVPQ1MEZjXd8PTpm9XR09wrHq0DrHvkArhI+2lpsZ3wm2DxeGl3mhv8U8jXgcMLjjGF+mDN2t0p58+Qawb7qfWUMjUQN2BhmKCT5fgzKu0dKUtZi7FSfU6U1223pwB2k8iGdYeLftNvEe0aZiTsa695IqjL1rUMRpiTlnn/ip2CiJZlZjem2rnnPZFNsfFQLuLGGDYWboTxz9nHbcsbvAMzckiqpXM1GdojMFSMPV8sS7mjLRhmOs6bG6nrsygNu26O4wdyULoL dkocher@mellifera.local
comment:13 Changed on Feb 25, 2019 at 11:12:12 AM by gavinband
Hi, Thanks - please try now: tanzania-gwas@…, accessed by sftp using the ssh key you've sent. The problem can be seen in:
/home/datarelease/data/callsets/genotyped/Tanzania_GWAS-2.5M_b37/master/
As reported the problems are
1: the list of files is incorrect (e.g. files with 'Tanzania_GWAS-2.5M_b37_chr19.vcf.gz' in appear twice in the list, including at the top after sorting by filename), while other files including the file 'Tanzania_GWAS-2.5M_b37_chr0X.vcf.gz' is missing.
2: There is equally a problem when downloading the whole folder of files - I think, but am not sure, that this has to do with trying to download Tanzania_GWAS-2.5M_b37_chr0X.vcf.gz but actually downloading Tanzania_GWAS-2.5M_b37_chr19.vcf.gz. (Sorry I realise this is vague, I can re-download to be more specific about this aspect of the problem, but maybe solving problem #1 will fix this.)
I am using Cyberduck 6.9.3 on a 2014 Macbook Pro running macOS High Sierra (10.13.6).
(Please note this data is restricted access - fine to download/access for testing purposes but please delete afterwards.)
Many thanks for your help with this, Gavin.
comment:14 follow-up: ↓ 18 Changed on Feb 25, 2019 at 11:14:36 AM by gavinband
FYI that should be:
tanzania-gwas AT ftp DOT malariagen DOT net .
Best wishes, Gavin.
comment:15 Changed on Feb 25, 2019 at 12:25:21 PM by dkocher
- Milestone set to 6.9.4
comment:16 Changed on Feb 25, 2019 at 1:28:50 PM by dkocher
The list view (View → as List) is not affected by this bug.
comment:17 Changed on Feb 25, 2019 at 5:12:50 PM by dkocher
We can reproduce the issue which is caused by a hashing collision for 19 and 0X.
comment:18 in reply to: ↑ 14 Changed on Feb 25, 2019 at 8:20:24 PM by dkocher
Replying to gavinband:
FYI that should be:
tanzania-gwas AT ftp DOT malariagen DOT net .
Best wishes, Gavin.
Thanks for providing the test account which was very helpful to find the cause of this bug. We can reproduce the issue and you can revoke access.
comment:19 Changed on Feb 26, 2019 at 8:44:10 AM by yla
- Resolution set to fixed
- Status changed from reopened to closed
In r46441. Thanks for reporting this.
comment:20 Changed on Aug 22, 2019 at 1:33:53 PM by dkocher
Revised in r47576.
File listing in Cyberduck