Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing and duplicated files in list #10385

Closed
cyberduck opened this issue Jul 3, 2018 · 16 comments
Closed

Missing and duplicated files in list #10385

cyberduck opened this issue Jul 3, 2018 · 16 comments
Assignees
Labels
bug fixed sftp SFTP Protocol Implementation
Milestone

Comments

@cyberduck
Copy link
Collaborator

528c4d9 created the issue

I'm using Cyberduck v6.6.2 (28219) on Mac OS X 10.13.4 High Sierra.
On the sites that I have tried to use Cyberduck with, it fails to list some files and will fail to download them properly.
(It also has other strange behaviour, such as failing to list files alphabetically when I sort by filename).
The attached screenshot shows an example - here I've sorted by filename, but the files with chr19 strangely appear at the top of the list, and the file Tanzania_GWAS-2.5M_b37_chr0X_aligned.vcf.gzis missing from the view. (It is missing from the whole listing although only the top is shown in the screenshot). The second screenshot shows the output of ls done in sftp` in the same folder, correctly listing the contents of the folder.

Digging a bit further, it appears both programs correctly list 104 files, but Cyberduck has duplicated information for some files. Specifically, the top 4 files in the Cyberduck listing (Tanzania_GWAS-2.5M_b37_chr19_aligned.vcf.gz, Tanzania_GWAS-2.5M_b37_chr19_aligned.vcf.gz.tbi, Tanzania_GWAS-2.5M_b37_chr19.vcf.gz and Tanzania_GWAS-2.5M_b37_chr0X.vcf.gz.tbi) all appear again further down the list. The associated information e.g. file size is also repeated. Presumably these duplicate entries replace those for the missing files.


Attachments

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Please post the transcript from the log drawer (⌘-L).

@cyberduck
Copy link
Collaborator Author

528c4d9 commented

I've attached the log drawer output (log.txt) from this sequence of operations:
1: connect to the server
2: navigate to the relevant folder
3: double-click on it.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Can you let us know if you still have this bug with the current release.

@cyberduck
Copy link
Collaborator Author

528c4d9 commented

This is still an issue in the updated version of Cyberduck, as shown in the attached screenshot.
Best,
Gavin.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

I suppose it is not possible to get a temporary account on the server to reproduce the issue?

@cyberduck
Copy link
Collaborator Author

528c4d9 commented

I can ask about this - can you send your public key?
Thanks,
g.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Replying to [comment:9 gavinband]:

I can ask about this - can you send your public key?
Thanks,
g.

-----BEGIN PGP PUBLIC KEY BLOCK-----
mQGiBD0Xeq8RBAD4ckqCbmcOFOFw7JIWBJB9yxPNzYgCkJY9SKgTmmDYQQa04MzQ
WCoQsIzYE0Xcpni/JdQYHqfrgn6bpPMyjFwGCgYgJA0qsUz5PAc01WGuaUHkBy+d
TYoO1z9XwDcEitukYalPErWiEGE0mYssH4j11ZOqviSU1OQF7eTGSOQuZwCgi4Jw
jQKlANMZhm7Gc3g1gAeDcw8EAMzDeT8bTVtnq4z9OBLBZFryMtsfK6ccyuetLXzY
l+Klnp7TU3qdwfkDOFFKWOZYQUkq3/H9KOQQNuYCxEHPgjEqTv1RaWzh7y6avqNC
NhZdY5dAEYm5XjZzkcv2y76WdXdeFVnHYYqxuuZp7lhXQYbQquVgUxGqZ5DyRbRS
iDIAA/4kYypTf1EJSjEgyPKzzPCkbfr/GNzadLt8SoV0rtg2EyJpSKe5Mt67E26d
LLTmnS8FlIS74dN5x9fU73q010ibHzs3bH/bgTqC13tnBSMhoRdvZKCjx74BhiH/
KTRLt56e2pNz9WzbWGS2wfomlL+g5mNC/qjz89KB2VJGsmDNFLQsRGF2aWQgS29j
aGVyIChTd2l0emVybGFuZCkgPGRrb2NoZXJAc3Vkby5jaD6IXwQTEQIAHwIbAwQL
BwMCAxUCAwMWAgECHgECF4AFAkNLAnECGQEACgkQdBH4SdTFvT5TKACfXt/oHGxb
YtEMOZq+aMHZz/EgSdgAnibONVRQAe1inP4e7GWpJC7L50z9iFwEExECABwFAj7N
SccCGwMECwcDAgMVAgMDFgIBAh4BAheAAAoJEHQR+EnUxb0+EDcAnAqlyFXoSULy
vRNVousjKUxxR772AJ92xIijVS1XoyVDly7nvABJOcFDBLQxRGF2aWQgS29jaGVy
IChTd2l0emVybGFuZCkgPGRrb2NoZXJAY3liZXJkdWNrLmNoPohcBBMRAgAcBQI+
zUnkAhsDBAsHAwIDFQIDAxYCAQIeAQIXgAAKCRB0EfhJ1MW9PkHsAJ4uETk/Z5MM
XEyZBIMebNPGhl9/rQCfVj00Ve1/CIkCcaaYAXNPrVdtXeC0L0RhdmlkIEtvY2hl
ciAoU3dpdHplcmxhbmQpIDxka29jaGVyQGl0ZXJhdGUuY2g+iGIEExECACIFAlIB
ZyUCGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJEHQR+EnUxb0+DjoAn0OM
bncv7y/OloIZZ5+LmE/o2c0sAJ9Axb+uei4jksGwSyarZrgyIIL0FbQna2V5YmFz
ZS5pby9ka29jaGVyIDxka29jaGVyQGtleWJhc2UuaW8+iFcEExEKABcFAj0Xeq8C
GwMDCwkHAxUKCAIeAQIXgAAKCRB0EfhJ1MW9Pl+FAJ9OH0fS/JXr9hr5OtGj8qtU
lOMpJwCeIioLBwulISb/QR+mHhU28td3uAa5AQ0EPRd6uRAEAKnRk+3fOic8Z++r
NvxK8zI4slJYsRAv0nywPsArkKRZOqqTIPOTG/YtVFFyBSIFciCZ8nFYUf87Shzz
VOYQvrftrHU8tDhvxejDOuh/Wr1aixZqEXxIgZ1+87P+UEmoH4q/uXiNiIZk6JKr
bVAPstBwBNikbxYi850kLh51pi7fAAMFA/0WiT3ca4LxboMZNGhKvYtQmdUSFwDZ
T2o5C0Vt7wvxYIUynp7lm++yJRsXQHiDVN8SE5hYKE9P6MK7UXt2V7Kn4BHxD532
M0Nzt36HrID0QQ5SPSa2Mz1beE2JWUozOby+dejYtTuN8gmAeAxWr+iqiJa2p14g
XJdgdgW/3qjlJ4hGBBgRAgAGBQI9F3q5AAoJEHQR+EnUxb0+LuQAnj2ZXnImA6nD
LZZJAQrc1Bo1nIQRAJ4+DX+v1pr2iGn3BbnZhsh+j1heWQ==
=w3zG
-----END PGP PUBLIC KEY BLOCK-----

@cyberduck
Copy link
Collaborator Author

528c4d9 commented

Hi, do you have an ssh public key for this, rather than a gpg one?
Thanks,
g.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Replying to [comment:11 gavinband]:

Hi, do you have an ssh public key for this, rather than a gpg one?
Thanks,
g.

ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDplFjFCWXZmp9/Dy30JsZdP8QcKBqhjVReiBrBEolpbywdbqqZyHFcVFyyPb92d2+4p6++M0cddMVPQ1MEZjXd8PTpm9XR09wrHq0DrHvkArhI+2lpsZ3wm2DxeGl3mhv8U8jXgcMLjjGF+mDN2t0p58+Qawb7qfWUMjUQN2BhmKCT5fgzKu0dKUtZi7FSfU6U1223pwB2k8iGdYeLftNvEe0aZiTsa695IqjL1rUMRpiTlnn/ip2CiJZlZjem2rnnPZFNsfFQLuLGGDYWboTxz9nHbcsbvAMzckiqpXM1GdojMFSMPV8sS7mjLRhmOs6bG6nrsygNu26O4wdyULoL dkocher@mellifera.local

@cyberduck
Copy link
Collaborator Author

528c4d9 commented

Hi,
Thanks - please try now: tanzania-gwas@ftp.malariagen.net, accessed by sftp using the ssh key you've sent. The problem can be seen in:

/home/datarelease/data/callsets/genotyped/Tanzania_GWAS-2.5M_b37/master/

As reported the problems are

1: the list of files is incorrect (e.g. files with 'Tanzania_GWAS-2.5M_b37_chr19.vcf.gz' in appear twice in the list, including at the top after sorting by filename), while other files including the file 'Tanzania_GWAS-2.5M_b37_chr0X.vcf.gz' is missing.

2: There is equally a problem when downloading the whole folder of files - I think, but am not sure, that this has to do with trying to download Tanzania_GWAS-2.5M_b37_chr0X.vcf.gz but actually downloading Tanzania_GWAS-2.5M_b37_chr19.vcf.gz. (Sorry I realise this is vague, I can re-download to be more specific about this aspect of the problem, but maybe solving problem #1 will fix this.)

I am using Cyberduck 6.9.3 on a 2014 Macbook Pro running macOS High Sierra (10.13.6).

(Please note this data is restricted access - fine to download/access for testing purposes but please delete afterwards.)

Many thanks for your help with this,
Gavin.

@cyberduck
Copy link
Collaborator Author

528c4d9 commented

FYI that should be:

tanzania-gwas AT ftp DOT malariagen DOT net .

Best wishes,
Gavin.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

The list view (View → as List) is not affected by this bug.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

We can reproduce the issue which is caused by a hashing collision for 19 and 0X.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Replying to [comment:14 gavinband]:

FYI that should be:

tanzania-gwas AT ftp DOT malariagen DOT net .

Best wishes,
Gavin.

Thanks for providing the test account which was very helpful to find the cause of this bug. We can reproduce the issue and you can revoke access.

@cyberduck
Copy link
Collaborator Author

@ylangisc commented

In 6683fdf. Thanks for reporting this.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Revised in 69de759.

@iterate-ch iterate-ch locked as resolved and limited conversation to collaborators Nov 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug fixed sftp SFTP Protocol Implementation
Projects
None yet
Development

No branches or pull requests

2 participants