Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

STAT for listings works poorly or not at all #2445

Closed
cyberduck opened this issue Aug 14, 2008 · 36 comments
Closed

STAT for listings works poorly or not at all #2445

cyberduck opened this issue Aug 14, 2008 · 36 comments
Assignees
Labels
bug fixed ftp FTP Protocol Implementation
Milestone

Comments

@cyberduck
Copy link
Collaborator

69aa58a created the issue

After installing 3.0.2 I found that directory listings from nearly all the servers I accessed (primarily Solaris and Linux) either returned nothing or timed out - and when retried, again returned nothing or timed out.

The change to use STAT instead of LIST seems to have broken Cyberduck, at least for me, to the point where it is absolutely unusable. I have backed off to 3.0.1 which (judging from the logs, uses LIST rather than STAT) works fine.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Refer to https://trac.cyberduck.io/wiki/help/en/problems#ListingdirectoriesfailsorshowsnocontentFTP for how to disable the use of STAT.

@cyberduck
Copy link
Collaborator Author

69aa58a commented

Unacceptable.

STAT works reliably on NONE of the Solaris or Linux systems I access on a regular basis. That includes systems in many data centers spread world-wide. STAT may work in various places, but given the ubiquity of its not working right, it should be the last resort, not the first, and users should not need to resort to rather arcane, kludgey workarounds to make the product work on a reliable basis.

LIST worked. STAT doesn't. Does "If it ain't broke, don't fix it" ring a bell?

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Replying to [comment:2 jeff.markel@gmail.com]:

Unacceptable.

STAT works reliably on NONE of the Solaris or Linux systems I access on a regular basis. That includes systems in many data centers spread world-wide. STAT may work in various places, but given the ubiquity of its not working right, it should be the last resort, not the first, and users should not need to resort to rather arcane, kludgey workarounds to make the product work on a reliable basis.

LIST worked. STAT doesn't. Does "If it ain't broke, don't fix it" ring a bell?

{{STAT}}} was introduced because it does not need a second connection to be opened to the server and is much faster therefore. The fallback mechanism not working properly must be fixed, I agree.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

#2417 closed as duplicate.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

#2428 closed as duplicate.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

#2470 closed as duplicate.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

#2471 closed as duplicate.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Can you please post some transcripts from View → Log Drawer for servers the file listing fails.

@cyberduck
Copy link
Collaborator Author

anonymous commented

Replying to [comment:9 dkocher]:

Can you please post some transcripts from View → Log Drawer for servers the file listing fails.

Attached some logs for you. I know that some of the clients I send files to use Cyberduck and I really cannot ask them to start farting around with the terminal so that I can implement the use of this server.

220 ProFTPD 1.3.1 Server (NETGEAR ReadyNAS) [xx.xx.xx.xx]
USER xxxxxx
331 Password required for xxxxxx
PASS xxxxxxxx
230 User xxxxxx logged in
PWD
257 "/" is the current directory
NOOP
200 NOOP command successful
SYST
215 UNIX Type: L8
STAT /
211-Status of /:
211-drwxr-xr-x   5 ftp      ftp          4096 Sep  2 07:54 .
211-drwxr-xr-x   5 ftp      ftp          4096 Sep  2 07:54 ..
211-drwxrwxrwx   7 FTPSERVER nogroup     16384 Sep  2 07:29 FTPSERVER
211 End of status
NOOP
200 NOOP command successful
STAT /FTPSERVER
211-Status of /FTPSERVER:
211-drwxrwxrwx   7 FTPSERVER nogroup     16384 Sep  2 07:29 .
211-drwxr-xr-x   5 ftp      ftp          4096 Sep  2 07:54 ..
211 End of status

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Thanks for the transcript. The server response

STAT /FTPSERVER
211-Status of /FTPSERVER:
211-drwxrwxrwx   7 FTPSERVER nogroup     16384 Sep  2 07:29 .
211-drwxr-xr-x   5 ftp      ftp          4096 Sep  2 07:54 ..
211 End of status

does not list any files in that folder. Does the server give a different listing when LIST is used? (You can test with another client or disable {{{STAT}} for Cyberduck as described in https://trac.cyberduck.io/wiki/help/en/problems#ListingdirectoriesfailsorshowsnocontentFTP)

Replying to [comment:10 anonymous]:

Replying to [comment:9 dkocher]:

Can you please post some transcripts from View → Log Drawer for servers the file listing fails.

Attached some logs for you. I know that some of the clients I send files to use Cyberduck and I really cannot ask them to start farting around with the terminal so that I can implement the use of this server.

220 ProFTPD 1.3.1 Server (NETGEAR ReadyNAS) [xx.xx.xx.xx]
USER xxxxxx
331 Password required for xxxxxx
PASS xxxxxxxx
230 User xxxxxx logged in
PWD
257 "/" is the current directory
NOOP
200 NOOP command successful
SYST
215 UNIX Type: L8
STAT /
211-Status of /:
211-drwxr-xr-x 5 ftp ftp 4096 Sep 2 07:54 .
211-drwxr-xr-x 5 ftp ftp 4096 Sep 2 07:54 ..
211-drwxrwxrwx 7 FTPSERVER nogroup 16384 Sep 2 07:29 FTPSERVER
211 End of status
NOOP
200 NOOP command successful
STAT /FTPSERVER
211-Status of /FTPSERVER:
211-drwxrwxrwx 7 FTPSERVER nogroup 16384 Sep 2 07:29 .
211-drwxr-xr-x 5 ftp ftp 4096 Sep 2 07:54 ..
211 End of status

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Similar report in #2530.

@cyberduck
Copy link
Collaborator Author

anonymous commented

Replying to [comment:12 dkocher]:

Similar report in #2530.

Yes this is a similar report however the solution in #2530 (disable STAT)did not work for me. I am having no problems accessing the site and directories via Transmit and CuteFTP.

Attached are the 3 logs, before and from after each terminal command (with restarting Cyberduck after each process)

The LIST command does not even seem to show the directories that STAT did (all be it empty)

220 ProFTPD 1.3.1 Server (NETGEAR ReadyNAS) [91.73.102.88]
USER xxxxxx
331 Password required for xxxxxx
PASS xxxxxx
230 User xxxxxx logged in
PWD
257 "/" is the current directory
NOOP
200 NOOP command successful
SYST
215 UNIX Type: L8
STAT /
211-Status of /:
211-drwxr-xr-x   5 ftp      ftp          4096 Sep  4 07:52 .
211-drwxr-xr-x   5 ftp      ftp          4096 Sep  4 07:52 ..
211-drwxrwxrwx   8 FTPSERVER nogroup     16384 Sep  3 14:41 FTPSERVER
211-drwxrwxrwx   8 xxxxxx   users        4096 Sep  3 14:44 Image_lib_backup
211 End of status
NOOP
200 NOOP command successful
STAT /Image_lib_backup
211-Status of /Image_lib_backup:
211-drwxrwxrwx   8 xxxxxx   users        4096 Sep  3 14:44 .
211-drwxr-xr-x   5 ftp      ftp          4096 Sep  4 07:52 ..
211 End of status




------AFTER RUNING TERMINAL COMMAND - defaults write ch.sudo.cyberduck ftp.sendStatListCommand false------

220 ProFTPD 1.3.1 Server (NETGEAR ReadyNAS) [xx.xx.xxx.xx]
USER xxxxxx
331 Password required for xxxxxx
PASS xxxxxx
230 User xxxxxx logged in
PWD
257 "/" is the current directory
NOOP
200 NOOP command successful
SYST
215 UNIX Type: L8
FEAT
211-Features:
 LANG en
 MDTM
 UTF8
 AUTH TLS
 PBSZ
 PROT
 REST STREAM
 SIZE
211 End
PASV
227 Entering Passive Mode (xx,xx,xxx,xx,248,163)
LIST -a





------AFTER RUNING TERMINAL COMMAND - defaults write ch.sudo.cyberduck ftp.sendExtendedListCommand false------

220 ProFTPD 1.3.1 Server (NETGEAR ReadyNAS) [xx.xx.xxx.xx]
USER xxxxxx
331 Password required for xxxxxx
PASS xxxxxx
230 User xxxxxx logged in
PWD
257 "/" is the current directory
NOOP
200 NOOP command successful
SYST
215 UNIX Type: L8
FEAT
211-Features:
 LANG en
 MDTM
 UTF8
 AUTH TLS
 PBSZ
 PROT
 REST STREAM
 SIZE
211 End
PASV
227 Entering Passive Mode (xx,xx,xxx,xx,248,75)
LIST

I have also attached a transcript log from CuteFTP where the LIST command seems to work properly

Connection attempt 1:
Connecting ftp.xxxxxxxxxxx.me (xx.xx.xx.xx:21)
220 ProFTPD 1.3.1 Server (NETGEAR ReadyNAS) [xx.xx.xxx.xx]
USER xxxxxx
331 Password required for xxxxxx
PASS xxxxxx
230 User xxxxxx logged in
SYST
215 UNIX Type: L8
Remote system is UNIX - text files transfer optimization is ON
MACB ENABLE
500 MACB not understood
PWD
257 "/" is the current directory
FEAT
211 Features:
 LANG en
 MDTM
 UTF8
 AUTH TLS
 PBSZ
 PROT
 REST STREAM
 SIZE
End
PORT xx,xx,xxx,xx,207,197
200 PORT command successful
LIST
150 Opening ASCII mode data connection for file list
226 Transfer complete
CWD Image_lib_backup
250 CWD command successful
PORT xx,xx,xx,xx,207,198
200 PORT command successful
LIST
150 Opening ASCII mode data connection for file list
226 Transfer complete

@cyberduck
Copy link
Collaborator Author

@dkocher commented

We suppose the most common case as described in #2549 is now fixed as of 96490a9. Please test the latest nightly build with ftp.sendStatListCommand property set to true.

@cyberduck
Copy link
Collaborator Author

anonymous commented

That didn't work, tested 96490a9 with the configuration you suggested above. Cyberduck lists the top (two) folders and then nothing in those folders. Testing alongside transmit which is working properly.

@cyberduck
Copy link
Collaborator Author

anonymous commented

The folders I am trying to access contain invisible mac files such as :2eDS_Store. Since . and .. files caused a problem is it possible that these are also causing the same problem?

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Replying to [comment:15 anonymous]:

That didn't work, tested 96490a9 with the configuration you suggested above. Cyberduck lists the top (two) folders and then nothing in those folders. Testing alongside transmit which is working properly.

Again, please post the transcript from View → Log Drawer.

@cyberduck
Copy link
Collaborator Author

b704f45 commented

Cyberduck 3 is unable to list files in some directories, on most FTP servers I've tried recently. Cyberduck 2.8.5 has no such problem.

After some trial-and-error testing, I see that the STAT command works fine IF the directory is short, otherwise a LIST command is required.

Using Cyberduck 3.0.3 (4180), here are two examples using Verizon's FTP server (215 UNIX Type: L8):

  1. defaults write ch.sudo.cyberduck ftp.sendStatListCommand true
  2. Launch Cyberduck.
  3. Connect to my account on Verizon's ftp server.
  4. Double-click on a large directory.

Cyberduck shows this:

NOOP
200 NOOP command successful
STAT /auctions
211-status of /auctions:
 drwxr-xr-x   3 web      web          8192 Aug 26 19:39 .
 drwxrwxrwx   6 1047     602          4096 Nov 10  2007 ..

then it sits there forever making no further progress.

Quit Cyberduck then

  1. defaults write ch.sudo.cyberduck ftp.sendStatListCommand false
  2. Launch Cyberduck.
  3. Connect to my account on Verizon's ftp server.
  4. Double-click on a large directory.

Cyberduck shows this:

NOOP
200 NOOP command successful
CWD /auctions
250 CWD command successful.
PORT 192,168,0,16,193,56
200 PORT command successful
LIST -a
150 Opening ASCII mode data connection for file list
drwxr-xr-x   3 web      web          8192 Aug 26 19:39 .
drwxrwxrwx   6 1047     602          4096 Nov 10  2007 ..
-rw-r--r--   1 web      web        201848 Aug 22 20:05 belgium-1991-50franc-back-big.jpg
-rw-r--r--   1 web      web         38309 Aug 22 20:05 belgium-1991-50franc-back.jpg
-rw-r--r--   1 web      web        223621 Aug 22 20:05 belgium-1991-50franc-big.jpg
-rw-r--r--   1 web      web         35839 Aug 22 20:05 belgium-1991-50franc.jpg
-rw-r--r--   1 web      web        153165 Aug  5 20:13 canada-1927-5cents-back-big.jpg
-rw-r--r--   1 web      web         36509 Aug  5 20:13 canada-1927-5cents-back.jpg
-rw-r--r--   1 web      web        157110 Aug  5 20:13 canada-1927-5cents-big.jpg
-rw-r--r--   1 web      web         33236 Aug  5 20:13 canada-1927-5cents.jpg

the above continues for a long long time then Cyberduck's list window is properly and fully populated.

IMO this bug is a critical/blocker... It pretty much makes Cyberduck totally useless for many people.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

#2500 is a specific issue STAT issue.

@cyberduck
Copy link
Collaborator Author

b704f45 commented

Two part fix suggestion:

  1. If the STAT command response pauses for more than n seconds (where n is VERY small), and fails to say something like "211 End of Status", quickly abort that method and switch to the LIST command.

  2. Provide a selector in the connection/bookmark window to let the user specify which to try first. And if (1) requires the switch, change it in that bookmark too!

Performance concern: If detecting the failure, aborting, and switching to the other command, takes too long, then the end user will perceive Cyberduck as being annoyingly slow. Might be a good idea to hit 'em with some status type dialog so they know what's happening.

HTH,

  • Dan.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Another example of a STAT listing not parsed properly always having absolute paths:

SYST
215 UNIX Type: L8
STAT /
211-status of /:
total 48
drwxrwx---   3 68463        4096 Aug 17  2004 /cgi
drwxrwx---   2 68463        4096 Aug 25  2004 /files
drwxrwx---   5 68463        4096 Nov 17  2006 /logs
drwxrwx---  16 68463       12288 Oct  9 00:09 /web
211 End of Status
NOOP
200 NOOP command successful.
STAT //logs
211-status of //logs:
total 24
drwxrwx---   2 68463        4096 Jul 13  2001 //logs/acct
drwxrwx---   2 68463        4096 Oct 25  2001 //logs/misc
drwxrwx---   2 68463        4096 Oct  8 23:40 //logs/web
211 End of Status
NOOP
200 NOOP command successful.
STAT //logs/logs/web
211-status of //logs/logs/web:
550 //logs/logs/web: No such file or directory.
211 End of Status
CWD //logs/logs/web
550 //logs/logs/web: No such file or directory.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Replying to [comment:21 dkocher]:

Another example of a STAT listing not parsed properly always having absolute paths:

SYST
215 UNIX Type: L8
STAT /
211-status of /:
total 48
drwxrwx---   3 68463        4096 Aug 17  2004 /cgi
drwxrwx---   2 68463        4096 Aug 25  2004 /files
drwxrwx---   5 68463        4096 Nov 17  2006 /logs
drwxrwx---  16 68463       12288 Oct  9 00:09 /web
211 End of Status
NOOP
200 NOOP command successful.
STAT //logs
211-status of //logs:
total 24
drwxrwx---   2 68463        4096 Jul 13  2001 //logs/acct
drwxrwx---   2 68463        4096 Oct 25  2001 //logs/misc
drwxrwx---   2 68463        4096 Oct  8 23:40 //logs/web
211 End of Status
NOOP
200 NOOP command successful.
STAT //logs/logs/web
211-status of //logs/logs/web:
550 //logs/logs/web: No such file or directory.
211 End of Status
CWD //logs/logs/web
550 //logs/logs/web: No such file or directory.

Fixed in 5e2fefc.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Another fix for servers only returning . and .. in 1206e43.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

I consider this fixed with the latest commits. Please test the nightly build.

@cyberduck
Copy link
Collaborator Author

b704f45 commented

Using Cyberduck 3.0.3 (4202), OS X 10.4.11, Java 1.5.0_16

with 'defaults write ch.sudo.cyberduck ftp.sendStatListCommand true'... Cyberduck still gets stuck in Verizon's ftp, just like reported (above). Instead of sitting forever tho, it now sits for half a minute or so then provides a window with no files listed. No error messages are thrown to system or console log. The log window shows the same as before:

STAT /auctions
211-status of /auctions:
 drwxr-xr-x   3 web      web          8192 Aug 26 19:39 .
 drwxrwxrwx   6 1047     602          4096 Nov 10  2007 ..

with 'defaults write ch.sudo.cyberduck ftp.sendStatListCommand false', Cyberduck works perfectly.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

#2660 closed as duplicate.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

We delay the resolution of the Verizon Compatibility issue. This is clearly a server bug as the response is never terminated.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

#2685 closed as duplicate.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Specific incompatibility with Microsoft FTP Service in #2702.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

STAT read timeout with WFTPD Pro 3.20.1.2 Server in #3086.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Milestone 3.2 deleted

@cyberduck
Copy link
Collaborator Author

@dkocher commented

As of 4a12a6c we now additionally support directory listings using MLSD as defined in RFC 3659.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Reopen because of #4116.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Another incompatiblity reported in #4297.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Another server failure in #4110.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Symbolic link failure in #4361.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Fallback if read times out for STAT in 1fd0614. This should fix all issues with broken servers.

@iterate-ch iterate-ch locked as resolved and limited conversation to collaborators Nov 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug fixed ftp FTP Protocol Implementation
Projects
None yet
Development

No branches or pull requests

2 participants