Cyberduck Mountain Duck CLI

#2445 closed defect (fixed)

STAT for listings works poorly or not at all

Reported by: jeff.markel@… Owned by: dkocher
Priority: normal Milestone: 3.8
Component: ftp Version: 3.0.2
Severity: normal Keywords:
Cc: dantearmok@… Architecture:
Platform: Mac OS X 10.6

Description

After installing 3.0.2 I found that directory listings from nearly all the servers I accessed (primarily Solaris and Linux) either returned nothing or timed out - and when retried, again returned nothing or timed out.

The change to use STAT instead of LIST seems to have broken Cyberduck, at least for me, to the point where it is absolutely unusable. I have backed off to 3.0.1 which (judging from the logs, uses LIST rather than STAT) works fine.

Change History (41)

comment:1 Changed on Aug 15, 2008 at 7:56:18 PM by dkocher

  • Component changed from core to ftp
  • Summary changed from STAT for listings in 3.0.2 works poorly or not at all to STAT for listings works poorly or not at all

Refer to help/en/problems? for how to disable the use of STAT.

comment:2 follow-up: Changed on Aug 15, 2008 at 8:42:41 PM by jeff.markel@…

Unacceptable.

STAT works reliably on NONE of the Solaris or Linux systems I access on a regular basis. That includes systems in many data centers spread world-wide. STAT may work in various places, but given the ubiquity of its not working right, it should be the last resort, not the first, and users should not need to resort to rather arcane, kludgey workarounds to make the product work on a reliable basis.

LIST worked. STAT doesn't. Does "If it ain't broke, don't fix it" ring a bell?

comment:3 Changed on Aug 15, 2008 at 8:56:51 PM by dkocher

  • Milestone changed from 3.1 to 3.0.3
  • Status changed from new to assigned

comment:4 in reply to: ↑ 2 Changed on Aug 15, 2008 at 8:58:33 PM by dkocher

Replying to jeff.markel@gmail.com:

Unacceptable.

STAT works reliably on NONE of the Solaris or Linux systems I access on a regular basis. That includes systems in many data centers spread world-wide. STAT may work in various places, but given the ubiquity of its not working right, it should be the last resort, not the first, and users should not need to resort to rather arcane, kludgey workarounds to make the product work on a reliable basis.

LIST worked. STAT doesn't. Does "If it ain't broke, don't fix it" ring a bell?

{{STAT}}} was introduced because it does not need a second connection to be opened to the server and is much faster therefore. The fallback mechanism not working properly must be fixed, I agree.

comment:5 Changed on Aug 15, 2008 at 9:07:54 PM by dkocher

#2417 closed as duplicate.

comment:6 Changed on Aug 15, 2008 at 9:09:56 PM by dkocher

#2428 closed as duplicate.

comment:7 Changed on Aug 20, 2008 at 7:44:33 AM by dkocher

#2470 closed as duplicate.

comment:8 Changed on Aug 20, 2008 at 7:45:36 AM by dkocher

#2471 closed as duplicate.

comment:9 follow-up: Changed on Aug 31, 2008 at 1:12:11 PM by dkocher

Can you please post some transcripts from View → Log Drawer for servers the file listing fails.

comment:10 in reply to: ↑ 9 ; follow-up: Changed on Sep 2, 2008 at 9:40:42 AM by anonymous

Replying to dkocher:

Can you please post some transcripts from View → Log Drawer for servers the file listing fails.

Attached some logs for you. I know that some of the clients I send files to use Cyberduck and I really cannot ask them to start farting around with the terminal so that I can implement the use of this server.

220 ProFTPD 1.3.1 Server (NETGEAR ReadyNAS) [xx.xx.xx.xx]
USER xxxxxx
331 Password required for xxxxxx
PASS xxxxxxxx
230 User xxxxxx logged in
PWD
257 "/" is the current directory
NOOP
200 NOOP command successful
SYST
215 UNIX Type: L8
STAT /
211-Status of /:
211-drwxr-xr-x   5 ftp      ftp          4096 Sep  2 07:54 .
211-drwxr-xr-x   5 ftp      ftp          4096 Sep  2 07:54 ..
211-drwxrwxrwx   7 FTPSERVER nogroup     16384 Sep  2 07:29 FTPSERVER
211 End of status
NOOP
200 NOOP command successful
STAT /FTPSERVER
211-Status of /FTPSERVER:
211-drwxrwxrwx   7 FTPSERVER nogroup     16384 Sep  2 07:29 .
211-drwxr-xr-x   5 ftp      ftp          4096 Sep  2 07:54 ..
211 End of status
Last edited on Jul 11, 2013 at 9:40:21 AM by dkocher (previous) (diff)

comment:11 in reply to: ↑ 10 Changed on Sep 3, 2008 at 5:30:36 PM by dkocher

Thanks for the transcript. The server response

STAT /FTPSERVER
211-Status of /FTPSERVER:
211-drwxrwxrwx   7 FTPSERVER nogroup     16384 Sep  2 07:29 .
211-drwxr-xr-x   5 ftp      ftp          4096 Sep  2 07:54 ..
211 End of status

does not list any files in that folder. Does the server give a different listing when LIST is used? (You can test with another client or disable {{{STAT}} for Cyberduck as described in help/en/problems?)

Replying to anonymous:

Replying to dkocher:

Can you please post some transcripts from View → Log Drawer for servers the file listing fails.

Attached some logs for you. I know that some of the clients I send files to use Cyberduck and I really cannot ask them to start farting around with the terminal so that I can implement the use of this server.

220 ProFTPD 1.3.1 Server (NETGEAR ReadyNAS) [xx.xx.xx.xx] USER xxxxxx 331 Password required for xxxxxx PASS xxxxxxxx 230 User xxxxxx logged in PWD 257 "/" is the current directory NOOP 200 NOOP command successful SYST 215 UNIX Type: L8 STAT / 211-Status of /: 211-drwxr-xr-x 5 ftp ftp 4096 Sep 2 07:54 . 211-drwxr-xr-x 5 ftp ftp 4096 Sep 2 07:54 .. 211-drwxrwxrwx 7 FTPSERVER nogroup 16384 Sep 2 07:29 FTPSERVER 211 End of status NOOP 200 NOOP command successful STAT /FTPSERVER 211-Status of /FTPSERVER: 211-drwxrwxrwx 7 FTPSERVER nogroup 16384 Sep 2 07:29 . 211-drwxr-xr-x 5 ftp ftp 4096 Sep 2 07:54 .. 211 End of status

comment:12 follow-up: Changed on Sep 3, 2008 at 8:04:48 PM by dkocher

Similar report in #2530.

comment:13 in reply to: ↑ 12 Changed on Sep 4, 2008 at 5:32:25 AM by anonymous

Replying to dkocher:

Similar report in #2530.

Yes this is a similar report however the solution in #2530 (disable STAT)did not work for me. I am having no problems accessing the site and directories via Transmit and CuteFTP.

Attached are the 3 logs, before and from after each terminal command (with restarting Cyberduck after each process)

The LIST command does not even seem to show the directories that STAT did (all be it empty)

220 ProFTPD 1.3.1 Server (NETGEAR ReadyNAS) [91.73.102.88]
USER xxxxxx
331 Password required for xxxxxx
PASS xxxxxx
230 User xxxxxx logged in
PWD
257 "/" is the current directory
NOOP
200 NOOP command successful
SYST
215 UNIX Type: L8
STAT /
211-Status of /:
211-drwxr-xr-x   5 ftp      ftp          4096 Sep  4 07:52 .
211-drwxr-xr-x   5 ftp      ftp          4096 Sep  4 07:52 ..
211-drwxrwxrwx   8 FTPSERVER nogroup     16384 Sep  3 14:41 FTPSERVER
211-drwxrwxrwx   8 xxxxxx   users        4096 Sep  3 14:44 Image_lib_backup
211 End of status
NOOP
200 NOOP command successful
STAT /Image_lib_backup
211-Status of /Image_lib_backup:
211-drwxrwxrwx   8 xxxxxx   users        4096 Sep  3 14:44 .
211-drwxr-xr-x   5 ftp      ftp          4096 Sep  4 07:52 ..
211 End of status




------AFTER RUNING TERMINAL COMMAND - defaults write ch.sudo.cyberduck ftp.sendStatListCommand false------

220 ProFTPD 1.3.1 Server (NETGEAR ReadyNAS) [xx.xx.xxx.xx]
USER xxxxxx
331 Password required for xxxxxx
PASS xxxxxx
230 User xxxxxx logged in
PWD
257 "/" is the current directory
NOOP
200 NOOP command successful
SYST
215 UNIX Type: L8
FEAT
211-Features:
 LANG en
 MDTM
 UTF8
 AUTH TLS
 PBSZ
 PROT
 REST STREAM
 SIZE
211 End
PASV
227 Entering Passive Mode (xx,xx,xxx,xx,248,163)
LIST -a





------AFTER RUNING TERMINAL COMMAND - defaults write ch.sudo.cyberduck ftp.sendExtendedListCommand false------

220 ProFTPD 1.3.1 Server (NETGEAR ReadyNAS) [xx.xx.xxx.xx]
USER xxxxxx
331 Password required for xxxxxx
PASS xxxxxx
230 User xxxxxx logged in
PWD
257 "/" is the current directory
NOOP
200 NOOP command successful
SYST
215 UNIX Type: L8
FEAT
211-Features:
 LANG en
 MDTM
 UTF8
 AUTH TLS
 PBSZ
 PROT
 REST STREAM
 SIZE
211 End
PASV
227 Entering Passive Mode (xx,xx,xxx,xx,248,75)
LIST

I have also attached a transcript log from CuteFTP where the LIST command seems to work properly

Connection attempt 1:
Connecting ftp.xxxxxxxxxxx.me (xx.xx.xx.xx:21)
220 ProFTPD 1.3.1 Server (NETGEAR ReadyNAS) [xx.xx.xxx.xx]
USER xxxxxx
331 Password required for xxxxxx
PASS xxxxxx
230 User xxxxxx logged in
SYST
215 UNIX Type: L8
Remote system is UNIX - text files transfer optimization is ON
MACB ENABLE
500 MACB not understood
PWD
257 "/" is the current directory
FEAT
211 Features:
 LANG en
 MDTM
 UTF8
 AUTH TLS
 PBSZ
 PROT
 REST STREAM
 SIZE
End
PORT xx,xx,xxx,xx,207,197
200 PORT command successful
LIST
150 Opening ASCII mode data connection for file list
226 Transfer complete
CWD Image_lib_backup
250 CWD command successful
PORT xx,xx,xx,xx,207,198
200 PORT command successful
LIST
150 Opening ASCII mode data connection for file list
226 Transfer complete

comment:14 Changed on Sep 5, 2008 at 7:23:54 PM by dkocher

  • Resolution set to fixed
  • Status changed from assigned to closed

We suppose the most common case as described in #2549 is now fixed as of r4177. Please test the latest nightly build with ftp.sendStatListCommand property set to true.

comment:15 follow-up: Changed on Sep 8, 2008 at 4:14:37 AM by anonymous

  • Resolution fixed deleted
  • Status changed from closed to reopened

That didn't work, tested r4177 with the configuration you suggested above. Cyberduck lists the top (two) folders and then nothing in those folders. Testing alongside transmit which is working properly.

comment:16 Changed on Sep 8, 2008 at 4:20:23 AM by anonymous

The folders I am trying to access contain invisible mac files such as :2eDS_Store. Since . and .. files caused a problem is it possible that these are also causing the same problem?

comment:17 in reply to: ↑ 15 Changed on Sep 9, 2008 at 7:54:29 PM by dkocher

Replying to anonymous:

That didn't work, tested r4177 with the configuration you suggested above. Cyberduck lists the top (two) folders and then nothing in those folders. Testing alongside transmit which is working properly.

Again, please post the transcript from View → Log Drawer.

comment:18 Changed on Sep 21, 2008 at 6:14:47 PM by darmok

  • Cc dantearmok@… added
  • Priority changed from normal to highest
  • Severity changed from normal to critical

Cyberduck 3 is unable to list files in some directories, on most FTP servers I've tried recently. Cyberduck 2.8.5 has no such problem.

After some trial-and-error testing, I see that the STAT command works fine IF the directory is short, otherwise a LIST command is required.

Using Cyberduck 3.0.3 (4180), here are two examples using Verizon's FTP server (215 UNIX Type: L8):

  1. defaults write ch.sudo.cyberduck ftp.sendStatListCommand true
  2. Launch Cyberduck.
  3. Connect to my account on Verizon's ftp server.
  4. Double-click on a large directory.

Cyberduck shows this:

NOOP
200 NOOP command successful
STAT /auctions
211-status of /auctions:
 drwxr-xr-x   3 web      web          8192 Aug 26 19:39 .
 drwxrwxrwx   6 1047     602          4096 Nov 10  2007 ..

then it sits there forever making no further progress.

Quit Cyberduck then

  1. defaults write ch.sudo.cyberduck ftp.sendStatListCommand false
  2. Launch Cyberduck.
  3. Connect to my account on Verizon's ftp server.
  4. Double-click on a large directory.

Cyberduck shows this:

NOOP
200 NOOP command successful
CWD /auctions
250 CWD command successful.
PORT 192,168,0,16,193,56
200 PORT command successful
LIST -a
150 Opening ASCII mode data connection for file list
drwxr-xr-x   3 web      web          8192 Aug 26 19:39 .
drwxrwxrwx   6 1047     602          4096 Nov 10  2007 ..
-rw-r--r--   1 web      web        201848 Aug 22 20:05 belgium-1991-50franc-back-big.jpg
-rw-r--r--   1 web      web         38309 Aug 22 20:05 belgium-1991-50franc-back.jpg
-rw-r--r--   1 web      web        223621 Aug 22 20:05 belgium-1991-50franc-big.jpg
-rw-r--r--   1 web      web         35839 Aug 22 20:05 belgium-1991-50franc.jpg
-rw-r--r--   1 web      web        153165 Aug  5 20:13 canada-1927-5cents-back-big.jpg
-rw-r--r--   1 web      web         36509 Aug  5 20:13 canada-1927-5cents-back.jpg
-rw-r--r--   1 web      web        157110 Aug  5 20:13 canada-1927-5cents-big.jpg
-rw-r--r--   1 web      web         33236 Aug  5 20:13 canada-1927-5cents.jpg

the above continues for a long long time then Cyberduck's list window is properly and fully populated.

IMO this bug is a critical/blocker... It pretty much makes Cyberduck totally useless for many people.

comment:19 Changed on Oct 6, 2008 at 1:53:02 PM by dkocher

#2500 is a specific issue STAT issue.

comment:20 Changed on Oct 15, 2008 at 6:05:26 PM by darmok

Two part fix suggestion:

1) If the STAT command response pauses for more than n seconds (where n is VERY small), and fails to say something like "211 End of Status", quickly abort that method and switch to the LIST command.

2) Provide a selector in the connection/bookmark window to let the user specify which to try first. And if (1) requires the switch, change it in that bookmark too!

Performance concern: If detecting the failure, aborting, and switching to the other command, takes too long, then the end user will perceive Cyberduck as being annoyingly slow. Might be a good idea to hit 'em with some status type dialog so they know what's happening.

HTH,

  • Dan.

comment:21 follow-up: Changed on Oct 17, 2008 at 12:49:39 PM by dkocher

Another example of a STAT listing not parsed properly always having absolute paths:

SYST
215 UNIX Type: L8
STAT /
211-status of /:
total 48
drwxrwx---   3 68463        4096 Aug 17  2004 /cgi
drwxrwx---   2 68463        4096 Aug 25  2004 /files
drwxrwx---   5 68463        4096 Nov 17  2006 /logs
drwxrwx---  16 68463       12288 Oct  9 00:09 /web
211 End of Status
NOOP
200 NOOP command successful.
STAT //logs
211-status of //logs:
total 24
drwxrwx---   2 68463        4096 Jul 13  2001 //logs/acct
drwxrwx---   2 68463        4096 Oct 25  2001 //logs/misc
drwxrwx---   2 68463        4096 Oct  8 23:40 //logs/web
211 End of Status
NOOP
200 NOOP command successful.
STAT //logs/logs/web
211-status of //logs/logs/web:
550 //logs/logs/web: No such file or directory.
211 End of Status
CWD //logs/logs/web
550 //logs/logs/web: No such file or directory.

comment:22 in reply to: ↑ 21 Changed on Oct 17, 2008 at 8:45:26 PM by dkocher

Replying to dkocher:

Another example of a STAT listing not parsed properly always having absolute paths:

SYST
215 UNIX Type: L8
STAT /
211-status of /:
total 48
drwxrwx---   3 68463        4096 Aug 17  2004 /cgi
drwxrwx---   2 68463        4096 Aug 25  2004 /files
drwxrwx---   5 68463        4096 Nov 17  2006 /logs
drwxrwx---  16 68463       12288 Oct  9 00:09 /web
211 End of Status
NOOP
200 NOOP command successful.
STAT //logs
211-status of //logs:
total 24
drwxrwx---   2 68463        4096 Jul 13  2001 //logs/acct
drwxrwx---   2 68463        4096 Oct 25  2001 //logs/misc
drwxrwx---   2 68463        4096 Oct  8 23:40 //logs/web
211 End of Status
NOOP
200 NOOP command successful.
STAT //logs/logs/web
211-status of //logs/logs/web:
550 //logs/logs/web: No such file or directory.
211 End of Status
CWD //logs/logs/web
550 //logs/logs/web: No such file or directory.

Fixed in r4199.

comment:23 Changed on Oct 17, 2008 at 9:08:40 PM by dkocher

Another fix for servers only returning . and .. in r4200.

comment:24 Changed on Oct 17, 2008 at 9:49:02 PM by dkocher

  • Resolution set to fixed
  • Status changed from reopened to closed

I consider this fixed with the latest commits. Please test the nightly build.

comment:25 Changed on Oct 19, 2008 at 2:12:39 PM by darmok

  • Resolution fixed deleted
  • Status changed from closed to reopened

Using Cyberduck 3.0.3 (4202), OS X 10.4.11, Java 1.5.0_16

with 'defaults write ch.sudo.cyberduck ftp.sendStatListCommand true'... Cyberduck still gets stuck in Verizon's ftp, just like reported (above). Instead of sitting forever tho, it now sits for half a minute or so then provides a window with no files listed. No error messages are thrown to system or console log. The log window shows the same as before:

STAT /auctions
211-status of /auctions:
 drwxr-xr-x   3 web      web          8192 Aug 26 19:39 .
 drwxrwxrwx   6 1047     602          4096 Nov 10  2007 ..

with 'defaults write ch.sudo.cyberduck ftp.sendStatListCommand false', Cyberduck works perfectly.

comment:26 Changed on Oct 20, 2008 at 8:21:53 AM by dkocher

#2660 closed as duplicate.

comment:27 Changed on Oct 20, 2008 at 8:26:33 AM by dkocher

  • Milestone changed from 3.0.3 to 3.1

We delay the resolution of the Verizon Compatibility issue. This is clearly a server bug as the response is never terminated.

comment:28 Changed on Oct 23, 2008 at 12:14:13 PM by dkocher

#2685 closed as duplicate.

comment:29 Changed on Oct 30, 2008 at 7:06:29 PM by dkocher

Specific incompatibility with Microsoft FTP Service in #2702.

comment:30 Changed on Mar 26, 2009 at 3:13:03 PM by dkocher

STAT read timeout with WFTPD Pro 3.20.1.2 Server in #3086.

comment:31 Changed on Apr 13, 2009 at 2:31:27 PM by dkocher

  • Milestone changed from 3.2 to 3.3

Milestone 3.2 deleted

comment:32 Changed on Apr 13, 2009 at 9:27:33 PM by dkocher

As of r4600 we now additionally support directory listings using MLSD as defined in RFC 3659.

comment:33 Changed on Dec 31, 2009 at 4:09:40 PM by dkocher

  • Resolution set to worksforme
  • Status changed from reopened to closed

comment:34 Changed on Jan 17, 2010 at 2:45:05 PM by dkocher

  • Milestone 3.4.1 deleted

comment:35 Changed on Jan 22, 2010 at 9:32:34 AM by dkocher

  • Priority changed from highest to normal
  • Resolution worksforme deleted
  • Severity changed from critical to normal
  • Status changed from closed to reopened

Reopen because of #4116.

comment:36 Changed on Mar 28, 2010 at 4:03:24 PM by dkocher

Another incompatiblity reported in #4297.

comment:37 Changed on Apr 1, 2010 at 3:54:02 PM by dkocher

  • Platform set to Mac OS X 10.6

Another server failure in #4110.

comment:38 Changed on Apr 10, 2010 at 2:02:01 PM by dkocher

Symbolic link failure in #4361.

comment:39 Changed on Aug 24, 2010 at 10:22:05 AM by dkocher

  • Milestone set to 3.6
  • Resolution set to worksforme
  • Status changed from reopened to closed

comment:40 Changed on Oct 28, 2010 at 11:59:09 AM by dkocher

  • Milestone changed from 3.6 to 4.0
  • Resolution worksforme deleted
  • Status changed from closed to reopened

comment:41 Changed on Oct 28, 2010 at 12:01:03 PM by dkocher

  • Resolution set to fixed
  • Status changed from reopened to closed

Fallback if read times out for STAT in r7422. This should fix all issues with broken servers.

Note: See TracTickets for help on using tickets.
swiss made software