Cyberduck Mountain Duck CLI

Changes between Version 1 and Version 2 of help/en/howto/mount/issues/fastcgi


Ignore:
Timestamp:
Jan 24, 2018 1:30:45 PM (21 months ago)
Author:
yla
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • help/en/howto/mount/issues/fastcgi

    v1 v2  
    1 === The annoying zero byte files issue with Nextcloud and ownCloud ===
     1== The annoying zero byte files issue with Nextcloud and ownCloud ==
     2
     3=== Summary ===
     4
     5Using a client to upload files with chunked transfer encoding to a server with fastcgi/php-fpm enabled can lead to zero byte files.
     6
     7=== Details ===
     8
     9There are basically two options to encode a `PUT` request. You either know the length of the data you want to transfer at the time of starting the request then you can set the HTTP `Content-Length` header appropriately or you don't know the length and thus have to choose a streaming approach using the chunked transfer encoding which does not need a `Content-Length` header at all. A client is free to choose either of the two request types as both are completely fine with the [https://tools.ietf.org/html/rfc2616 HTTP specification].
     10
     11Most clients out there, including web browsers, use the former method and thus do not hit this issue. There are use cases though which make it necessary to transfer data chunked. For example [https://mountainduck.io Mountain Duck] that implements a virtual file system for accessing your cloud storage online. The write callbacks we get from the OS just include an offset, a buffer length and the buffer itself. Mountain Duck does not know the final file size in advance. From a virtual filesystem perspective the callbacks Mountain Duck get for an upload are as follows.
     12
     13{{{
     141 - CreateFile myFile.txt
     152 - SetAllocationSize myFile.txt, 1024
     163 - WriteFile myFile.txt, offset 0, length 1024, buffer
     174 - SetAllocationSize myFile.txt, 2048
     185 - WriteFile myFile.txt, offset 1024, length 1024, buffer
     196 - CloseFile myFile.txt
     20}}}
     21
     22In step 3 [https://mountainduck.io Mountain Duck] opens a connection to the remote server, sends a `PUT` request with the http header `Transfer-Encoding: chunked` and streams through all subsequent write callbacks. Finally in step 6 the connection is closed.
     23
     24Since the expected length is missing in the header the streaming characteristic of such requests makes them more difficult to be handled in the components being passed. From our research the issue only exists in environments that use FastCGI to speak to their PHP application. Related bug reports - [https://bugs.php.net/bug.php?id=60826 here] and [https://bz.apache.org/bugzilla/show_bug.cgi?id=53332 here].
     25
     26The nginx developers try to work around this issue by simply buffering the entire incoming stream and forward it through the FastCGI interface as a request with a well-known length. Nginx does request buffering by [http://nginx.org/en/docs/http/ngx_http_fastcgi_module.html#fastcgi_request_buffering default]. There are several [http://nginx.org/en/docs/http/ngx_http_fastcgi_module.html options] to adjust the buffering behaviour.
     27
     28Regarding Apache HTTP Server we are not aware of any buffering strategies in [https://httpd.apache.org/mod_fcgid mod_fcgid] or [https://wiki.apache.org/httpd/PHP-FPM mod_proxy_fcgi / php-fpm] and thus you should avoid using FastCGI. It's recommended to use `mod_php` which is known to handle chunked transfers correctly.
     29
     30=== Clients using chunked transfer encoding ===
     31
     32* Mountain Duck
     33* MacOS Finder
     34* ...
     35
     36=== Resources ===
     37
     38* [http://sabre.io/dav/clients/finder/ sabre/dav] - the WebDAV library used in NC and OC
     39 * [http://sabre.io/dav/clients/finder/ Finder]
     40 * [http://sabre.io/dav/0bytes/ Zero Byte Files]
     41
swiss made software