Cyberduck Mountain Duck CLI

Version 5 (modified by dkocher, on Jan 24, 2018 at 1:41:06 PM) (diff)

--

Zero byte file truncate issue when uploading to Nextcloud and ownCloud

Summary

Using a client to upload files with HTTP chunked transfer encoding to a server with fastcgi/php-fpm enabled can lead to zero byte files. Chunked transfer encoding is used when the content length is unknown at the time the transfer is started and no Content-Length header can be set.

Details

There are basically two options to encode a PUT request. You either know the length of the data you want to transfer at the time of starting the request then you can set the HTTP Content-Length header appropriately or you don't know the length and thus have to choose a streaming approach using the chunked transfer encoding which does not need a Content-Length header at all. A client is free to choose either of the two request types as both are completely fine with the HTTP specification.

Most clients out there, including web browsers, use the former method and thus do not hit this issue. There are use cases though which make it necessary to transfer data chunked. For example Mountain Duck that implements a virtual file system for accessing your cloud storage online. The write callbacks we get from the OS just include an offset, a buffer length and the buffer itself. Mountain Duck does not know the final file size in advance. From a virtual filesystem perspective the callbacks Mountain Duck get for an upload are as follows.

1 - CreateFile myFile.txt
2 - SetAllocationSize myFile.txt, 1024
3 - WriteFile myFile.txt, offset 0, length 1024, buffer
4 - SetAllocationSize myFile.txt, 2048
5 - WriteFile myFile.txt, offset 1024, length 1024, buffer
6 - CloseFile myFile.txt

In step 3 Mountain Duck opens a connection to the remote server, sends a PUT request with the http header Transfer-Encoding: chunked and streams through all subsequent write callbacks. Finally in step 6 the connection is closed.

Since the expected length is missing in the header the streaming characteristic of such requests makes them more difficult to be handled in the components being passed. From our research the issue only exists in environments that use FastCGI to speak to their PHP application. Related bug reports - here and here.

The nginx developers try to work around this issue by simply buffering the entire incoming stream and forward it through the FastCGI interface as a request with a well-known length. Nginx does request buffering by default. There are several options to adjust the buffering behaviour.

Regarding Apache HTTP Server we are not aware of any buffering strategies in mod_fcgid or mod_proxy_fcgi / php-fpm and thus you should avoid using FastCGI. It's recommended to use mod_php which is known to handle chunked transfers correctly.

Clients using chunked transfer encoding

  • Mountain Duck
  • MacOS Finder
  • ...

Resources

swiss made software