Cyberduck Mountain Duck CLI

#10564 closed defect (fixed)

Transfer with many files take a long time to finish

Reported by: sebastian Owned by: dkocher
Priority: normal Milestone: 6.9.0
Component: cryptomator Version: 6.8.3
Severity: normal Keywords:
Cc: Architecture: Intel
Platform: Windows 10

Description

When uploading or downloading a large job (e.g. a directory containing 5000 files, 75 GB in total) to/from an S3-cryptomator-vault, it takes very long (up to one hour and more) until the job is actually marked as finished. The CPU load during this phase is high.

It can be observed within CyberDuck and Duck.sh, on Windows and on Linux (I can not test on a Mac).

It makes queuing jobs impossible, because the job still counts as active.

The message that is displayed while nothing is being transferred but the job is still unfinished looks like this (note the missing ?? minutes remaining phrase):

[▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮] 71.4 GiB (76,687,296,696 bytes) of 71.4 GiB (100%, 2.6 MB/sec)

Change History (7)

comment:1 follow-up: Changed on Dec 31, 2018 at 9:55:59 PM by dkocher

  • Milestone set to 6.9.0
  • Owner set to dkocher
  • Status changed from new to assigned

This is caused by verifying the checksum of the downloaded file. Related to #10215.

comment:2 Changed on Jan 1, 2019 at 5:53:16 PM by dkocher

  • Resolution set to duplicate
  • Status changed from assigned to closed

comment:3 in reply to: ↑ 1 Changed on Jan 5, 2019 at 2:21:06 PM by sebastian

  • Milestone 6.9.0 deleted
  • Resolution duplicate deleted
  • Status changed from closed to reopened

Replying to dkocher:

This is caused by verifying the checksum of the downloaded file. Related to #10215.

Unfortunately this can not be the reason, because (as the ticket states) the behaviour occurs also on uploads where the checksum is calculated before the transfer.

So I did some more testing (on Windows using the Cyberduck GUI and the file:// protocol) and can provide more, hopefully helpful insights:

  • The waiting time only occurs when working with cryptomator vaults, not when working directly with an unencrypted local folder.
  • The waiting time scales (quadratically) with the number of files, not the amount of data:
    • 10.000 MB in 3 files produced no noticeable waiting time.
    • 0,8 MB in 100 files produced about 1 second of waiting time.
    • 1,6 MB in 200 files produced about 5 seconds of waiting time.
    • 3,2 MB in 400 files produced about 20 seconds of waiting time.
    • 6,4 MB in 800 files produced about 82 seconds of waiting time.
    • I did not wait for the job of 80 MB in 10.000 files to finish, but the waiting time calculates to about 3,5 hours given the observed quadratic growth.

My assumption is that for each file transferred (either up- or downloaded) from/to a cryptomator vault some data structure is left over and has to be cleaned up after the transfer is finished and that cleanup is done in an inefficient manner.

comment:4 Changed on Jan 5, 2019 at 2:24:48 PM by sebastian

  • Component changed from core to cryptomator
  • Summary changed from Large jobs take a long time after the last transfer to be marked as finished to Jobs with many files take a long time after the last transfer to be marked as finished

I changed the subject and component of this ticket to reflect my new findings.

comment:5 Changed on Jan 5, 2019 at 4:31:25 PM by dkocher

  • Milestone set to 6.9.0

comment:6 Changed on Jan 9, 2019 at 10:56:07 AM by yla

  • Resolution set to fixed
  • Status changed from reopened to closed

In r45960.

comment:7 Changed on Jan 13, 2019 at 3:19:05 PM by dkocher

  • Summary changed from Jobs with many files take a long time after the last transfer to be marked as finished to Transfer with many files take a long time to finish
Note: See TracTickets for help on using tickets.
swiss made software