Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Transfer with many files take a long time to finish #10564

Closed
cyberduck opened this issue Dec 28, 2018 · 4 comments
Closed

Transfer with many files take a long time to finish #10564

cyberduck opened this issue Dec 28, 2018 · 4 comments
Assignees
@dkocher
Labels
bug cryptomator Cryptomator Vault fixed
Milestone
6.9.0

Comments

@cyberduck
Copy link
Collaborator

9ebe7cf created the issue

When uploading or downloading a large job (e.g. a directory containing 5000 files, 75 GB in total) to/from an S3-cryptomator-vault, it takes very long (up to one hour and more) until the job is actually marked as finished. The CPU load during this phase is high.

It can be observed within CyberDuck and Duck.sh, on Windows and on Linux (I can not test on a Mac).

It makes queuing jobs impossible, because the job still counts as active.

The message that is displayed while nothing is being transferred but the job is still unfinished looks like this (note the missing ?? minutes remaining phrase):

[▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮▮] 71.4 GiB (76,687,296,696 bytes) of 71.4 GiB (100%, 2.6 MB/sec)
@cyberduck
Copy link
Collaborator Author

@dkocher commented

This is caused by verifying the checksum of the downloaded file. Related to #10215.

@cyberduck
Copy link
Collaborator Author

9ebe7cf commented

Replying to [comment:1 dkocher]:

This is caused by verifying the checksum of the downloaded file. Related to #10215.

Unfortunately this can not be the reason, because (as the ticket states) the behaviour occurs also on uploads where the checksum is calculated before the transfer.

So I did some more testing (on Windows using the Cyberduck GUI and the file:// protocol) and can provide more, hopefully helpful insights:

  • The waiting time only occurs when working with cryptomator vaults, not when working directly with an unencrypted local folder.
  • The waiting time scales (quadratically) with the number of files, not the amount of data:
  • 10.000 MB in 3 files produced no noticeable waiting time.
  • 0,8 MB in 100 files produced about 1 second of waiting time.
  • 1,6 MB in 200 files produced about 5 seconds of waiting time.
  • 3,2 MB in 400 files produced about 20 seconds of waiting time.
  • 6,4 MB in 800 files produced about 82 seconds of waiting time.
  • I did not wait for the job of 80 MB in 10.000 files to finish, but the waiting time calculates to about 3,5 hours given the observed quadratic growth.

My assumption is that for each file transferred (either up- or downloaded) from/to a cryptomator vault some data structure is left over and has to be cleaned up after the transfer is finished and that cleanup is done in an inefficient manner.

@cyberduck
Copy link
Collaborator Author

9ebe7cf commented

I changed the subject and component of this ticket to reflect my new findings.

@cyberduck
Copy link
Collaborator Author

@ylangisc commented

In fd67d33.

@iterate-ch iterate-ch locked as resolved and limited conversation to collaborators Nov 26, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@dkocher dkocher

Labels
bug cryptomator Cryptomator Vault fixed
Projects
None yet
Milestone
6.9.0
Development

No branches or pull requests
2 participants
@dkocher @cyberduck