Opened on Feb 8, 2019 at 12:10:49 PM
Last modified on Jan 29, 2021 at 7:44:45 AM
#10604 new enhancement
Slow execution due to low entropy in /dev/random
Reported by: | jkvbe | Owned by: | |
---|---|---|---|
Priority: | normal | Milestone: | |
Component: | cli | Version: | 6.9.2 |
Severity: | normal | Keywords: | |
Cc: | Architecture: | ||
Platform: | Linux |
Description (last modified by dkocher)
- OS: Ubuntu 16.04
- Java: openjdk version "11.0.1" 2018-10-16
- Cyberduck version: 6.9.1 (29776)
duck -h takes more than 5 minutes to execute. I have the impression it runs against a timeout, but I don't see what resources duck needs to print the help information.
This happens for each duck command.
Change History (13)
comment:1 Changed on Feb 8, 2019 at 12:11:56 PM by jkvbe
- Description modified (diff)
comment:2 Changed on Feb 8, 2019 at 1:01:19 PM by jkvbe
comment:3 Changed on Feb 14, 2019 at 11:05:18 AM by dkocher
- Component changed from core to cli
comment:4 Changed on Feb 14, 2019 at 1:50:22 PM by dkocher
- Description modified (diff)
- Summary changed from Even duck -h takes minutes to execute to Running program takes minutes to execute
comment:5 in reply to: ↑ description Changed on Feb 22, 2019 at 2:20:39 PM by little
Replying to jkvbe:
It takes about 38 minutes.
$ time duck --help Usage:duck [options...] URLs must ... ... Purchase a donation key to support the development of this software. real 38m37.287s user 0m2.524s sys 0m0.564s $
I confirm behaviour since version 6.9.1.29776. Current testing with ubuntu 18.04 and version 6.9.3.30061 is still the same
On version 6.8.3.29107 duck CLI was running fine. It is independent from OS version (16.04 or 18.04). No java installed since this is no dependency.
- OS: Ubuntu 16.04
- Java: openjdk version "11.0.1" 2018-10-16
- Cyberduck version: 6.9.1 (29776)
duck -h takes more than 5 minutes to execute. I have the impression it runs against a timeout, but I don't see what resources duck needs to print the help information.
This happens for each duck command.
comment:6 Changed on Mar 27, 2019 at 11:57:51 PM by frwessling
Hey, we ran into this today and it appears as though the issues is with the use of /dev/random in certain calls that are being made.
If the system in use is low on entropy then /dev/random will block until there are enough bits to satisfy whatever request is being made.
How to test
On a system w/low entropy (Linux VMs that don't see a lot of CPU I/O are good candidates):
In one terminal or screen session run:
time duck -h
In another run:
lsof |grep duck|grep random
You will see calls to both /dev/urandom and /dev/random
Now on that same system (as root) do the following:
(if necessary "ctrl-c" your previously running duck session)
"mv /dev/random /dev/random.orig"
"cp -a /dev/urandom /dev/random"
Once you've replaced /dev/random with /dev/urandom run:
"time duck -h"
It should return a *whole* lot quicker.
In our case it went from 11m to 8s.
Don't forget to put your system back the way it was:
"rm -f /dev/random"
"mv /dev/random.orig /dev/random"
You can tell how much entropy is available (in bits) for /dev/random by:
"cat /proc/sys/kernel/random/entropy_avail"
In our test case where "duck -h" took 11m to run, we had about 118 bits of available entropy.
Some Suggestions for the devs:
- Only be use /dev/random (or libraries that invoke it) if absolutely necessary; otherwise use /dev/urandom
- Provide a run-time option that would allow the user to choose the use /dev/urandom exclusively
comment:7 Changed on Mar 28, 2019 at 7:32:19 AM by dkocher
- Summary changed from Running program takes minutes to execute to Slow execution due to low entropy in /dev/random
- Type changed from defect to enhancement
comment:8 Changed on Mar 28, 2019 at 7:34:50 AM by dkocher
Because of this we actually run with -Djava.security.egd=file:/dev/./urandom. We will have to look into when/what libraries still use /dev/random directly.
comment:9 Changed on Mar 28, 2019 at 9:59:59 AM by dkocher
- Platform set to Linux
comment:10 follow-up: ↓ 11 Changed on Mar 28, 2019 at 1:41:30 PM by frwessling
Suggestion for End-Users:
If you need duck to work now, and cannot wait for the devs (sometimes this happens).
You can increase the entropy of your systems' /dev/random by using something like haveged.
This acts as a quite decent source of randomness to feed /dev/random in a very efficient (non-CPU-intensive) way. And works excellently for VM/containers that have no access to more traditional sources of randomness.
There are some other projects and methods but something like haveged should be fairly available and infrastructure independent, so no need to worry about virtio / kvm / qemu ...
Also, a big thank you to the cyberduck team for their support.
Edit: To add running duck -h with haveged running with a low-watermark of 2000 (general recommendations but the debian/ubuntu default of 1024 is probably good enough for this case) caused the execution time to go from the 11m area to 3.5s.
comment:11 in reply to: ↑ 10 ; follow-up: ↓ 12 Changed on Mar 28, 2019 at 2:42:20 PM by dkocher
Replying to frwessling:
Thanks for sharing this!
comment:12 in reply to: ↑ 11 Changed on Mar 28, 2019 at 5:21:34 PM by frwessling
comment:13 Changed on Jan 29, 2021 at 7:44:45 AM by dkocher
Now documented.
Note: in rare cases the response is immediate