Cyberduck Mountain Duck CLI

#10604 new enhancement

Slow execution due to low entropy in /dev/random

Reported by: jkvbe Owned by:
Priority: normal Milestone:
Component: cli Version: 6.9.2
Severity: normal Keywords:
Cc: Architecture:
Platform: Linux

Description (last modified by dkocher)

  • OS: Ubuntu 16.04
  • Java: openjdk version "11.0.1" 2018-10-16
  • Cyberduck version: 6.9.1 (29776)

duck -h takes more than 5 minutes to execute. I have the impression it runs against a timeout, but I don't see what resources duck needs to print the help information.

This happens for each duck command.

Change History (13)

comment:1 Changed on Feb 8, 2019 at 12:11:56 PM by jkvbe

  • Description modified (diff)

comment:2 Changed on Feb 8, 2019 at 1:01:19 PM by jkvbe

Note: in rare cases the response is immediate

comment:3 Changed on Feb 14, 2019 at 11:05:18 AM by dkocher

  • Component changed from core to cli

comment:4 Changed on Feb 14, 2019 at 1:50:22 PM by dkocher

  • Description modified (diff)
  • Summary changed from Even duck -h takes minutes to execute to Running program takes minutes to execute

comment:5 in reply to: ↑ description Changed on Feb 22, 2019 at 2:20:39 PM by little

Replying to jkvbe:

It takes about 38 minutes.

$ time duck --help
Usage:duck [options...]


URLs must ...
... Purchase a donation key to support the development of this software.

real	38m37.287s
user	0m2.524s
sys	0m0.564s
$

I confirm behaviour since version 6.9.1.29776. Current testing with ubuntu 18.04 and version 6.9.3.30061 is still the same

On version 6.8.3.29107 duck CLI was running fine. It is independent from OS version (16.04 or 18.04). No java installed since this is no dependency.

  • OS: Ubuntu 16.04
  • Java: openjdk version "11.0.1" 2018-10-16
  • Cyberduck version: 6.9.1 (29776)

duck -h takes more than 5 minutes to execute. I have the impression it runs against a timeout, but I don't see what resources duck needs to print the help information.

This happens for each duck command.

comment:6 Changed on Mar 27, 2019 at 11:57:51 PM by frwessling

Hey, we ran into this today and it appears as though the issues is with the use of /dev/random in certain calls that are being made.

If the system in use is low on entropy then /dev/random will block until there are enough bits to satisfy whatever request is being made.

How to test
On a system w/low entropy (Linux VMs that don't see a lot of CPU I/O are good candidates):
In one terminal or screen session run:
time duck -h

In another run:
lsof |grep duck|grep random
You will see calls to both /dev/urandom and /dev/random

Now on that same system (as root) do the following:
(if necessary "ctrl-c" your previously running duck session)
"mv /dev/random /dev/random.orig"
"cp -a /dev/urandom /dev/random"

Once you've replaced /dev/random with /dev/urandom run:
"time duck -h"
It should return a *whole* lot quicker.
In our case it went from 11m to 8s.
Don't forget to put your system back the way it was:
"rm -f /dev/random"
"mv /dev/random.orig /dev/random"

You can tell how much entropy is available (in bits) for /dev/random by:
"cat /proc/sys/kernel/random/entropy_avail"

In our test case where "duck -h" took 11m to run, we had about 118 bits of available entropy.

Some Suggestions for the devs:

  1. Only be use /dev/random (or libraries that invoke it) if absolutely necessary; otherwise use /dev/urandom
  2. Provide a run-time option that would allow the user to choose the use /dev/urandom exclusively
Last edited on Mar 28, 2019 at 12:07:18 AM by frwessling (previous) (diff)

comment:7 Changed on Mar 28, 2019 at 7:32:19 AM by dkocher

  • Summary changed from Running program takes minutes to execute to Slow execution due to low entropy in /dev/random
  • Type changed from defect to enhancement

comment:8 Changed on Mar 28, 2019 at 7:34:50 AM by dkocher

Because of this we actually run with -Djava.security.egd=file:/dev/./urandom. We will have to look into when/what libraries still use /dev/random directly.

comment:9 Changed on Mar 28, 2019 at 9:59:59 AM by dkocher

  • Platform set to Linux

comment:10 follow-up: Changed on Mar 28, 2019 at 1:41:30 PM by frwessling

Suggestion for End-Users:

If you need duck to work now, and cannot wait for the devs (sometimes this happens).

You can increase the entropy of your systems' /dev/random by using something like haveged.

This acts as a quite decent source of randomness to feed /dev/random in a very efficient (non-CPU-intensive) way. And works excellently for VM/containers that have no access to more traditional sources of randomness.

There are some other projects and methods but something like haveged should be fairly available and infrastructure independent, so no need to worry about virtio / kvm / qemu ...

Also, a big thank you to the cyberduck team for their support.

Edit: To add running duck -h with haveged running with a low-watermark of 2000 (general recommendations but the debian/ubuntu default of 1024 is probably good enough for this case) caused the execution time to go from the 11m area to 3.5s.

Last edited on Mar 28, 2019 at 1:44:36 PM by frwessling (previous) (diff)

comment:11 in reply to: ↑ 10 ; follow-up: Changed on Mar 28, 2019 at 2:42:20 PM by dkocher

Replying to frwessling:

Thanks for sharing this!

comment:12 in reply to: ↑ 11 Changed on Mar 28, 2019 at 5:21:34 PM by frwessling

Replying to dkocher:

Replying to frwessling:

Thanks for sharing this!

My pleasure, glad to help.

Note: See TracTickets for help on using tickets.