Cyberduck Mountain Duck CLI

#11049 closed defect (duplicate)

High memory usage when downloading

Reported by: exmate Owned by: yla
Priority: high Milestone: 7.8.5
Component: sftp Version: 7.3.1
Severity: major Keywords:
Cc: Architecture: Intel
Platform: macOS 10.15

Description

Cyberduck has been using an insane amount of memory since 7.3.1 on MacOS when downloading files using the SFTP protocol. I have tried FTP and the memory usage is fine. The bug still exists in the latest snapshot build (7.3.2 (32852)).

Attachments (3)

Memory #1.png (97.7 KB) - added by exmate on May 8, 2020 at 2:01:31 AM.
Memory #2.png (101.1 KB) - added by exmate on May 8, 2020 at 2:01:38 AM.
Memory_Leak.mov (18.8 MB) - added by rootie on May 26, 2020 at 5:01:20 PM.
Video that shows the memory leak

Change History (18)

Changed on May 8, 2020 at 2:01:31 AM by exmate

Changed on May 8, 2020 at 2:01:38 AM by exmate

comment:1 Changed on May 11, 2020 at 11:18:02 AM by rootie

I've just reported something similar with #11051 (https://trac.cyberduck.io/ticket/11051).

Must be a new issue as I've never encountered this before.

comment:2 Changed on May 19, 2020 at 12:27:40 PM by lvogt

  • Owner set to dkocher

comment:3 Changed on May 24, 2020 at 11:04:24 AM by dkocher

  • Component changed from core to sftp
  • Summary changed from High Memory Usage Since 7.3.1 on MacOS when Downloading from SFTP to High memory usage when downloading

comment:4 Changed on May 26, 2020 at 8:16:48 AM by yla

I have tried to replicate the issue with different file sizes and protocols but was not successful unfortunately. As long as I'm not able to replicate it the only way to find the cause would be to create a heap dump of the Cyberduck process. Not sure how technical you are but the steps would as follows:

  • install OpenJDK 13 - JDK version (if not yet available on your system)
  • jmap -dump:file=cyberduck-dump.hprof [PID of Cyberduck process]
    
  • provide a link to the dump file

The dump file could be quite large. Let me know if you want me to share a link where you can drop the file.

comment:5 Changed on May 26, 2020 at 5:00:48 PM by rootie

I'm quite technical so I'm sure we can find that issue together. Before I install this on my productive system I'd like to share a video I've made showing the memory leak. I'll try to exactly explain again what my setup is and what I've done to reproduce it.

  • Updated to the newest Cyberduck version 7.4.0
  • Logged onto OneDrive and selected Sicherungen\Eigene Dateien\Musik - this is an online only vault and NOT synched to my hard drive
  • Inside the "Musik" folder there are 3 sub folders and within those sub folders there are tons of other folders with singles and albums. Basically this is my (legal!) backup of all my music sorted in sub directories
  • I drag and dropped the "Musik" main folder onto a finder window. I did NOT select single files - I want to have the whole "Musik" folder downloaded
  • The preparation of the files and directories starts which took about 30 minutes. During that time no excessive memory usage was there
  • After the preparation the download has started and as soon as this happens the memory usage goes up

I monitored it with the terminal and the "top" command. Maybe you did not try with a directory or drag/drop and that's why you weren't able to reproduce it.

I can offer you a TeamViewer session together with a telephone call. This might ease up things significantly if you're still not able to reproduce this.

Best regards rootie

Changed on May 26, 2020 at 5:01:20 PM by rootie

Video that shows the memory leak

comment:6 Changed on May 26, 2020 at 5:02:09 PM by rootie

One addition: As soon as i click on "Stop" on the transfer window to interrupt the download, the memory usage is back to normal. I think the garbage collector is then executed.

comment:7 Changed on May 31, 2020 at 7:37:42 AM by rootie

Any news on this? Please give an update on whether or not you were able to reproduce that issue.

comment:8 Changed on Jun 5, 2020 at 1:14:02 PM by dkocher

  • Milestone set to 8.0

#11051 closed as duplicate.

comment:9 Changed on Jun 5, 2020 at 1:14:10 PM by dkocher

  • Owner changed from dkocher to yla

comment:10 Changed on Nov 2, 2020 at 1:24:43 PM by dkocher

  • Milestone changed from 8.0 to 7.7.0

Milestone renamed

comment:11 Changed on Nov 2, 2020 at 1:24:43 PM by dkocher

  • Milestone changed from 7.7.0 to 8.0.0

Ticket retargeted after milestone closed

comment:12 Changed on Nov 2, 2020 at 1:26:14 PM by dkocher

  • Milestone changed from 8.0.0 to 8.0

Milestone renamed

comment:13 Changed on Jan 13, 2021 at 5:58:23 PM by bingocaller

Cyberduck 7.8 is still using excessive RAM for SFTP transfers. RAM usage grows the longer Cyberduck is open and transferring files. One of my organization's users has seen up to 65 GB RAM in use by Cyberduck after transferring a large amount of files over sftp.

comment:14 Changed on Mar 24, 2021 at 3:34:05 PM by dkocher

  • Milestone changed from 8.0 to 7.8.4
  • Resolution set to duplicate
  • Status changed from new to closed

Following up in #11151.

comment:15 Changed on Apr 1, 2021 at 7:16:22 AM by dkocher

  • Milestone changed from 7.8.4 to 7.8.5

Milestone renamed

Note: See TracTickets for help on using tickets.