Cyberduck Mountain Duck CLI

#11151 closed defect (fixed)

Large memory usage for downloads with segmented transfer option

Reported by: Tideman Owned by:
Priority: highest Milestone: 7.8.5
Component: core Version: 7.8.3
Severity: blocker Keywords:
Cc: Architecture: Intel
Platform: macOS 11

Description

The two latest versions is eating up all memory, and this is tested under macOS 10.15.6 and also latest macOS Big Sur public beta.

After I download 5GB the app is using 10GB of memory, after 10GB of download it using 20GB of memory.

Attachments (2)

Screenshot 2020-08-20 at 08.06.34.png (33.9 KB) - added by Tideman on Aug 20, 2020 at 6:43:12 AM.
Screenshot 2021-03-20 at 05.42.18.png (100.7 KB) - added by Tideman on Mar 20, 2021 at 8:09:46 AM.

Download all attachments as: .zip

Change History (35)

comment:1 Changed on Oct 10, 2020 at 8:51:01 PM by dkocher

  • Summary changed from Version 7.5.1 (33324) eating up all memory and starts swapping! to Large memory usage when downloading

What protocol are you connecting with?

comment:2 Changed on Oct 22, 2020 at 1:03:42 AM by paulbroomfield

I have the same issue - downloading several GBs via regular FTP on Mac OS 10.15.7 (19H2). Ram usage seems to be similar to amount of data downloaded, eventually Mac OS will say I'm out of memory and I'll have to force close Cyberduck.

Eventually I can download everything, takes multiple crashes and reopens.

Cyberduck Version 7.6.2 (33520)

comment:3 Changed on Feb 14, 2021 at 2:58:12 PM by Tideman

  • Version changed from 7.5.1 to 7.8.2

This is still in the latest version 7.8.2 (34203)!! Put 100GB+ for download from my home server, come home to find my computer was turned off. 85GB was transfered and then Cyberduck killed my computer.

When will this annoying bug be resolved?

Edit: Have updated the ticket from 7.5.1 to 7.8.2!

comment:4 follow-up: Changed on Feb 17, 2021 at 4:04:50 PM by dkocher

We are unable to reproduce the problem with a 20GB transfer using both FTP and SFTP.

comment:5 in reply to: ↑ 4 Changed on Feb 19, 2021 at 3:22:03 AM by Tideman

Replying to dkocher:

We are unable to reproduce the problem with a 20GB transfer using both FTP and SFTP.

That's strange, tried yesterday with a SFTP transfer of a couple of gigs. After 700MB, Cyberduck was using 2GB of RAM.

comment:6 Changed on Mar 18, 2021 at 9:42:42 AM by fjanon

I updated to Version 7.8.3 (34411) a couple of days ago on my Mac with BigSur 11.2.2 and I see a huge increase of RAM usage after starting a file transfer with SFTP. It's about 10MB or RAM increase every 5-10 seconds. My Mac started to complain at 62GB of RAM usage. On top of that the transfer is very slow 1MB/sec instead of 6MB/sec. I didn't have these issues with the previous version and I update each time Cyberduck suggests me a new update.

Is there a way to downgrade to the previous version?

Thanks

comment:7 Changed on Mar 18, 2021 at 10:45:21 AM by fjanon

I downgraded to 7.8.2 and it works as intended. No RAM increase and usual transfer speeds. So my guess is that 7.8.3 introduced a major bug.

I want to mention also that using the Mac installer for 7.8.2 over an existing 7.8.3 doesn't work. It says that it was installed but the version is still 7.8.3. I had to manually delete 7.8.3 from the Applications folder and install 7.8.2 to get it to work.

comment:8 Changed on Mar 20, 2021 at 3:04:02 AM by alex.shultz

Mac Mini M1 on Mac OS 11.2.3 with CyberDuck 7.8.3 (34411) Connecting via SFTP using port 2222 Having the memory hogging problem. The ram usage starts increasing steadily as the download progresses. Usually can't make it through a 2GB without having to force-quit and resume the download, sometimes more than once. Haven't noticed yet if the memory usage goes up only with time or only with amount downloaded.

comment:9 Changed on Mar 20, 2021 at 5:33:25 AM by Tideman

Updated to latest 7.8.3 this morning, still hogging memory!

First run up to 37GB when I was forced to quit the app, it hogged 20,18GB of RAM.

Second run, a resume of remaining 20.7GB. 7.9GB transfer consumed 15.01GB RAM!

The first run it consumed RAM half the amount of data, the second run it consumed double the amount of data in RAM.

Third run, closed the app before and resume the rest of 10.8GB. The app consumed 2,59GB of RAM when downloaded 603.4MB!

This memory hogging has been with us for a long time now.

comment:10 Changed on Mar 20, 2021 at 5:34:39 AM by Tideman

  • Platform changed from macOS 10.15 to macOS 11
  • Version changed from 7.8.2 to 7.8.3

comment:11 follow-up: Changed on Mar 20, 2021 at 5:31:41 PM by albill

I have the same issue. After updating to 7.8.3 from 7.8.2 on OS X 11.2.3, Cyberduck will continue to use more and more RAM (getting up to 60 GB or so) before causing my system to shutdown. This has happned repeatedly over the last day, making it impossible to download large files. This is over SFTP.

Reverting to 7.8.2 resolved this issue so y'all definitely put in a memory leak or something in 7.8.3.

comment:12 in reply to: ↑ 11 ; follow-up: Changed on Mar 21, 2021 at 4:26:37 AM by Tideman

Replying to albill:

I have the same issue. After updating to 7.8.3 from 7.8.2 on OS X 11.2.3, Cyberduck will continue to use more and more RAM (getting up to 60 GB or so) before causing my system to shutdown. This has happned repeatedly over the last day, making it impossible to download large files. This is over SFTP.

Reverting to 7.8.2 resolved this issue so y'all definitely put in a memory leak or something in 7.8.3.

The memory bug has been with us since at least 7.5.1 to latest version. :/

For smal transfers it works, but with larger transfers it's a NO GO!

comment:13 in reply to: ↑ 12 Changed on Mar 21, 2021 at 4:46:26 AM by albill

Replying to Tideman:

The memory bug has been with us since at least 7.5.1 to latest version. :/

For smal transfers it works, but with larger transfers it's a NO GO!

I've routinely been downloading large (25 GB+) files fine until 7.8.3.

comment:14 follow-up: Changed on Mar 21, 2021 at 8:13:24 AM by fjanon

I've routinely been downloading large (25 GB+) files fine until 7.8.3

I too have been downloading 5GB files daily with 7.8.2 without issues.

comment:15 in reply to: ↑ 14 Changed on Mar 21, 2021 at 4:58:52 PM by Tideman

Replying to fjanon:

I've routinely been downloading large (25 GB+) files fine until 7.8.3

I too have been downloading 5GB files daily with 7.8.2 without issues.

7.8.2 was hogging memory for me, and Cyberduck has been like this since 7.5.1 when I first filled the bug report. :/

comment:16 Changed on Mar 22, 2021 at 3:37:34 PM by davyM

Same issue for me on a Mac Mini M1, 16GB memory, OS 11.1. Trying to download a massive 216GB server image from Rackspace. Found my computer had restarted after the first attempt and second attempts. Third attempt, it had paused Cyberduck and showed it was consuming 61GB of physical and virtual memory. I installed Cyberduck v 7.5.0 and the download completed without complication. Monitoring that, Cyberduck's memory increased from ~500MB to ~600MB, but nothing like the previous issues.

comment:17 follow-up: Changed on Mar 23, 2021 at 10:27:50 AM by yla

I can reproduce the issue with segmented transfers enabled. Can you confirm that disabling segmenting downloads (Preferences -> Transfers -> General) resolves the issue?

comment:18 in reply to: ↑ 17 Changed on Mar 24, 2021 at 3:19:05 AM by davyM

Replying to yla:

I can reproduce the issue with segmented transfers enabled. Can you confirm that disabling segmenting downloads (Preferences -> Transfers -> General) resolves the issue?

Disabling segmenting downloads DOES seem to resolve it. I tried my 216GB download again and ran it a third of the way, memory usage never exceeded 600MB. BUT the download speed was very very slow, less than 3Mb/sec, compared to 10+Mb/sec before. This is from Rackspace Cloud btw.

comment:19 Changed on Mar 24, 2021 at 8:31:03 AM by yla

Thanks for confirming. We are working on a fix to close the memory leak for segmented downloads.

comment:20 Changed on Mar 24, 2021 at 3:33:49 PM by dkocher

  • Milestone set to 7.8.4

comment:21 Changed on Mar 24, 2021 at 3:34:13 PM by dkocher

#11049 closed as duplicate.

comment:22 Changed on Mar 24, 2021 at 3:36:23 PM by dkocher

#11616 closed as duplicate.

comment:23 Changed on Mar 24, 2021 at 3:41:04 PM by dkocher

#11622 closed as duplicate.

comment:24 Changed on Mar 24, 2021 at 3:47:28 PM by dkocher

Please try to disable Preferences → Transfers → Generals → Downloads → Segmeneted downloads with multiple connections per file as a workaround.

comment:25 Changed on Mar 25, 2021 at 9:18:54 AM by dkocher

#11628 closed as duplicate.

comment:26 Changed on Mar 30, 2021 at 1:16:24 PM by dkocher

Improved SFTP transfer performance with segmented downloads option enabled in r50846 to disable read ahead past segment length.

comment:27 Changed on Mar 30, 2021 at 1:19:11 PM by dkocher

  • Resolution set to fixed
  • Status changed from new to closed

In r50833. Icon of segment file was updated continuously due to false assertion of segment length.

comment:28 Changed on Mar 30, 2021 at 1:20:49 PM by dkocher

Reverted r50844 no longer required with fix to only update icon 10 times per segment at most during a transfer. Auto release pool is drained after the transfer finishes.

comment:29 Changed on Mar 30, 2021 at 6:59:01 PM by dkocher

Can you please confirm this issue ix fixed in the current snapshot build by updating from within Cyberduck in Preferences → Update → Automatically check for updates in → Snapshot Builds.

comment:30 Changed on Mar 31, 2021 at 12:49:25 PM by marcuse7

Seems fixed in snapshot 7.8.5 (34490) on macOS 11.2.3! Had the same issue; crashed my Mac.

comment:31 Changed on Apr 1, 2021 at 7:16:22 AM by dkocher

  • Milestone changed from 7.8.4 to 7.8.5

Milestone renamed

comment:32 follow-up: Changed on Apr 10, 2021 at 3:04:24 AM by Tideman

  • Resolution fixed deleted
  • Status changed from closed to reopened

The bug is somehowed fixed... But the fix added a new bug!

Now Cyberduck is using 100-160% of the CPU when uploading instead and the fans is maxing out! :/

Was uploading and wonder why the heck my MacBook Pro 16" was maxing out on the fans, fired up Activity Monitor only to see Cyberduck in top with 106% CPU.

comment:33 in reply to: ↑ 32 Changed on Apr 12, 2021 at 8:34:55 AM by dkocher

  • Resolution set to fixed
  • Status changed from reopened to closed
  • Summary changed from Large memory usage when downloading to Large memory usage for downloads with segmented transfer option

Replying to Tideman:

The bug is somehowed fixed... But the fix added a new bug!

Now Cyberduck is using 100-160% of the CPU when uploading instead and the fans is maxing out! :/

Was uploading and wonder why the heck my MacBook Pro 16" was maxing out on the fans, fired up Activity Monitor only to see Cyberduck in top with 106% CPU.

Please open a separate issue as this is unrelated. This tickets was about memory usage with segmented downloads.

Note: See TracTickets for help on using tickets.