Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Large memory usage for downloads with segmented transfer option #11151

Closed
cyberduck opened this issue Aug 20, 2020 · 31 comments
Closed

Large memory usage for downloads with segmented transfer option #11151

cyberduck opened this issue Aug 20, 2020 · 31 comments

Comments

@cyberduck
Copy link
Collaborator

28c5a3d created the issue

The two latest versions is eating up all memory, and this is tested under macOS 10.15.6 and also latest macOS Big Sur public beta.

After I download 5GB the app is using 10GB of memory, after 10GB of download it using 20GB of memory.


Attachments

@cyberduck
Copy link
Collaborator Author

@dkocher commented

What protocol are you connecting with?

@cyberduck
Copy link
Collaborator Author

7bca7c8 commented

I have the same issue - downloading several GBs via regular FTP on Mac OS 10.15.7 (19H2). Ram usage seems to be similar to amount of data downloaded, eventually Mac OS will say I'm out of memory and I'll have to force close Cyberduck.

Eventually I can download everything, takes multiple crashes and reopens.

Cyberduck Version 7.6.2 (33520)

@cyberduck
Copy link
Collaborator Author

28c5a3d commented

This is still in the latest version 7.8.2 (34203)!! Put 100GB+ for download from my home server, come home to find my computer was turned off. 85GB was transfered and then Cyberduck killed my computer.

When will this annoying bug be resolved?

Edit: Have updated the ticket from 7.5.1 to 7.8.2!

@cyberduck
Copy link
Collaborator Author

@dkocher commented

We are unable to reproduce the problem with a 20GB transfer using both FTP and SFTP.

@cyberduck
Copy link
Collaborator Author

28c5a3d commented

Replying to [comment:4 dkocher]:

We are unable to reproduce the problem with a 20GB transfer using both FTP and SFTP.

That's strange, tried yesterday with a SFTP transfer of a couple of gigs. After 700MB, Cyberduck was using 2GB of RAM.

@cyberduck
Copy link
Collaborator Author

40a0569 commented

I updated to Version 7.8.3 (34411) a couple of days ago on my Mac with BigSur 11.2.2 and I see a huge increase of RAM usage after starting a file transfer with SFTP. It's about 10MB or RAM increase every 5-10 seconds. My Mac started to complain at 62GB of RAM usage. On top of that the transfer is very slow 1MB/sec instead of 6MB/sec. I didn't have these issues with the previous version and I update each time Cyberduck suggests me a new update.

Is there a way to downgrade to the previous version?

Thanks

@cyberduck
Copy link
Collaborator Author

40a0569 commented

I downgraded to 7.8.2 and it works as intended. No RAM increase and usual transfer speeds. So my guess is that 7.8.3 introduced a major bug.

I want to mention also that using the Mac installer for 7.8.2 over an existing 7.8.3 doesn't work. It says that it was installed but the version is still 7.8.3. I had to manually delete 7.8.3 from the Applications folder and install 7.8.2 to get it to work.

@cyberduck
Copy link
Collaborator Author

3144b53 commented

Mac Mini M1 on Mac OS 11.2.3 with CyberDuck 7.8.3 (34411)
Connecting via SFTP using port 2222
Having the memory hogging problem. The ram usage starts increasing steadily as the download progresses. Usually can't make it through a 2GB without having to force-quit and resume the download, sometimes more than once. Haven't noticed yet if the memory usage goes up only with time or only with amount downloaded.

@cyberduck
Copy link
Collaborator Author

28c5a3d commented

Updated to latest 7.8.3 this morning, still hogging memory!

First run up to 37GB when I was forced to quit the app, it hogged 20,18GB of RAM.

Second run, a resume of remaining 20.7GB. 7.9GB transfer consumed 15.01GB RAM!

The first run it consumed RAM half the amount of data, the second run it consumed double the amount of data in RAM.

Third run, closed the app before and resume the rest of 10.8GB. The app consumed 2,59GB of RAM when downloaded 603.4MB!

This memory hogging has been with us for a long time now.

@cyberduck
Copy link
Collaborator Author

700885e commented

I have the same issue. After updating to 7.8.3 from 7.8.2 on OS X 11.2.3, Cyberduck will continue to use more and more RAM (getting up to 60 GB or so) before causing my system to shutdown. This has happned repeatedly over the last day, making it impossible to download large files. This is over SFTP.

Reverting to 7.8.2 resolved this issue so y'all definitely put in a memory leak or something in 7.8.3.

@cyberduck
Copy link
Collaborator Author

28c5a3d commented

Replying to [comment:11 albill]:

I have the same issue. After updating to 7.8.3 from 7.8.2 on OS X 11.2.3, Cyberduck will continue to use more and more RAM (getting up to 60 GB or so) before causing my system to shutdown. This has happned repeatedly over the last day, making it impossible to download large files. This is over SFTP.

Reverting to 7.8.2 resolved this issue so y'all definitely put in a memory leak or something in 7.8.3.

The memory bug has been with us since at least 7.5.1 to latest version. :/

For smal transfers it works, but with larger transfers it's a NO GO!

@cyberduck
Copy link
Collaborator Author

700885e commented

Replying to [comment:12 Tideman]:

The memory bug has been with us since at least 7.5.1 to latest version. :/

For smal transfers it works, but with larger transfers it's a NO GO!

I've routinely been downloading large (25 GB+) files fine until 7.8.3.

@cyberduck
Copy link
Collaborator Author

40a0569 commented

I've routinely been downloading large (25 GB+) files fine until 7.8.3

I too have been downloading 5GB files daily with 7.8.2 without issues.

@cyberduck
Copy link
Collaborator Author

28c5a3d commented

Replying to [comment:14 fjanon]:

I've routinely been downloading large (25 GB+) files fine until 7.8.3

I too have been downloading 5GB files daily with 7.8.2 without issues.

7.8.2 was hogging memory for me, and Cyberduck has been like this since 7.5.1 when I first filled the bug report. :/

@cyberduck
Copy link
Collaborator Author

14d47ec commented

Same issue for me on a Mac Mini M1, 16GB memory, OS 11.1. Trying to download a massive 216GB server image from Rackspace. Found my computer had restarted after the first attempt and second attempts. Third attempt, it had paused Cyberduck and showed it was consuming 61GB of physical and virtual memory. I installed Cyberduck v 7.5.0 and the download completed without complication. Monitoring that, Cyberduck's memory increased from ~500MB to ~600MB, but nothing like the previous issues.

@cyberduck
Copy link
Collaborator Author

@ylangisc commented

I can reproduce the issue with segmented transfers enabled. Can you confirm that disabling segmenting downloads (Preferences -> Transfers -> General) resolves the issue?

@cyberduck
Copy link
Collaborator Author

14d47ec commented

Replying to [comment:17 yla]:

I can reproduce the issue with segmented transfers enabled. Can you confirm that disabling segmenting downloads (Preferences -> Transfers -> General) resolves the issue?

Disabling segmenting downloads DOES seem to resolve it. I tried my 216GB download again and ran it a third of the way, memory usage never exceeded 600MB. BUT the download speed was very very slow, less than 3Mb/sec, compared to 10+Mb/sec before. This is from Rackspace Cloud btw.

@cyberduck
Copy link
Collaborator Author

@ylangisc commented

Thanks for confirming. We are working on a fix to close the memory leak for segmented downloads.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

#11049 closed as duplicate.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

#11616 closed as duplicate.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

#11622 closed as duplicate.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Please try to disable Preferences → Transfers → Generals → Downloads → Segmeneted downloads with multiple connections per file as a workaround.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

#11628 closed as duplicate.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Improved SFTP transfer performance with segmented downloads option enabled in 39bd906 to disable read ahead past segment length.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

In 1b7df51. Icon of segment file was updated continuously due to false assertion of segment length.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Reverted 6fd5fb6 no longer required with fix to only update icon 10 times per segment at most during a transfer. Auto release pool is drained after the transfer finishes.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Can you please confirm this issue ix fixed in the current snapshot build
by updating from within Cyberduck in ''Preferences → Update →
Automatically check for updates in → Snapshot Builds''.

@cyberduck
Copy link
Collaborator Author

01e553c commented

Seems fixed in snapshot 7.8.5 (34490) on macOS 11.2.3! Had the same issue; crashed my Mac.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Milestone renamed

@cyberduck
Copy link
Collaborator Author

28c5a3d commented

The bug is somehowed fixed... But the fix added a new bug!

Now Cyberduck is using 100-160% of the CPU when uploading instead and the fans is maxing out! :/

Was uploading and wonder why the heck my MacBook Pro 16" was maxing out on the fans, fired up Activity Monitor only to see Cyberduck in top with 106% CPU.

@cyberduck
Copy link
Collaborator Author

@dkocher commented

Replying to [comment:32 Tideman]:

The bug is somehowed fixed... But the fix added a new bug!

Now Cyberduck is using 100-160% of the CPU when uploading instead and the fans is maxing out! :/

Was uploading and wonder why the heck my MacBook Pro 16" was maxing out on the fans, fired up Activity Monitor only to see Cyberduck in top with 106% CPU.
Please open a separate issue as this is unrelated. This tickets was about memory usage with segmented downloads.

@iterate-ch iterate-ch locked as resolved and limited conversation to collaborators Nov 27, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

1 participant