New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Out of memory error when uploading larger files #10392
Comments
Ticket retargeted after milestone closed |
I tried again with Version 6.6.3 (28581 latest nightly build) and noticed a different behavior: Could upload a CentOS 7 DVD image (4,4GB) without any issues. Later I tried a VDI file from my VM with a total size of 39,3GB. After 13.0GB (14.0005.158.086 bytes, 33%) the upload stucks. The file transfer window say that we have a transfer rate of 345.2 KB/s, but this doesn't seem to be true: The transfered file size doesn't increase and the taskmanager show me a network usage from cyberduck of 0 Mbit/s. Even after some hours later, this behavior doesn't change. But it's not possible to stop or delete the file transfer any more. Seems like Cyberduck get stuck here. To stop the upload I have to exit Cyberduck, confirm the killing of running transfers and start it again. UPDATE: After uploading the same file again, the behavior of Cyberduck is the same, but now after 11.0GB instead of 13GB. |
Since large files doesn't work well and I'd assume that OneDrive has file size limits, too (business has 15GB https://support.microsoft.com/en-us/help/3125202/restrictions-and-limitations-when-you-sync-files-and-folders), my workaround is to create 7z archives with a maximum size of 8GB per file. So it's possible to avoid those problems on big files and the total size shrinks by compression, too. So one of my larger VM could be split in 2x8GB and 1x 0,8GB segments. I tried to upload all three files using Cyberduck and now the upload stucks at 1.7GB from 16.8GB total. The transfer window show me a rate of 40.8 KB/sec and about 110 hours remaining. Cyberducks main window is useable but the file transfer popup seems completely frozen, can't even move the window any more. After restarting Cyberduck, I see that the smallest part #3 (0,8GB) seems uploaded successfully, but part 1 and 2 (8GB each) not. I re-uploaded part #1 and #2 and a few percent before they were transfered complete, I got an error dialog "Connection failed: Java.lang.OutOfMemoryError". After this error, the file transfer window doesn't freeze. It was possible to delete the failed transfer. Try number 3: Instead of uploading both files (#1 and #2), I first uploaded #2 and then #1, this works. |
I'd like to know more about the time Cyberduck fails. I have been able to upload huge amounts of data to OneDrive (32 GB splitted into 8 GB files) without Cyberduck yet crashing (at the time of writing this test does continue to run, currently 21 GB uploaded). So I have checked some things, which may indicate this OutOfMemory exception which is not reproducible on our systems could be harder to investigate than others. So, my questions are:
|
Thanks for taking care about this issue. At the time of writing, my system had 24GB memory, so it should not be a real lack of physical memory. But I have a reconnect from my provider every 24 hours. So when I start uploading in the evening like now, my connection got corrupted in a few hours for some seconds. This shouldn't been a big issue for Cyberduck, or am I wrong? Replying to [comment:8 jmalek]:
|
I ran two tests: Test 1 Test 2 The cyberduck.log in appdata folder has an entry near 3PM:
Again you can see that I got an OutOfMemoryException diretly before the DNS resolve error. This can't be related to my memory since my machine has 32GB memory. Currently about 22GB of them are usable. BUT since Cyberduck is a 32 Bit application, it can't allocate more than ~4GB for a single process. To be sure, I looked in the logs of my DSL router. There were no entrys in the hole day, so the WAN connection seems stable. This doesn't automatically mean that there were no outages by the DNS server from the ISP. But on the other hand, even if there was a DNS failture: Why does this abort the hole upload process? Would expect that in this case Cyberduck wait a minute and re-try the upload process. And I don't really belive that the DNS issue is the cause since the OutOfMemoryException was thrown only milliseconds before. It seems that for some reason the app ran out of memory and the DNS error is some aftereffect of this. |
Thanks for retesting. So: This is a major issue I'm currently investigating but it might take time as this is not easily resolved. There is not "just" the 4 GB limit but also the limitation that if Cyberduck allocates space outside the first 4 GB of system memory it will crash. The IP disconnect should not affect Cyberduck uploading files as it just repeats until it can upload successfully. Just for my records: Out of 32 GB you had 22 GB left, then 10 GB had been in use which could lead to this error. |
Do you have any idea why there are problems with allocating memory outside the first 4GB? |
I try to upload the hole files of a VM (VHDD, snapshots and so on). The files are up to 11.6GB big. After ~2.4 GB the upload stops and I got an Java.lang.OutOfMemoryError. The debug log is empty, maybe because the upload was done in an external window. I'm using OneDrive as upload target.
My assumption was that Cyberduck try to buffer the hole file in RAM, while a default JVM doesn't have enough RAM assigned. But I found a lot of quite old tickets, which say that those buffering of hole files was fixed years ago. Since Cyberduck seems to be a mix of .NET and Java and I'm not a Java developer, I don't know to to increase those limits for Cyberduck. Maybe the default limit is to low for the default buffer?
Attachments
Screenshot_3.jpg
(48.9 KiB)The text was updated successfully, but these errors were encountered: