Ipfire crashes and loses all connectivity in and out of ipfire with CU201 and then same with CU202 when performing rclone backups to various external cloud based storage. CU200 and prior all worked just fine… Have backup logs from over a year ago. The web interface goes down as well however I am able to login locally into the CLI on the ipfire machine and reboot. After reboot everything works normally until the next rclone schedule hits.
We have the IPS and DNS firewall both active. A current screen shot of the memory foot print is below. We tried inactivating and deselecting options but the outcome is still the same. Also set up a competitor’s software on the same hardware and backups completed without error, although naturally the setup is not the same, just the hardware.
Any idea why this is happening? What logs should I review?
I will be out of the office the rest of today, and will review the status again on Monday.
I would suggest that the most likely thing is that you are running out of memory and the OOM Killer is shutting down a package, most likely unbound.
The DNS Firewall was introduced with CU201 and it was found that especially if you only have 4MiB ram and you have several of the larger DNS Firewall categories selected then the system cab run out of memory and then unbound gets killed and is not restarted.
The likelihood is that your system, without rclone running, is close to the memory limit and therefore when rclone starts it uses enough ram to go over the limit and then unbound is killed.
It was identified, between CU201 and CU202 that there are several issues with unbound that don’t look like they will be fixed in the short term. So the decision was made to move from unbound to knot-resolver in CU203. That version is now out for Testing and the ram consumption is much much better than with unbound.
The Testing release info gives more details of the improvements.
If your issue is the ram consumption, this can be seen and checked on the memory usage WUI page, then your best bet is to disable all the DNS Firewall categories until CU203 is released.
If the issue is not related to running out of memory (my best guess) then you will need to provide some log information to be able to make a better analysis.
A suggestion for the case of OOM is, to run /etc/init.d/unbound restart from a shell.
This clears the cache memory of unbound. The DNS service is down during the restart operation.
The OOM is caused by the fast_reload operation of unbound, which is called with each change to the DNS firewall config.
As a workaround till CU203 you can try to config DNS firewall at ‘right’ times only. And use the process ‘restart → config → restart’. This means two DNS downs, but do the config with necessary memory consumption only and resetting to this state after config.
A second way to solve your problem is to test CU203 and report problems, to speed up the relaease.
If you can accept the downtimes ( the ‘right’ times ), this procedure can be applied in business or critical use ( are these systems with 4GB only? ).
It does no unusual steps: config with the standard WUI. Garbage collection by restart with a standard bash script. Only difference, it doesn’t use the fast_reload function (which is buggy).
First I was remiss not describing the system setup. Rclone runs from client windows 10/11 machines or windows servers of various lineages Note: Rclone does NOT run on IPF..
Per suggestions, I attempted to remove the DNS firewall lists, Ads, Malware, and Social that were selected. Hit save, screen cleared and then “kernel panic” reboot. Logged back in and lists still indicated that they were selected, again unselected and pressed save. Screen cleared, then I lost connection with IPF/internet from the windows machine. I issued a reboot.from the CLI then rebooted windows as well. Logged back in and the DNS firewall setup windows was empty. Checked memory and it has reduced as well. Loaded " Top " on IPF and started the rclone routines. Took several pics of Top screen and all indicated OK. Couple Kernel panics later the system settled down and appeared to be performing the rclone routines. But 3 hours later into an approx 1.5 hour job TLS connections errors started appearing on the windows machine - could not again connect to IPF or internet. IPF Top locked normal, no cpu or memory issues and suricata running. Then canceled the rclone routine and tried reloading unbound which indicated that it succeeded. Still could not connect, rebooted IPF and the windows machine.
I am attaching an “Top” screen image and will try to upload the 2026.06.29 kernel log for review. Sorry this gui refuses the text kernel log upload. Let me know if there is another way. Thx for all your help.
One further note: The kernel panic only seems to occur when running rclone. I am not yet noticing this condition otherwise. Will watch closely and report if it does. Thx again.
So rclone is running on a windows machine on the green network and it is trying to do a backup to a cloud system on the internet and when it is running on a windows machine it causes a kernel panic in IPFire.
I would be thinking that the kernel in the IPFire system has some bug of some sort that does not work well with rclone.
However you say the problem occurs with CU201 & CU202 but not with CU200 but CU200 and CU201 have the same kernel (6.18.7) and CU202 has a different kernel (6.18.32)
So kernel version cannot have anything to do with the kernel panic being triggered by rclone.
When running rclone and IPFire has the kernel panic can you please show the log messages created about the panic. We need to see those to understand what is triggering the panic.
panic=10 in this line means that the kernel has been instructed to reboot the system 10 seconds after having a panic.
kernel: simple-framebuffer simple-framebuffer.0: [drm] Registered 1 planes
with drm panic
This line means that the Direct Rendering Manager (DRM) has successfully registered a display plane that can handle panic messages. This means that if a kernel panic occurs, this plane will be used to display relevant error information. ie you will be able to see it on your console.
Neither of those two types of lines are actual kernel panics.
Some typical kernel panic lines could be
Kernel panic - not syncing: Fatal exception
Kernel panic - not syncing: Attempted to kill init!
Kernel panic - not syncing: VFS: Unable to mount root fs
The words Kernel panic are nowhere in the logs.
When your system freezes up or prevents you having any access then you need to note the time and then when you get access to it again (if required by reboot, although better if you can access via the console so nothing gets lost with the reboot) look in the /var/log/messages log file or in the /var/log/httpd/error_log in the same time period as you found the problem and then see what sort of error messages were occurring at or around that time frame.
This would suggest that there is some problem with the windows machine.
As you are sending the backup data to a cloud system it would be worth looking at the logs of the cloud server in the time frame you started to have the TLS connection errors on the windows machine and see if there are any errors in the cloud backup server and compare the messages in the windows rclone logs.
I attached a full kernel log of the test IPF machine for the 29th and 30th in a previous email response for your review. The IPF machine does in fact display a “Kernel Panic” error on the screen for about 5-10 seconds then auto reboots. The logs also have some entries about the green interface as I remember. The TLS error only appears on the web browser after an initial IPF failure. Besides locally the same IPF failure occurs when executing uploads from a Ubuntu Linux test machine setup and a local windows 10 laptop. As mentioned, clients using the same setup are experiencing some of the same issues.
One final test today - attempted to upload 11 Quickbooks backup files of 4-50 meg each to a Google drive test folder using just drag and drop… Here are the results after only 3 files updated..
Sorry that should have read 40 - 50 meg each on the file sizes and that rebooting IPF was the only way I found to bring the system back. I can upload a full log file again if needed. Thx
I assume you wanted the screen print at the time of failure, however with the failure we are unable to access or log into the WUI until reboot. Am attaching a “TOP” screen print just taken before the copy/paste test. Hope this helps.
