For a few years now, I have been running two IPFire systems on Raspberry Pi 4Bs in two different buildings. Since the beginning of April, both have been running Core Update 199. They boot from USB sticks.
Both systems perform a fully automated reboot every night.
After about 3–4 days, one of the systems fails to boot up. It reports corrupted files within the initramfs. I thought of problems in dc-power and a hardware fault. Consequently, I performed a full read-write test on the USB stick; no errors were found.
I then downloaded Core Update 199 again, performed a restore, and the system was operational once more. However, after three nights, the error recurred.
Since the OpenVPN connection failed to start, I currently have no remote access to the system; I therefore have to drive to the location to attend to it. I’m going to change the USB stick.
In preparation for this visit, I would like to know how I can configure IPFire to execute fsck during every automated reboot.
In an older forum post, there was a note stating that fsck.repair does not function within GRUB. This is presumably because IPFire does not utilize systemd.
I created a file named /forcefsck and found results in /var/log/bootlog, but my file /forcefsck is automatically deleted after each reboot. How do I prevent this?
Do you recommend additional options to perform automatic repair?
I’d consider upgrading the memory on these Raspberries. In my opinion, it’s better to use SD cards instead of USB. The problem may be caused by excessive swap usage. Perhaps you should review the configuration and set it to a state where swap isn’t used.
Okay, now I understand why you’re using USB instead of SD.
RAM utilization is fine, and swap space isn’t being used either.
The question is, why the daily reboot?
Here is a description of defining automatic tasks:
I remember an how-to prevent raspberrys from destroying filesystems on sd-cards but I found this after I had switched to usb.
Espescially the machine which crashed never ran on sd-cards and never crashed. It was installed in february 2024.
My other machine never crashed on usb after installation in about 2023.
I only crashed on sd-cards before.
I reboot nightly because my fritzbox and ipfire’s unbound never became best friends.
My fritzbox renews its IP nigthly, too.
Ipfires DNS strict and fritzbox are not best friends, either..
So I use DNS from fritzbox and additional DNS from several servers recommended by ipfire.org.
Setup: Fritz → ipfire red → LAN.
and:
lan location #1 → fritz at location #1 → internet ← fritz at location #2 ← lan location #2
Both Fritzboxes are connected by dyndns of fritz called “myfritz”. Inside this myfritz-wireguard-tunnel I connected the red IPs of both ipfires using openvpn.
I tested your option on my still running system. It reboots and there are entries from fsck in my bootlog. It will take some days to find out why my other system died.
Hi @firecherry,
I’m really glad that your systems ran nicely for several years on USB storage, with satisfaction about reliability and peformances.
However, microSD, SD and USB flash drives usually share the same source about NAND chips: they are never 1st, 2nd or 3rd choice. Most of the times are sub-commodity level.
So, as a personal point of view, it’s ok that you are satisfied, but they will fail, or they are failing. And probably logwriting is part of the deterioration of that kind of flash storage.
While being less sensitive to vibration and mostly immune to magnetic fields, flash storage usually fails in less predictable way than hdd, and when happens… usually fail spectacularly. This also applies to S.M.A.R.T. enabled devices like NVMe cards and SATA boards (a Phison controller on a Kingston SSD still laughing at me).
As advice, consider to replace sooner than later your USB drives. Nice opportunity to take out an updated backup and do a fresh-er install, with bigger boot partition thanks to the changes of the mainline.
Upgrade/update of a working system is usually easier than rushing a disaster recovery.
