A tale of hardware failure, misplaced confidence, recovery procedures, and one very suspicious SATA cable.
It was a perfectly ordinary morning in Skåne. Coffee was hot, the network was working, and I sat down to upgrade Surtveggur with the kind of confidence you get from having prepared properly. There was a checklist, BookStack documentation, backups, exported configuration, screenshots, firewall.local backup, IPMI access, recovery media, and an AI assistant standing by.
CU203 has been out for roughly a month, I had watched the forums, and nothing suggested that this particular upgrade should turn into an archaeological expedition through GRUB, mdadm and dracut.
Pakfire completed without errors. I even wrote, “Finished with no errors,” followed shortly afterwards by, “Now for the restart.”
That was where optimism ended.
Instead of IPFire returning, I got:
error: failure reading sector 0x102800 from hd0
error: invalid arch-independent EFI magic
Entering rescue mode...
grub rescue>
The network disappeared at the same time, as would be expected.
What followed was a fairly systematic attempt to work out whether GRUB itself was broken or whether something underneath it had failed. The drives were visible, but modules would not load correctly, normal.mod produced ELF errors, and the troubleshooting tree expanded rapidly.
Eventually the real culprit emerged. One SSD had accumulated SATA interface errors for quite some time. The drive itself tested healthy: SMART good, 99% life remaining. But the SATA PHY error count was 698.
The SSD was innocent. The Akasa ProSlim SATA cable was not.
Apparently it had been quietly producing transmission errors without causing enough trouble to attract attention. The CU203 update, including boot-related writes, was simply the moment the problem became fatal. A 5€ cable had successfully taken down the firewall.
Next came rescue media. I wrote the IPFire ISO to USB using Rufus in DD mode. It did not boot. I also briefly rediscovered that the carefully saved `.iso file was not a bootable image but the IPFire configuration backup. Maybe I took the wrong ISO. Not so obvious in planning, less adequate while the firewall is dead.
A second USB eventually booted manually from the EFI shell using:
fs2:\EFI\BOOT\BOOTX64.EFI
That produced the installer menu, including the easily overlooked c — console option. The AI suggested pressing c. I did, about three menus later than would have been useful. Eventually ‘c’ got pressed.
After manually booting the installed system through GRUB via IPMI, on a Swedish keyboard through IPMI Console where producing characters such as | became its own minor project, IPFire dropped into a dracut emergency shell.
/dev/md1 does not exist
Running mdadm --assemble --scan produced the rather revealing:
md: kicking non-fresh sdb from array
The RAID assembled on the surviving member, which was progress. Then came roughly 45 minutes of identifying the partition layout: p1 BOOTLDR, p2 BOOT, p3 ESP, p4 SWAP and p5 ROOT. Useful information, although preferably learned under calmer circumstances.
From there the repair route became increasingly unattractive. chroot was unavailable, grub-install was unavailable, vfat behaved strangely, and the IPMI keyboard continued adding entertainment value where none was required.
At some point I wrote, “I give up.”
What I really meant was that spending another hour trying to repair the installation made no sense when a good configuration backup already existed.
So: fresh install.
The suspect drive was physically removed. There was effectively a 50/50 chance of choosing the correct one first, and for once probability behaved itself. IPFire installed normally, GRUB installed normally, and the machine booted normally.
The ISP then refused to route traffic for well over an hour.
The fiber box had already been power-cycled, the ISP could see traffic reaching their equipment, and nothing obvious remained wrong locally. Eventually connectivity simply returned. Whether that was ARP state, something at the ISP, the fiber-box restart, or networking folklore doing its thing remains unclear.
Recovery itself was almost disappointingly easy. The IPFire web interface accepts the .ipf backup directly, so I uploaded it from the network share and restored the system. The USB stick I had carefully copied it onto was, of course, unnecessary, as redundant backups might be when a plan actually starts working.
firewall.local was there. My custom fcronuser was there. Guardian was there. The configuration returned with very little drama.
The wife arrived home to a functioning network. I had aged approximately four years since breakfast.
What caused it: most likely an Akasa ProSlim SATA cable with 698 recorded PHY errors. The SSD itself tested healthy.
CU203, Knot Resolver and IPFire were innocent bystanders; the upgrade merely exposed an existing hardware problem at exactly the wrong moment.
What saved it: IPMI access, the .ipf backup, and good documentation. Once repairing the existing install stopped making sense, rebuilding the firewall became a mechanical recovery job rather than guesswork.
What changes now:
- Akasa slim SATA cables → gone
- Kingston consumer SSDs → retiring from firewall duty
- Samsung Enterprise P883 SSDs → incoming
- Delocke locking standard SATA cables → incoming
- mdadm email alerts → pending
- AI assistant → still operational, but now deeply suspicious of SATA cables.
The upgrade policy stays. So do the backup and documentation policies. If anything, the whole mess proved why they exist.
Surtveggur is now running CU203, restored and functional on a single Kingston SSD while the replacement hardware is on the way. The cable is in the bin. The coffee had been cold for hours.
SecCon out. 
