Core update 182 caused Grub boot failure

EDIT: This is long and unresolved, so here’s a summary I’ll update:

  • Cleanly updated existing IPFire installation to update 182

  • Rebooted from the WUI. System will not boot. grub rescue > is displayed immediately after the BIOS prompt with "error: file /grub/x86_64-efi/boot.mod not found.". It seems that the system is unable to perform a legacy BIOS boot and is attempting UEFI which is failing and triggering the grub rescue > prompt. I have attempted using Grub Rescue but have not been able to get the system to boot.

  • No evidence of filesystem or disk corruption

  • The system can be manually booted via recovery USB stick, such as SuperGrub, but not only by choosing to “extract entries from grub.cfg” not the default detected boot options, which are all EFI.

  • Forcing an initramfs rebuild followed by using the IPFire /usr/bin/install-bootloader script does not resolve the problem

  • A rebuild may fix this but I wish to avoid that at all costs, especially as the rest of the OS and storage is fully intact. I use a custom partition layout which has a dedicated partition for a KVM VM and while I have backups, I’ve not tested them and do not know what I’ll miss.

Other people have rebooted boot problems, with a variety of symptoms, however none match what I describe. I understand most have recovered their systems by using SuperGrub or reinstalling IPFire and restoring an IPFire backup.

EDIT: the directory and the boot.mod file exist… odd

However when I try to list it in grub recovery I get error: unknown filesystem. This error is displayed even after other files are listed successfully.

The partition is xfs

EDIT: An observant reader will note that the list command I ran at the grub rescue prompt didn’t use the correct partition! I should have run ls (hd1,msdos1)/grub/x86_64-efi/boot.mod not msdos2. Either way it can’t/won’t read that file.

Why is it now trying to find /grub/x86_64-efi/boot.mod on a legacy BIOS system?!

I happened to have an IPFire test VM I’d previously installed. It hasn’t had the latest update applied and doesn’t try to use EFI, but I’m really struggling to find how I can fix my main router to not use EFI.

After hours of intense troubleshooting, I gave up and began backing everything up, including my VM’s partition (Yes, I’m using a VM on a partition, not a file as performance matters. Yes, I know it will make re-installation a pain!). I put in a bootable IPFire installer and booted from it…then the system booted the original OS!!!

EDIT: the problem may be with the newer Grub version.
The only version of IPFire which I could find to download was the last update. This means that the USB stick which booted my system was using the previous version of grub2, not the one from the latest update

So things are working for now and my wife and family are no longer on my back(!) but the local grub2 configuration is likely still broken. At least now I can work from within IPFire to fix it.

I ran a grub-mkconfig -o /var/tmp/grub.cfg then compared the file with /boot/grub/grub.cfg and only these entries existed in the new file:

<   set locale_dir=$prefix/locale
<   set lang=en_US
<   insmod gettext

I’m unsure what the gettext part is about, but I’m guessing these three lines won’t help my problem.

Any ideas please? What should I check next?

Not intrinsically. There must be some other factor that also needs to occur or be present making it do that.

I had no problem with the new grub on all three of my systems that I updated to CU182. Two are legacy bios and one is efi.

Also none of the devs or other people that ran the Testing phase had a problem with the grub system.

If you are running your IPFire with the previous version from a USB stick then the bootlog should still be available on the main hard disk.

I would raise a bug on this issue and include the bootlog from it so one of the experienced devs with booting can look at it and see what might have happened.

2 Likes

I had the same issue occur on a Dell Optiplex 3010 based IPFire.
I chose Advanced options from boot menu, pressed enter at boot option (no 'e’dit).
Boot continued in legacy mode.
Have tested restarting and power off and bootloader OK.

Thank you, but this problem is happening the instant the system starts. If you see the screenshot I posted above you can see that the BIOS prompt: Press <DEL> or <ESC> to enter setup. is right before the failure message which starts error: file... So I have no opportunity to do anything at all. The system starts and instantly goes to grub rescue mode.

I’ve double checked the BIOS and as far as I can tell it’s using legacy BIOS boot (it’s a bit unclear as it is a much more limited BIOS than most PCs as this is a low power/IoT model).

Thanks again @bonnietwin !

Could you please explain which file the bootlog is recorded in?

I don’t understand the early boot process well, but it seems that my system’s unique boot configuration has not been corrupted/damaged or otherwise it would not have been able to boot from the USB stick with Core update 181.

I read that Michael has now updated the download image to update 182, so when I can afford to have some downtime (in a few days) I’ll make a USB stick with that version and attempt to boot from it. If works then that will prove that the grub update is not the cause.

My next step, if this was a major Linux OS, would be to re-install the problematic package (grub in this case) over the top to fix any corrupted files. How can I do that with IPFire please? It doesn’t have discrete OS packages, just the entire Core updates. As far as I know there’s no way to reinstall them.

Thank you

i have issue after reboot too

1 Like

The file is located in

/var/log/bootlog

or in

/var/log/bootlog.x.gz

for older versions.

The files are rotated on a weekly basis on Sunday at 00:01 or when the files have reached a certain size.

Several people have now reported this issue in this thread but no one has yet raised a bug report on this, as I suggested in post 5

Here is the link to bug reporting in the wiki
https://wiki.ipfire.org/devel/bugzilla

and here is the link to the IPFire Bugzilla system (link also in the wiki.
https://bugzilla.ipfire.org/

The login credentials for the IPFire Bugzilla are the same email address and password as you will have set up for your IPFire People account.

I am not familiar with grub or the whole booting process enough to be able to help.
If you raise a bug on it then it is recorded in a process that is regularly reviewed by the devs.

Thanks for that.

The issue with raising a bug is that it needs evidence and time to gather all the details.

I can’t afford to spend the time on this now but will have a chance to try in a few days.

If anyone else does then please go ahead and I’ll also add the details from my system when I’m able.

I updated too, and no more bootuing. But I resolved the problem.

  1. Download the SuperGrub: https://www.supergrubdisk.org/category/download/supergrub2diskdownload/super-grub2-disk-stable/
  2. Boot SuperGrub
  3. Choose Detect And Show
  4. Choose IpFire…from menu
  5. Then begin boot to the Ipfire
  6. When booted, login with root
  7. Run the following two commands:
  • /usr/bin/install-bootloader
  • grub-mkconfig -o /boot/grub/grub.cfg
  1. Run the reboot command, and unmount SuperGrub iso

And voila, Ipfire booting again.

4 Likes

I’m seeing the same problem.
On an Rpi4.

Unfortunately, SuperGrub doesn’t have an ARM build. :frowning:

SuperGrub provide a way for “detect” a bootable environment, than provide boot.
I’m totally not familiar with Rpi4, might be something similar?
AFAIK there are no “BIOS/UEFI-alike” for ARM…

Hello,

I’m sorry to hear that other people are having problems. Unfortunately you have not provided enough details or have given just enough for me to believe your problem is different.

I worked through the problem I had in the first (admittedly verbose) posts and the screenshot I provided is totally different to the rest of you.

My system is x86_64 and instantly failed to boot (right after the BIOS screen) after a clean upgrade to the latest ipfire update followed by a clean reboot.

It does not seem to produce a boot log, the failure is too early.

@shyciii thank you for responding!

I came across the idea of using supergrub in my internet searches when I was first troubleshooting this, but discounted the idea. Supergrub doesn’t seem to have been maintained since 2019 and I’m having a problem which is related to a very modern version of grub2. It seems unwise to repair an modern installation using files and/or configuration from many years ago, so I have not tried it.

EDIT: I just reread your post and if I understand it correctly your only used supergrub to get the system to boot the first time. You then use ipfire to update grub. Is that right please?

Update: this doesn’t resolve my problem.

I had tried running /usr/bin/install-bootloader
And grub-mkconfig -o /boot/grub/grub.cfg but that didn’t resolve the problem on next boot.

So I tried supergrub. No matter what I do it just reboots the system in to the BIOS setup and if I attempt to boot ipfire I have the same problem I reported above.

Worse, I can’t reproduce whatever I did to get to to boot via the IPFire USB stick again so I’m stuck with no internet except for the phone I’m writing on.

EDIT: I was eventually able to get IPFire back up by using Supergrub to “boot manually” then selecting the “Advanced options for IPFire 2.27 …” option which for some reason still works. (all other options are EFI - see screenshot below)

So I’m back in IPFire, but I do not know how to resolve this problem. It will happen again on next reboot. I would love it if you have any knowledge please!

This screenshot is from the supergrub “boot manually”, “operating system” option.

Notice that everything is EFI even though I’ve always used legacy boot?

My system boots every USB stick I’ve tried, there’s something wrong with the OS install, despite the steps I ran before trying supergrub.

Do you see this when you select the Detect And Show menu item in the first menu?
Because if I select the Detect And Show menu item, then in the new menu I see quite a few EFI entries, and then an IPFire…(maybe version number) line, and it must be started. You will then boot into IPFire. By the way, the two commands I wrote afterwards will not use supergrub’s own grub for reinstallation, but IPFire’s grub2. That’s the good thing about Supergrub. that he can boot into the system, and then you can arrange it with his own applications in the system itself. Now I can’t see exactly what the menu looks like for me, because IPFire has to run constantly at our place because it operates a critical system. Anyway, Supergrub has pulled me out of the game quite a few times, because modern Grub2 updates are fucked up…