Update 161 --> 162 IPfire doesn't boot

Hello and Merry Christmas to all,

at 24. i decided to do the upgrade to Core 162 which unfortunately ended with Ipfire not booting any more.

System : Atom C2550, Supermicro A1SAM-2550F
OS : Dom0, Debian Buster
Hypervisor : Xen 4.11.4

The update process went perfectly through without any issues:

Dec 24 18:25:46 ipfire pakfire: PAKFIRE INFO: IPFire Pakfire 2.27-x86_64 started!
Dec 24 18:25:46 ipfire pakfire: CORE INFO: core-list.db is 66281 seconds old. - DEBUG: force
Dec 24 18:25:46 ipfire pakfire: DOWNLOAD STARTED: lists/core-list.db
Dec 24 18:25:46 ipfire pakfire: MIRROR INFO: 24 servers found in list
Dec 24 18:25:46 ipfire pakfire: DOWNLOAD INFO: Host: ipfire.infania.net (HTTPS) - File: pakfire2/2.27-x86_64/lists/core-list.db
Dec 24 18:25:46 ipfire pakfire: DOWNLOAD INFO: pakfire2/2.27-x86_64/lists/core-list.db has size of 903 bytes
Dec 24 18:25:47 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
Dec 24 18:25:47 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
Dec 24 18:25:47 ipfire pakfire: DOWNLOAD INFO: Signature of core-list.db is fine.
Dec 24 18:25:47 ipfire pakfire: DOWNLOAD FINISHED: pakfire2/2.27-x86_64/lists/core-list.db
Dec 24 18:25:47 ipfire pakfire: CORE UPGR: Upgrading from release 161 to 162
Dec 24 18:25:47 ipfire pakfire: DOWNLOAD STARTED: meta/meta-core-upgrade-162
Dec 24 18:25:47 ipfire pakfire: MIRROR INFO: 24 servers found in list
Dec 24 18:25:47 ipfire pakfire: DOWNLOAD INFO: Host: mirror.cedia.org.ec (HTTPS) - File: ipfire/pakfire2/2.27-x86_64/meta/meta-core-upgrade-162
Dec 24 18:25:48 ipfire pakfire: DOWNLOAD INFO: ipfire/pakfire2/2.27-x86_64/meta/meta-core-upgrade-162 has size of 997 bytes
Dec 24 18:25:49 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
Dec 24 18:25:49 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
Dec 24 18:25:49 ipfire pakfire: DOWNLOAD INFO: Signature of meta-core-upgrade-162 is fine.
Dec 24 18:25:49 ipfire pakfire: DOWNLOAD FINISHED: ipfire/pakfire2/2.27-x86_64/meta/meta-core-upgrade-162
Dec 24 18:25:49 ipfire pakfire: DOWNLOAD STARTED: paks/core-upgrade-2.27-162.ipfire
Dec 24 18:25:49 ipfire pakfire: MIRROR INFO: 24 servers found in list
Dec 24 18:25:49 ipfire pakfire: DOWNLOAD INFO: Host: mirror.uta.edu.ec (HTTPS) - File: ipfire/pakfire2/2.27-x86_64/paks/core-upgrade-2.27-162.ipfire
Dec 24 18:25:50 ipfire pakfire: DOWNLOAD INFO: ipfire/pakfire2/2.27-x86_64/paks/core-upgrade-2.27-162.ipfire has size of 80142736 bytes
Dec 24 18:26:27 ipfire pakfire: DOWNLOAD INFO: HTTP-Status-Code: 200 - 200 OK
Dec 24 18:26:28 ipfire pakfire: DOWNLOAD INFO: File received. Start checking signature...
Dec 24 18:26:29 ipfire pakfire: DOWNLOAD INFO: Signature of core-upgrade-2.27-162.ipfire is fine.
Dec 24 18:26:29 ipfire pakfire: DOWNLOAD FINISHED: ipfire/pakfire2/2.27-x86_64/paks/core-upgrade-2.27-162.ipfire
Dec 24 18:26:29 ipfire pakfire: PAKFIRE UPGR: core-upgrade-162: Decrypting...
Dec 24 18:26:29 ipfire pakfire: CLEANUP: tmp
Dec 24 18:26:29 ipfire pakfire: DECRYPT STARTED: core-upgrade-162
Dec 24 18:26:31 ipfire pakfire: DECRYPT FINISHED: core-upgrade-162 - Status: 0
Dec 24 18:26:32 ipfire pakfire: PAKFIRE UPGR: core-upgrade-162: Upgrading files and running post-upgrading scripts...
Dec 24 18:27:15 ipfire pakfire: CLEANUP: tmp
Dec 24 18:27:15 ipfire pakfire: PAKFIRE UPGR: core-upgrade-162: Finished.
Dec 24 18:27:15 ipfire pakfire: CORE INFO: core-list.db is 88 seconds old. - DEBUG: noforce
Dec 24 18:27:15 ipfire pakfire: PAKFIRE INFO: Pakfire has finished. Closing.

But after rebooting the system doesn’t came up again.
Altough the kernel seems to be compiled with all neccessary options to be needed for an paravitualized machine, at least as far as my understanding reaches, according to Ipfire 2.x Kernel 5.15.6 commit

During the boot process Ipfire isn’t started with paravirtualized kernel which i suspect to be the main reason:
Boot on core 162:

Booting kernel on Xen
Xen version: 4.11.4 (preserve-AD)

Boot on Core 161:

[    1.717044] Booting paravirtualized kernel on Xen
[    1.717047] Xen version: 4.11.4 (preserve-AD)

For investigation of the Problem i supply the update-core-upgrade-162.log an the rdsosreport.txt.

If anything further is needed i will supply that info too.
logs.zip (31.2 KB)

Looks like storage isn’t found. The kernel is booting fine.

Can you check what modules are being loaded/packaged in the initrd?

Hi,

took me quite some time to be able to give you that info.
For me it’s the first time looking into an initramfs.

Looks like at least the xen_blkfront drivers are missing, somewhat as i assumed as in the dracut shell i also had no blkid to look for what drives are available:

drivers/block kernel 5.15.6:

lib/modules/5.15.6-ipfire/kernel/drivers/block
lib/modules/5.15.6-ipfire/kernel/drivers/block/floppy.ko.xz
lib/modules/5.15.6-ipfire/kernel/drivers/block/sx8.ko.xz
lib/modules/5.15.6-ipfire/kernel/drivers/block/mtip32xx
lib/modules/5.15.6-ipfire/kernel/drivers/block/mtip32xx/mtip32xx.ko.xz
lib/modules/5.15.6-ipfire/kernel/drivers/block/virtio_blk.ko.xz

drivers/block kernel 5.10.55:

lib/modules/5.10.55-ipfire/kernel/drivers/block
lib/modules/5.10.55-ipfire/kernel/drivers/block/floppy.ko.xz
lib/modules/5.10.55-ipfire/kernel/drivers/block/rsxx
lib/modules/5.10.55-ipfire/kernel/drivers/block/rsxx/rsxx.ko.xz
lib/modules/5.10.55-ipfire/kernel/drivers/block/sx8.ko.xz
lib/modules/5.10.55-ipfire/kernel/drivers/block/skd.ko.xz
lib/modules/5.10.55-ipfire/kernel/drivers/block/mtip32xx
lib/modules/5.10.55-ipfire/kernel/drivers/block/mtip32xx/mtip32xx.ko.xz
lib/modules/5.10.55-ipfire/kernel/drivers/block/virtio_blk.ko.xz
lib/modules/5.10.55-ipfire/kernel/drivers/block/xen-blkfront.ko.xz
lib/modules/5.10.55-ipfire/kernel/drivers/block/umem.ko.xz

Full list of contents of initramfs-5.15.6-ipfire.img i appended
initrd_5_15_6.zip (7.1 KB)

Thanks for looking into this and for the great Job you all are doing permanently.
For the moment i am comfortable with a backup of ipfire core 160.

I never heard of initrd or initramfs so I decided to look around also. Michael may be looking to run this command:

[root@ipfire ~] # lsinitrd -h
Usage: lsinitrd [options] [<initramfs file> [<filename> [<filename> [...] ]]]
Usage: lsinitrd [options] -k <kernel version>

-h, --help                  print a help message and exit.
-s, --size                  sort the contents of the initramfs by size.
-m, --mod                   list modules.
-f, --file <filename>       print the contents of <filename>.
-k, --kver <kernel version> inspect the initramfs of <kernel version>.

[root@ipfire ~] # 

But I don’t know if he is looking for the short list:

lsinitrd -m

or the l-o-n-g list

lsinitrd          # <- no parameters

@sysiphus

From ipfire2.x Kernel 5.15.6 commit : 656 # CONFIG_KVM_XEN is not set

Could that be the reason why ?

Same problem here.

I don’t believe so.
I am really not experienced with kernel stuff, but i think the main culprit is the kernel isn’t able to access the storage as Michael Tremer stated before.

If i am not wrong the Blockdevice driver is needed for the kernel to access the storage.
It seems to be compiled in as module, but the modules aren’t in the initramfs.

For sure Arne Fitzenreiter will find out whats going wrong.

I fully agree.
But since

is missing in kernel 5.15.6, i was wondering if it was simply forgotten.

I guess you will have to wait,

Hi,
I have a simular problem (Update from .161 to .162 on XEN). Drucat is not finding the disk.

I just tried a fresh flush-image with the some error message.

Some more info I may/can share?

Best regards
B. Hafenwasser

log from console:
Key type dns_resolver registered
IPI shorthand broadcast: enabled
AVX2 version of gcm_enc/dec engaged.
AES CTR mode by8 optimization enabled
sched_clock: Marking stable (1054978441, 416028)->(1056815452, -1420983)
registered taskstats version 1
Loading compiled-in X.509 certificates
Loaded X.509 cert ‘IPFire.org: Build time autogenerated kernel key: a6a6963e785feaad5e1c5e6a9340b157a331b013’
xenbus_probe_frontend: Device with no driver: device/vbd/51744
xenbus_probe_frontend: Device with no driver: device/vif/0
xenbus_probe_frontend: Device with no driver: device/vif/1
xenbus_probe_frontend: Device with no driver: device/vif/2
Freeing unused kernel image (initmem) memory: 2100K
Write protecting the kernel read-only data: 22528k
Freeing unused kernel image (text/rodata gap) memory: 2036K
Freeing unused kernel image (rodata/data gap) memory: 1780K
x86/mm: Checked W+X mappings: passed, no W+X pages found.
Run /init as init process
dracut: dracut-038
udevd[258]: starting eudev-3.2.6
setfont: putfont: 512,8x16:failed: -1
putfont: PIO_FONTX: Inappropriate ioctl for device

dracut Warning: Could not boot.
dracut Warning: Could not boot.
dracut Warning: /dev/disk/by-uuid/f23d4e7b-f53e-4814-89dd-a6dbc374989e does not exist
dracut Warning: /dev/disk/by-uuid/f23d4e7b-f53e-4814-89dd-a6dbc374989e does not exist

Generating “/run/initramfs/rdsosreport.txt”
You might want to save “/run/initramfs/rdsosreport.txt” to a USB stick or /boot
after mounting them and attach it to a bug report.

To get more debug information in the report,
reboot with “rd.debug” added to the kernel command line.

Dropping to debug shell.

dracut:/#

Hi,
i don’t think further info is needed.
As mentioned above the block-device drivers for XEN are missing in the initramfs, probably a simple packaging error.
We just have to wait for an corrected kernel and stay on core_161 meanwhile.

3 Likes

Hello,
My 3 IPFire machines do boot but on each and every of them I get these errors (also seen above)

Setting up Linux console…
setfont: putfont: 512,8x16:failed: -1
putfont: PIO_FONTX: Inappropriate ioctl for device [ FAIL ]

Anybody can tell me if this is a missing file and where can I get it from?

Hi Sysiphus
I have the same issue and could confirm, it is a missing driver in the initramdisk. The kernel 5.15 boots fine but cannot mount the root filesystem due to missing xen-blkfront driver.

To add the missing driver to the initrd, mkinitrd is your friend. The following command creates a working initramdisk with needed xen-blkfront driver (—force is needed to overwrite the existing ramdisk):
mkinitrd --force --with=xen-blkfront /boot/initramfs-5.15.6-ipfire.img 5.15.6-ipfire

This command must be run in a still running system after upgrade to 162 prior rebooting or in a chroot environment. Therefore, mount the root partition for instance in /mnt/root first, then the boot partition correspondingly under /mnt/root/boot and bind /proc and /sys to the tree with:
sudo mount -o bind /proc /mnt/root/proc
sudo mount -o bind /sys /mnt/root/sys

Finally use chroot to enter the ipfire environment:
sudo chroot /mnt/root

Hope this helps.

This has nothing to do with the kernel module. On non VGA consoles this error is normal since the last coreutils (i think) update. There is no font that you can configue and the old putfont binary has ignored the error.

Ups. The xen-blkfront module is new older kernels don’t have it.
I will add this to the dracut config in the next update.

1 Like