Ubuntu 22.04 or 23.04 can't update after IPFire upgrade to Core-Update-Level 178

Hi all,

I’ve got a serious problem I can’t seem to solve in any way:

  • have got a bunch of Ubuntu machines behind a IPFire router, all physical, no VMs
  • IPFire was recently 2023-08-30 updated to Core-Update-Level: 178
  • all of my Ubuntu machines, be it 22.04 LTS or 23.04, stopped updating from archive.ubuntu.com, archive.canonical.com, so they don’t even get security updates and can’t install a thing from their repos
  • other repos (for example added 3rd party PPAs) do update and work just fine
  • tried alternative connection not using my IPFire instance, everything worked fine
  • tried restoring IPFire from backup /var/ipfire/backup/2023-07-12-08:44.ipf, no effect
  • other stuff seems to work on Ubuntu machines, “Check DNS Servers” on IPFire says all OK

Any ideas what could be wrong, please?

Best wishes,
Mark

Welcome to our community.

If the “Intrusion Prevention” feature is enabled, the issue could be due to a false positive triggered by one of its rules.

1 Like

Hi cfusco,

thanks for your reply.

Sadly, it won’t be the case, as Intrusion Prevention System’s daemon is stopped.

Any other ideas?

Maybe you can find the solution in the links below

edit

1 Like

I do have my IPS completely disabled and still get the issue.

Are you using location block?

2 Likes

Yes, let’s check the logs. First, connect to IPFire console and issue this command tail -f /var/log/messages. This command opens the kernel logs and it will display events as they happens (ctrl-c to exit). Next, start the update. Copy and paste here whatever happens to the kernel after you start the update.

You should also have a look at the apt logs on your Ubuntu machine. They should be in /var/log/apt/history.log.

Are you using IPFire proxy? As @hvacguy has already asked, what about location block and IP Address Blocklists?

3 Likes

No, Location Block is disabled.

Hi cfusco,

no, no proxy configured. No location block, and no blocked IPs either.

Ubuntu problem:

marek@desktop:~$ sudo apt update
Ign:1 http://archive.canonical.com/ubuntu lunar InRelease
Ign:2 http://archive.ubuntu.com/ubuntu lunar InRelease                                                                                                                                                                          
Ign:3 http://archive.ubuntu.com/ubuntu lunar-updates InRelease                                                                                                                                                                  
Ign:4 http://archive.ubuntu.com/ubuntu lunar-security InRelease                                                                                                               
Ign:5 http://archive.ubuntu.com/ubuntu lunar-backports InRelease                                                                                                              
Hit:6 https://brave-browser-apt-release.s3.brave.com stable InRelease                                                                                                         
Hit:7 https://ppa.launchpadcontent.net/kdenlive/kdenlive-stable/ubuntu lunar InRelease                                                                                        
Hit:8 https://ppa.launchpadcontent.net/mozillateam/ppa/ubuntu lunar InRelease                          
Hit:9 https://ppa.launchpadcontent.net/tomtomtom/yt-dlp/ubuntu lunar InRelease                         
Hit:10 https://dbeaver.io/debs/dbeaver-ce  InRelease                                                   
Hit:11 https://ppa.launchpadcontent.net/ubuntuhandbook1/keepass2/ubuntu lunar InRelease
Ign:1 http://archive.canonical.com/ubuntu lunar InRelease  
Ign:2 http://archive.ubuntu.com/ubuntu lunar InRelease
Ign:3 http://archive.ubuntu.com/ubuntu lunar-updates InRelease
Ign:4 http://archive.ubuntu.com/ubuntu lunar-security InRelease
Ign:5 http://archive.ubuntu.com/ubuntu lunar-backports InRelease
Ign:1 http://archive.canonical.com/ubuntu lunar InRelease
Ign:2 http://archive.ubuntu.com/ubuntu lunar InRelease
Ign:3 http://archive.ubuntu.com/ubuntu lunar-updates InRelease
Ign:4 http://archive.ubuntu.com/ubuntu lunar-security InRelease
Ign:5 http://archive.ubuntu.com/ubuntu lunar-backports InRelease
Err:1 http://archive.canonical.com/ubuntu lunar InRelease
  Could not connect to archive.canonical.com:80 (91.189.91.15). - connect (111: Connection refused) Could not connect to archive.canonical.com:80 (185.125.188.12). - connect (111: Connection refused) Could not connect to archive.canonical.com:80 (185.125.188.87). - connect (111: Connection refused)
Err:2 http://archive.ubuntu.com/ubuntu lunar InRelease
  Could not connect to archive.ubuntu.com:80 (91.189.91.83). - connect (111: Connection refused) Could not connect to archive.ubuntu.com:80 (185.125.190.36). - connect (111: Connection refused) Could not connect to archive.ubuntu.com:80 (91.189.91.81). - connect (111: Connection refused) Could not connect to archive.ubuntu.com:80 (91.189.91.82). - connect (111: Connection refused) Could not connect to archive.ubuntu.com:80 (185.125.190.39). - connect (111: Connection refused)
Err:3 http://archive.ubuntu.com/ubuntu lunar-updates InRelease
  Unable to connect to archive.ubuntu.com:http:
Err:4 http://archive.ubuntu.com/ubuntu lunar-security InRelease
  Unable to connect to archive.ubuntu.com:http:
Err:5 http://archive.ubuntu.com/ubuntu lunar-backports InRelease
  Unable to connect to archive.ubuntu.com:http:
Reading package lists... Done

IPFire:

[root@privri ~]# tail -f /var/log/messages
Sep  4 08:16:10 privri kernel: DROP_INPUT IN=red0 OUT= MAC=ff:ff:ff:ff:ff:ff:18:e8:29:87:e1:31:08:00 SRC=185.134.214.1 DST=255.255.255.255 LEN=229 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=52952 DPT=10002 LEN=209
Sep  4 08:16:20 privri freshclam[2823]: Received signal: wake up
Sep  4 08:16:20 privri freshclam[2823]: ClamAV update process started at Mon Sep  4 08:16:20 2023
Sep  4 08:16:20 privri freshclam[2823]: daily.cld database is up-to-date (version: 27020, sigs: 2040238, f-level: 90, builder: raynman)
Sep  4 08:16:20 privri freshclam[2823]: main.cvd database is up-to-date (version: 62, sigs: 6647427, f-level: 90, builder: sigmgr)
Sep  4 08:16:20 privri freshclam[2823]: bytecode.cvd database is up-to-date (version: 334, sigs: 91, f-level: 90, builder: anvilleg)
Sep  4 08:16:20 privri freshclam[2823]: --------------------------------------
Sep  4 08:16:40 privri kernel: DROP_INPUT IN=red0 OUT= MAC=ff:ff:ff:ff:ff:ff:18:e8:29:87:e1:31:08:00 SRC=185.134.214.1 DST=255.255.255.255 LEN=229 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=45626 DPT=10002 LEN=209
Sep  4 08:16:44 privri The database has been updated recently
Sep  4 08:17:10 privri kernel: DROP_INPUT IN=red0 OUT= MAC=ff:ff:ff:ff:ff:ff:18:e8:29:87:e1:31:08:00 SRC=185.134.214.1 DST=255.255.255.255 LEN=229 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=35508 DPT=10002 LEN=209
Sep  4 08:17:40 privri kernel: DROP_INPUT IN=red0 OUT= MAC=ff:ff:ff:ff:ff:ff:18:e8:29:87:e1:31:08:00 SRC=185.134.214.1 DST=255.255.255.255 LEN=229 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=49262 DPT=10002 LEN=209
Sep  4 08:18:10 privri kernel: DROP_INPUT IN=red0 OUT= MAC=ff:ff:ff:ff:ff:ff:18:e8:29:87:e1:31:08:00 SRC=185.134.214.1 DST=255.255.255.255 LEN=229 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=52944 DPT=10002 LEN=209
Sep  4 08:18:40 privri kernel: DROP_INPUT IN=red0 OUT= MAC=ff:ff:ff:ff:ff:ff:18:e8:29:87:e1:31:08:00 SRC=185.134.214.1 DST=255.255.255.255 LEN=229 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=46662 DPT=10002 LEN=209
Sep  4 08:19:10 privri kernel: DROP_INPUT IN=red0 OUT= MAC=ff:ff:ff:ff:ff:ff:18:e8:29:87:e1:31:08:00 SRC=185.134.214.1 DST=255.255.255.255 LEN=229 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=47405 DPT=10002 LEN=209
Sep  4 08:19:40 privri kernel: DROP_INPUT IN=red0 OUT= MAC=ff:ff:ff:ff:ff:ff:18:e8:29:87:e1:31:08:00 SRC=185.134.214.1 DST=255.255.255.255 LEN=229 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=48660 DPT=10002 LEN=209
Sep  4 08:20:10 privri kernel: DROP_INPUT IN=red0 OUT= MAC=ff:ff:ff:ff:ff:ff:18:e8:29:87:e1:31:08:00 SRC=185.134.214.1 DST=255.255.255.255 LEN=229 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=59424 DPT=10002 LEN=209
Sep  4 08:20:40 privri kernel: DROP_INPUT IN=red0 OUT= MAC=ff:ff:ff:ff:ff:ff:18:e8:29:87:e1:31:08:00 SRC=185.134.214.1 DST=255.255.255.255 LEN=229 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=54525 DPT=10002 LEN=209
Sep  4 08:21:10 privri kernel: DROP_INPUT IN=red0 OUT= MAC=ff:ff:ff:ff:ff:ff:18:e8:29:87:e1:31:08:00 SRC=185.134.214.1 DST=255.255.255.255 LEN=229 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=UDP SPT=54927 DPT=10002 LEN=209

Back in Ubuntu:

marek@desktop:~$ tail -f /var/log/apt/history.log
Requested-By: marek (1000)
End-Date: 2023-09-03  23:35:51

Start-Date: 2023-09-04  07:50:58
Requested-By: marek (1000)
End-Date: 2023-09-04  07:50:59

Start-Date: 2023-09-04  07:50:59
Requested-By: marek (1000)
End-Date: 2023-09-04  07:50:59

Thanks for any suggestions!

Could this be an issue with the etc/apt/sources.list?

Same issue reported on several Ubuntu forums.

1 Like

Checked them too…

marek@desktop:~$ cat /etc/apt/sources.list
deb http://archive.ubuntu.com/ubuntu/ lunar main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu/ lunar main restricted universe multiverse

deb http://archive.ubuntu.com/ubuntu/ lunar-updates main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu/ lunar-updates main restricted universe multiverse

deb http://archive.ubuntu.com/ubuntu/ lunar-security main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu/ lunar-security main restricted universe multiverse

deb http://archive.ubuntu.com/ubuntu/ lunar-backports main restricted universe multiverse
deb-src http://archive.ubuntu.com/ubuntu/ lunar-backports main restricted universe multiverse

deb http://archive.canonical.com/ubuntu lunar partner
deb-src http://archive.canonical.com/ubuntu lunar partner

If I remove IPFire from the chain, things work on all my machines (desktops, servers, phone…), so I think I’d narrowed it down to some configuration problem after the upgrade to Core 178. No clue what changed, but it all started there.

If you had enabled the WebProxy i would spot the fast flux detection.

But without i have no idea yet.

The only change in Core Update 178 was a kernel update to implement some fixes for the latest hardware vulnerabilities from Intel and AMD. There was also a fix in the kernel to workaround the bug in Hyper-V causing those vm systems to be unable to boot with Core Update 177.

There was nothing else changed. It was an interim update because of the CVE’s announced from the new hardware vulnerabilities.

https://blog.ipfire.org/post/ipfire-2-27-core-update-178-released

Really struggle to see why the above could cause a problem to do apt-get updates.

Looking through the update code for Core Update 178 it is very simple.

  • Stop squid (Web Proxy) but you don’t have that running anyway.
  • Make sure that a kernel is actually present.
  • Check the disk space.
  • remove the old kernel.
  • Install the new kernel.
  • restart sshd, reload unbound and start squid if it is enabled.
  • remove lm_sensors config to force sensor update with new kernel at next reboot.
  • Mark that the update requires a reboot.
  • Update the grub config to display new core version.

and that’s it.

Can you confirm that you were running Core Update 177 before upgrading to Core Update 178.

Can other people running with Core Update 178 and using Ubuntu machines in their network confirm if they do or do not have problems trying to do updates?

Seems like I resolved it!
Disabled Advanced Web Proxy, re-enabled it again without any other changes.

apt works again on my Ubuntu Server and others Desktops. Even after rebooting IPFire and all the stuff behind it.

My own Desktop needed a tweak in /etc/apt/sources.list:
From the above mentioned I removed:

deb http://archive.canonical.com/ubuntu lunar partner
deb-src http://archive.canonical.com/ubuntu lunar partner

How the Core-Update is related to the issue I don’t know.

You said in the earlier post that no proxy was configured but now you say that you disabled and then enabled again your web proxy and it fixed the problem.

Why did you say that no proxy was configured?

I have IPS enabled. In fact I have everything blocked, all countries and locations, and adding to that these rulesets:

Abuse.ch SSLBL Blacklist Rules
Emergingthreats.net Community Rules
Snort/VRT GPLv2 Community Rules

and I have two virtual Ubuntu’s running and they can update.

So it must be that Web Proxy from what is stated earlier.

At the time I think it was true - was trying switching things on / off, and restoring backups too.
I also state I don’t know how the Core-Update is related to the issue.
In other words: yes, it got resolved, but how exactly isn’t clear to me.

Also the Ubuntu partner repo wasn’t it, because it was configured only on one PC out of many, and all had the same problem.

1 Like