Weird problem how should I proceed?

Hello everyone,
Sorry for this wall of text, but a log or now special messages would be rather counterproductive here due to the total failure and thousands of error messages, so I’ll try to describe…how it came about and where I start why it came about…

I need tips on whether something can still be saved.
I can’t say exactly what the cause was or how it came about, nor what really happened, but my Ipfire has a permanent fault and I’m trying to provide all the relevant information here as to where I think the problem lies.
What happened?
I added another provider to the IPS, reloaded the log from time to time on the side, I noticed that the IP block lists were also increasingly blocking Tor addresses, which was otherwise not the case. I had already cleaned it up and removed the affected list, as I didn’t want to disrupt Tor operations.

Then it was suddenly no longer possible to reload the IPfire web interface or call up any website via the Squid in the browser.
Apart from that, the internet connection via Squid was stable. I’ve had this before, when the Squid got stuck, either it recovered or crashed completely, then you could access the ipfire web interface via the Squid proxy and just had to restart the Squid. But this time it was different, I could no longer establish a connection to the Ipfire. SSH was deactivated, no chance to start the SSH server to see what was going on on the console. I had no choice but to press the hardware button and then the problems really started.
First I tried everything after the reboot and realized that nothing was working, no web interface, my PC didn’t get its IP from the DHCP, something was wrong… so I went to the ipfire box, last chance to find out something… but when the monitor came on, the firewall log rattled relentlessly, it was not possible to make an entry, the text overwrote everything… I didn’t know anything like that…, so I rebooted and followed the boot process, an error in the file system was displayed, but it could be repaired and was now checked for the second time.
After starting the firewall, the text started again, but I recognized my external IP again, turned off the modem and had the opportunity to log in properly on the ipfire.
I started the setup and set up a native configuration, because I had a card with 2 VLANs assigned. So peace and quiet returned to the IPfire, then I reconfigured the switch because of the VLANs, restarted everything and I had my computer back on the network and access to the Ipfire web interface.
I still have a red error message when booting, the displayed file has something to do with ipset and firewall.local, because both processes are started or tried to be started there, but I’ll get to that in a moment. The path to this file was something like /etc/rc.d/6rc.d/98schlagmichtot.ja

But that was not all, I have restored the previous network configuration with the two VLANs, everything is running normally again, everything works as before, only the IPS can no longer be started, no matter which provider which network card I select. As soon as I press Save, a blank white page is loaded, the options are accepted, but the process is not started.

Now to my suspicions, a few weeks ago I found a script that creates ASN lists based on company names and creates the corresponding networks and enters them directly into the groups created for this purpose in the ipfire firewall group function. Everything is nicely organized, first the networks, then the respective groups, but unfortunately the script also wants to create the corresponding firewall rules, which probably doesn’t work, the script ends with an error message.
Thanks to the groups, however, the rules were created quite quickly and I thought that would be fine. Everything worked without me noticing anything special.
After a while, however, I noticed that after automatic script execution using the fcrontab entry, an error occurred that affected the system, but it took me a while to find the connection.
The iptables entries for my tor node were deleted, and I always wondered why the Tor process was running, but no connections to other nodes were established.
However, this could easily be corrected by restarting the Tor process. So I took the entry out of the fcrontab again and decided to start a manual script so that I could then restart the Tor process so that it would continue normally without any problems.
So everything worked again without me noticing anything.
I suspect that by changing the IPS provider, there were now complications in connection with the groups created by the script, or with the entries in the iptables with the wrong references, which finally completely shot down the system.
Although the script is not executed automatically, the exact same error message also appears when booting the Ipfire…as if it were being executed. The script tries to read the iptables in the same way as during the boot process.
The message says that the iptables do not know the companies or groups and the start of firewall.local is terminated twice with an error.
The script wants to create the firewall rules with the created firewall groups and this leads to the errors that the rules for the Tor node disappear and probably permanently wrong references/groups have been entered with which iptables cannot do anything, which leads to this permanent error message.

I don’t have much idea how to write iptables, how to write scripts or how to read scripts, but my conclusions are logical to the error messages, the script also comes from a trustworthy source, it was written especially for ipfire, but has not been updated for two years, which is probably the reason why it only works partially. Unfortunately the creator has not answered me and my attempts to find the script or the error myself failed, as I said I am a layman who only has a basic understanding of all this.

What is my question to you now… How do I get the IPS running again, or how do I clean up the iptables so that the conflicts no longer occur? Or is my only option a complete reinstallation of IPfire?

@ummeegge With you I see the biggest hope that you can help me to find out what exactly went wrong, because you also provide a similar script, which automates a blocking via the DNS level of the ASN entries with companies, if you want I can link you the script or post it here. The process is similar to yours, IP lists are created, I find this solution even more elegant because of the firewall groups, which you can then use for rules in the web interface, because you can determine exactly who has access to which networks. I would have already shared this with you if it had been error-free, I wanted to try it first to see if it remained stable despite the error, which is probably not the case.

Thank you in advance for reading and maybe someone can help, give their opinion or just laugh at me…

Hi @mumpitz,
difficult to say since there are a lot of different topics (possible problems) in your text. Also, the info is less may a link to the script can help also some warnings, errors are useful otherwise it is difficult to help.

I wish you all a happy new year and all the best for the new year.

Best,

Erik

3 Likes

Hi :smiley:
Yes I wish you too a happy new year.

I see in my research you already know the script … ASN_Script
this is the error message on boot and after execute the script →

—[Restarting firewall]—
iptables v1.8.9 (legacy): Set companies doesn’t exist.

Try `iptables -h’ or ‘iptables --help’ for more information.
iptables v1.8.9 (legacy): Set companies doesn’t exist.

Try `iptables -h’ or ‘iptables --help’ for more information.
iptables v1.8.9 (legacy): Set companies doesn’t exist.

Try `iptables -h’ or ‘iptables --help’ for more information.
ERROR: /etc/sysconfig/firewall.local reload
—[All done!]—

Second Tor entries in iptables gone



After restarting the Tor process


The Tor entries in iptables are back

Inhalt von firewall.local

#!/bin/sh

Used for private firewall rules

IPSETCOMPANY=“/sbin/iptables”

See how we were called.

case “$1” in
start)
# → Automatic generated IPSET COMPANY FW entries in start section BEGIN
${IPSETCOMPANY} -I CUSTOMFORWARD -m set --match-set companies dst -j REJECT
${IPSETCOMPANY} -I CUSTOMINPUT -m set --match-set companies dst -j REJECT
${IPSETCOMPANY} -I CUSTOMOUTPUT -m set --match-set companies dst -j REJECT
# → Automatic generated IPSET COMPANY FW entries in start section END
## add your ‘start’ rules here
;;
stop)
# → Automatic generated IPSET COMPANY FW entries in stop section BEGIN
${IPSETCOMPANY} -F CUSTOMFORWARD
${IPSETCOMPANY} -F CUSTOMINPUT
${IPSETCOMPANY} -F CUSTOMOUTPUT
# → Automatic generated IPSET COMPANY FW entries in stop section END
## add your ‘stop’ rules here
;;
reload)
$0 stop
$0 start
## add your ‘reload’ rules here
;;
*)
echo “Usage: $0 {start|stop|reload}”
;;
esac

This is the file mention at boot /etc/rc.d/rc3.d/S98rc.local–>

#!/bin/sh
###############################################################################

IPFire.org - A linux based firewall

Copyright (C) 2007 Michael Tremer & Christian Schmidt

This program is free software: you can redistribute it and/or modify

it under the terms of the GNU General Public License as published by

the Free Software Foundation, either version 3 of the License, or

(at your option) any later version.

This program is distributed in the hope that it will be useful,

but WITHOUT ANY WARRANTY; without even the implied warranty of

MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the

GNU General Public License for more details.

You should have received a copy of the GNU General Public License

along with this program. If not, see http://www.gnu.org/licenses/.

###############################################################################

Used for private calls after boot

###############################################################################
ipset restore < /etc/ipset/ipset.conf && /etc/sysconfig/firewall.local reload;

I get this in Kernel log after did my last change before ipfire freeze

02:13:24 kernel: note: modprobe[20944] exited with irqs disabled
02:13:24 kernel: CR2: 0000000000000090 CR3: 0000000103360005 CR4: 00000000003706e0
02:13:24 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
02:13:24 kernel: FS: 00007c93f01703c0(0000) GS:ffff92595f040000(0000) knlGS:0000000000000000
02:13:24 kernel: R13: 0000000000000003 R14: 0000000000000000 R15: ffffb053845f7d80
02:13:24 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffc09a5048
02:13:24 kernel: RBP: ffff925606636000 R08: 0000000000000000 R09: 0000000000000000
02:13:24 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
02:13:24 kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
02:13:24 kernel: RSP: 0018:ffffb053845f7ca0 EFLAGS: 00010286
02:13:24 kernel: Code: 4c 8d 24 c5 00 50 9a c0 55 53 e9 92 00 00 00 44 89 f7 e8 80 aa 5d f6 48 89 c5 48 85 c0 0f 84 a5 00 00 00 48 8b 80 40 02 00 00 <8b> 90 90 00 00 00 44 8b 80 98 00 00 00 44 8b 88 9c 00 00 00 8b 88
02:13:24 kernel: RIP: 0010:amd_pstate_ut_check_freq+0x43/0x10e [amd_pstate_ut]
02:13:24 kernel: —[ end trace 0000000000000000 ]—
02:13:24 kernel: CR2: 0000000000000090
02:13:24 kernel: Modules linked in: amd_pstate_ut(+) xt_set ip_set_hash_net ip_set xt_hashlimit x t_mac xt_multiport xt_policy xt_TCPMSS xt_conntrack xt_comment ipt_REJECT nf_rej ect_ipv4 xt_LOG xt_limit xt_mark xt_connmark nf_log_syslog iptable_raw iptable_m angle iptable_filter vfat fat sch_cake iwlmvm 8021q garp mac80211 intel_tcc_cool ing x86_pkg_temp_thermal intel_powerclamp coretemp libarc4 kvm_intel iwlwifi iTC O_wdt iTCO_vendor_support mei_pxp ee1004 e1000e igb kvm cfg80211 wmi_bmof i2c_i8 01 dca ptp tpm_crb i2c_algo_bit irqbypass tpm_tis mei_me i2c_smbus tpm_tis_core pps_core rfkill pcspkr i2c_core mei intel_pch_thermal tpm acpi_tad acpi_pad lp p arport_pc parport efivarfs crct10dif_pclmul crc32_pclmul polyval_clmulni polyval generic ghash_clmulni_intel sha512_ssse3 video wmi dm_mirror dm_region_hash dm log dm_mod
02:13:24 kernel:
02:13:24 kernel: R13: 0000000000000000 R14: 0000000002307c00 R15: 0000000002308070
02:13:24 kernel: R10: 0000000000000071 R11: 0000000000000206 R12: 00000000023164a0
02:13:24 kernel: RBP: 000000000042b498 R08: 0000000000000001 R09: 0000000000000000
02:13:24 kernel: RDX: 000000000042b498 RSI: 00000000000044d7 RDI: 00000000023164a0
02:13:24 kernel: RAX: ffffffffffffffda RBX: 0000000002307b80 RCX: 00007c93efb2ffde
02:13:24 kernel: RSP: 002b:00007ffd987a0448 EFLAGS: 00000206 ORIG_RAX: 00000000000000af
02:13:24 kernel: Code: 48 8b 0d 45 5e 0c 00 f7 d8 64 89 01 48 83 c8 ff c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 49 89 ca b8 af 00 00 00 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 8b 0d 12 5e 0c 00 f7 d8 64 89 01 48
02:13:24 kernel: RIP: 0033:0x7c93efb2ffde
02:13:24 kernel: entry_SYSCALL_64_after_hwframe+0x64/0xce
02:13:24 kernel: ? exit_to_user_mode_prepare+0x1a/0x130
02:13:24 kernel: ? do_syscall_64+0x6c/0x90
02:13:24 kernel: ? syscall_exit_to_user_mode+0x37/0x50
02:13:24 kernel: ? exit_to_user_mode_prepare+0x1a/0x130
02:13:24 kernel: ? do_syscall_64+0x6c/0x90
02:13:24 kernel: ? syscall_exit_to_user_mode+0x37/0x50
02:13:24 kernel: ? exit_to_user_mode_prepare+0x1a/0x130
02:13:24 kernel: do_syscall_64+0x5d/0x90
02:13:24 kernel: __do_sys_init_module+0x18a/0x1c0
02:13:24 kernel: do_init_module+0x4a/0x1f0
02:13:24 kernel: do_one_initcall+0x5a/0x230
02:13:24 kernel: ? 0xffffffffc0405000
02:13:24 kernel: amd_pstate_ut_init+0x1e/0x1000 [amd_pstate_ut]
02:13:24 kernel: ? amd_pstate_ut_check_freq+0x43/0x10e [amd_pstate_ut]
02:13:24 kernel: ? asm_exc_page_fault+0x26/0x30
02:13:24 kernel: ? exc_page_fault+0x444/0x6d0
02:13:24 kernel: ? page_fault_oops+0x180/0x2d0
02:13:24 kernel: ? __die+0x59/0xa9
02:13:24 kernel:
02:13:24 kernel: Call Trace:
02:13:24 kernel: CR2: 0000000000000090 CR3: 0000000103360005 CR4: 00000000003706e0
02:13:24 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
02:13:24 kernel: FS: 00007c93f01703c0(0000) GS:ffff92595f040000(0000) knlGS:0000000000000000
02:13:24 kernel: R13: 0000000000000003 R14: 0000000000000000 R15: ffffb053845f7d80
02:13:24 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffffffffc09a5048
02:13:24 kernel: RBP: ffff925606636000 R08: 0000000000000000 R09: 0000000000000000
02:13:24 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
02:13:24 kernel: RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000000000000
02:13:24 kernel: RSP: 0018:ffffb053845f7ca0 EFLAGS: 00010286
02:13:24 kernel: Code: 4c 8d 24 c5 00 50 9a c0 55 53 e9 92 00 00 00 44 89 f7 e8 80 aa 5d f6 48 89 c5 48 85 c0 0f 84 a5 00 00 00 48 8b 80 40 02 00 00 <8b> 90 90 00 00 00 44 8b 80 98 00 00 00 44 8b 88 9c 00 00 00 8b 88
02:13:24 kernel: RIP: 0010:amd_pstate_ut_check_freq+0x43/0x10e [amd_pstate_ut]
02:13:24 kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./H370M-ITX/ac, BIOS P4.00 03/19/2019
02:13:24 kernel: CPU: 1 PID: 20944 Comm: modprobe Not tainted 6.1.61-ipfire #1
02:13:24 kernel: Oops: 0000 [#1] PREEMPT SMP PTI
02:13:24 kernel: PGD 80000001050f4067 P4D 80000001050f4067 PUD 107ba2067 PMD 0
02:13:24 kernel: #PF: error_code(0x0000) - not-present page
02:13:24 kernel: #PF: supervisor read access in kernel mode
02:13:24 kernel: BUG: kernel NULL pointer dereference, address: 0000000000000090
02:13:24 kernel: amd_pstate_ut: 3 amd_pstate_ut_check_perf^I fail!
02:13:24 kernel: amd_pstate_ut: amd_pstate_ut_check_perf cppc_get_perf_caps ret=-19 error!
02:13:24 kernel: amd_pstate_ut: 2 amd_pstate_ut_check_enabled^I success!
02:13:24 kernel: amd_pstate_ut: 1 amd_pstate_ut_acpi_cpc_valid^I fail!
02:13:24 kernel: amd_pstate_ut: amd_pstate_ut_acpi_cpc_valid the _CPC object is not present in SB IOS!

So this was my first look to find messages and hints … don’t know if it helps

Hello @mumpitz ,
OK, let´s try to solve them step by step.

this entries are from my script from this topic → Blocking Reverse Proxys from cloudflare - #17 by ummeegge . There was a bug in it that´s why i loaded a fixed version at that time up and left a note → Blocking Reverse Proxys from cloudflare - #43 by ummeegge which fixed the

Try `iptables -h’ or ‘iptables --help’ for more information.
iptables v1.8.9 (legacy): Set companies doesn’t exist.

problem. It seems that you used the old version and/or never deleted those entries which should be done via the CLI helper →


just use ‘d’ and [ENTER] .

This ASN script from maloe and Mike Kuketz differs to my script so the firewall.local entries are related to my script and have nothing to do with their script since they use firewall groups for this.
My concern at that time was that to many entries in the IPTable chains (vast lists) can bring the system down that´s why i used IPSet for this but also only firewall.local which have a better overview and not too much work to delete it manually.

this entry is a restore command for IPSet sets and i use it also in my script but not in this location but in ‘/etc/sysconfig/rc.local’ so i don´t know where it comes from.
Also, i currently can not say much to this ASN_IPFire script (haven´t used it) and i would also use it for the first only in a VM to test or debug it.

According to the Tor problem, i would fix for the first the company_blocker (delete both of them) and check your system if it runs again.

According to the kernel message, this jumps in my eyes

. I would try to split the problems in different parts and use the problem substitution step by step to see how the system reacts.

Hopefully some light comes into this.

Best,

Erik

P.S.: Am on the road without net the next days so some days needed to answer again if needed, or may some others jump in…

4 Likes

fixed :smiley:

it is not clear for me, i changed the code to the new one, but did not know if I execute it again… but it seems so… I removed your script entries now and remove all the firewall entries with the ASN_Script …did reboot and my IDS is running again, it is an unbelievable feeling not to lose the ipfire installation…

with Tor i get a problem after executing the ASN_Script, that the iptables rules were gone , so now i will have a try and i will see of this happen again.

First of all, a big thank you from me! @ummeegge
You have brought order into it, I thought all the time only to the ASN script that here the error is to be looked for, because I always had the error messages after the execution of this.
After restarting, the IPS started up again and the system was saved for the time being

this error do not appear every day i count 4 times since 01.12.

What could that be?

Hello @mumpitz ,

It appears to be a problem (NULL pointer) for your OEM (ASRock ?) board to handle the ‘amd_pstate_ut’ kernel module correctly which is responsible for power management and performance scaling on AMD processors (P-State). A NULL pointer does not point to a valid memory address so it can causes also system crashes at the same time it could be a kernel bug but also an module programming problem…

Possible solutions?: Have read that an update of your BIOS or an updated version of ‘amd_pstate_ut’ but also a memory test might help. In case non of them works, the manufacturer of the OEM board or the developer of the module should be contacted in question for further support.
This is not my humble opinion but to save also some credits to an AI beneath some other useful forum and Github issues since i do not have the in depth knowledge of such kernel problems.

For the other topics, glad that you could bring your system back but i would check it further since there seems to be also other problems not related to the company_blocker thing.

I would also check WHY the ASN_Script modfies the Tor IPTable chains resp. when it does modify it. You can debug the script with e.g.
bash -x script.sh
or ask the developer (i think notabug.org have also an issue section) which is always a good idea if you can break down the problem before as close as you can.

Best,

Erik

[EDIT] Upps:
https://lkml.org/lkml/2023/5/15/512
If related to the link above, with Kernel version >= 6.7 a potential? fix is included → Linux 6.7-rc4 [LWN.net]
and for general information →
amd-pstate CPU Performance Scaling Driver — The Linux Kernel documentation

1 Like

Also…
If it’s a “recent enough” AMD CPU (Ryzen and derived), Sunnyvale corp released a bazilion of AGESA updates due to issues and vulnerabilities.
So.
A bios Update for the mainboard might be pleasant idea…

1 Like

Thank you for your answers, but you both spoken about an AMD CPU Problem, right?
Because my ipfire has an i5 Intel CPU. I think the mainboard is from Gigabyte, but I will check it.
Could this kernel error also mean problems for me with Intel boards and CPUs?

But next good news, after I had clean a little bit the installed addons, some dependencies in Pakfire were no longer correct, no longer the log is overwrting in the terminal.

Unfortunately, the ASN script still deletes the Tor iptables entries. So I would only have to add another command at the end of the script where the firewall is restarted to restart Tor as well, but it would be better if the entries would not disappear.

This and other kernel errors might (or might not!) triggered by imperfect management from the CPU, BIOS (UEFI, whatever) of some commands from kernel. Also Intel has released quite a few updates (the Spectre-Meltdown gig started in 2019) so a lot of producers released bios updates.
Might this update totally solve the issue? I don’t know. Looking deeper into logs might give you messages to look for around kernel forums for identify if this can be solved via BIOS update.
However… Updating usually brings fixes. It’s a unpleasant task sometimes, and require some care (like UPS). But i suggest to verify that anyway.

Hi all,
@mumpitz

are you sure that the tor chains are the only one ? I think this behavior is in general not wanted (also not with an restart), it looks like a bug which have currently unpredictable effects in an important section.

Best,

Erik

Yes and I checked many more things until now

Firstly the kernel message

amd_pstate_ut: amd_pstate_ut_acpi_cpc_valid the _CPC object is not present in SBIOS!

only appears if I installed the addon cpufrequltis which wants to execute amd cpu functions for unknown reasons, although I have an intel i5 processor.

Most time till Sep/2023 4 days after core 179 update, for the first time, I got this Kernel Warnings, I went back to Feb/2023

WARNING: CPU: 5 PID: 118 at drivers/net/wireless …: 1 Time(s)
WARNING: CPU: 5 PID: 118 at kernel/rcu/tree_plug …: 1 Time(s)

In the ASN script at the end there is a command to restart the firewall

/etc/init.d/firewall restart 1> /dev/null

and here is the problem, if I also restart the firewall manually the entries for the tor relay in the iptables were gone, it is an ipfire problem and not the script itself. Can anybody confirm this behavior, at restarting the firewall the entries are gone in the iptables running tor relay ?

Hello @mumpitz ,

this one is, i am sad to say, busted. There are more problems if an
/etc/init.d/firewall restart
will be executed but first things first.
The topic name is “Weird problem how should i proceed” and a community rule (not sure if it is written but it makes sense) is, try to avoid different problems in one topic. So my idea is, open up other topics which describes one problem and this as precise as possible e.g. →

  1. Tor chains are deleted if the firewall will be manually restarted (have a possible solution for this but not for the other problem which i discover).
  2. CPUfrequtil specifies wrong CPU
    or something like that, so we have a better readability in here. Also, some ending words which problems have been solved in here and what else needs to be done (step-by-step or problem specific) can be linked in here.

Should we do this like that ?

Best,

Erik

3 Likes

I did it before but nobody answered of my topic…

I had already explained that everything was solved by your resolution of where the error messages came from. I was convinced that it had something to do with the ASN script, but it was your script. Both at the same time does not work.
I had already mentioned in the other thread that restarting the firewall rules causes the Tor chains to be deleted, but no one else responded.
I only found out now that the ASN script executes a restart.

As suggested, I carried out a bios update, but this did not bring any improvement.
My search then brought up the tip
“After uninstalling cpufrequtil and restarting IPFire, amd_pstate_ut errors disappear.
After reinstalling cpufrequtils and restarting IPFire, the errors are visible.”
from here

This solves everything except the question of why the Tor entries in the iptables are deleted when the firewall is restarted.

Alright, so check out then the last problem in here → Tor stop working without stop the process or give an error message? .

Best,

Erik