Hello IPFire community
At first sorry for my bad English.
We have a 2 years old all time good running IPFire systems and since the last version (139), we have a problem with this. So that she stop working after a while (about a week later, after the last reboot). I have only found this bad things in the log file.
date: Februar 05, 2020
04:57:30 kernel: watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [collectd:11676]
04:57:30 kernel: Modules linked in: ledtrig_netdev ledtrig_heartbeat ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_hashlimit xt_mark xt_policy xt_TCPMSS nf_nat_irc nf_conntrack_irc nf_nat_tftp nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat_h323 nf_conntrack_h323 xt_CT xt_helper nf_nat_sip nf_conntrack_sip xt_conntrack xt_comment ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG xt_limit iptable_raw iptable_mangle iptabl
I have also upload the kernel log file from the first event of the lockup from one week ago that have more error messages by this event.
Is this a hardware problem or maybe a bug in the kernel and How can I check where I get this problem?
Hardware:
kernel_release: 4.14.154-ipfire
language: de.utf8
memory: 4084900
model: apu3
release: IPFire 2.23 (x86_64) - core139
root_size: 15638616
vendor: PC Engines
virtual: false
bios_version: v4.10.0.3
running services:
default IPFire with dhcp-derver
install addons:
check_mk_agent
dehydrated
firmware-update
flashrom
iftop
pcengines-apu-firmware
The kernel log from the first event of the lockup from one week ago that have more error messages by this event.
IPFire diagnostics
Abschnitt: kernel
Datum: Januar 30, 2020
02:55:53 kernel: watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [collectd:12623]
02:55:53 kernel: Modules linked in: ledtrig_netdev ledtrig_heartbeat ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_hashlimit xt_mark xt_policy xt_TCPMSS nf_nat_irc nf_conntrack_irc nf_nat_tftp nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat_h323 nf_conntrack_h323 xt_CT xt_helper nf_nat_sip nf_conntrack_sip xt_conntrack xt_comment ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG xt_limit iptable_raw iptable_mangle iptable_filter sch_fq_codel amd64_edac_mod edac_mce_amd kvm_amd kvm igb irqbypass crct10dif_pclmul crc32_pclmul sp5100_tco ghash_clmulni_intel ptp pps_core i2c_piix4 pcspkr dca ccp i2c_algo_bit fam15h_power k10temp i2c_core leds_gpio gpio_keys pinctrl_amd pcc_cpufreq acpi_cpufreq lp parport_pc parport video sdhci_pci sdhci mmc_core
02:55:53 kernel: CPU: 1 PID: 12623 Comm: collectd Not tainted 4.14.154-ipfire #1
02:55:53 kernel: Hardware name: PC Engines apu3/apu3, BIOS v4.10.0.3 11/07/2019
02:55:53 kernel: task: ffff8fb1681a2940 task.stack: ffffa0f40095c000
02:55:53 kernel: RIP: 0010:smp_call_function_single+0xce/0xf0
02:55:53 kernel: RSP: 0018:ffffa0f40095fd00 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff10
02:55:53 kernel: RAX: 0000000000000000 RBX: 00000006fb2851f6 RCX: 0000000000000000
02:55:53 kernel: RDX: 0000000000000001 RSI: 00000000000000fb RDI: 0000000000000246
02:55:53 kernel: RBP: ffffa0f40095fd58 R08: ffff8fb1673ecab8 R09: ffff8fb169b51e40
02:55:53 kernel: R10: 0000000000000004 R11: 0000000000000000 R12: ffff8fb168658000
02:55:53 kernel: R13: ffffffffa64d1600 R14: 0000000000000001 R15: ffff8fb168992980
02:55:53 kernel: FS: 00007499b5597700(0000) GS:ffff8fb16ec80000(0000) knlGS:0000000000000000
02:55:53 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
02:55:53 kernel: CR2: 00007b99454f5880 CR3: 0000000128524000 CR4: 00000000000406e0
02:55:53 kernel: Call Trace:
02:55:53 kernel: ? arch_smt_update.cold.6+0x7a/0x7a
02:55:53 kernel: ? ktime_get+0x36/0xa0
02:55:53 kernel: aperfmperf_snapshot_cpu+0x40/0x50
02:55:53 kernel: arch_freq_get_on_cpu+0x2c/0x70
02:55:53 kernel: show_scaling_cur_freq+0x15/0x80
02:55:53 kernel: show+0x33/0x50
02:55:53 kernel: sysfs_kf_seq_show+0x93/0x110
02:55:53 kernel: seq_read+0x16b/0x460
02:55:53 kernel: __vfs_read+0x36/0x1b0
02:55:53 kernel: vfs_read+0x91/0x140
02:55:53 kernel: SyS_read+0x57/0xd0
02:55:53 kernel: do_syscall_64+0x67/0x100
02:55:53 kernel: entry_SYSCALL_64_after_hwframe+0x3d/0xa2
02:55:53 kernel: RIP: 0033:0x7499b96422c4
02:55:53 kernel: RSP: 002b:00007499b55966a0 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
02:55:53 kernel: RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007499b96422c4
02:55:53 kernel: RDX: 0000000000001000 RSI: 00007499a80a59c0 RDI: 0000000000000004
02:55:53 kernel: RBP: 00007499a80a59c0 R08: 0000000000000000 R09: 00007499a80007f0
02:55:53 kernel: R10: 00007499a80008d0 R11: 0000000000000246 R12: 0000000000001000
02:55:53 kernel: R13: 00007499b9718340 R14: 000000000000000f R15: 00007499a8000b20
02:55:53 kernel: Code: 48 8b 4c 24 38 65 48 33 0c 25 28 00 00 00 75 34 c9 c3 48 89 d1 48 89 f2 48 89 e6 e8 7d fe ff ff 8b 54 24 18 83 e2 01 74 0b f3 90 <8b> 54 24 18 83 e2 01 75 f5 eb ca 8b 05 f9 bc 50 01 85 c0 75 88
02:56:11 kernel: INFO: rcu_sched detected stalls on CPUs/tasks:
02:56:11 kernel: ^I0-…: (1 GPs behind) idle=a3e/140000000000001/0 softirq=8051191/8051192 fqs=5982
02:56:11 kernel: ^I(detected by 2, t=18002 jiffies, g=2984325, c=2984324, q=339)
02:56:11 kernel: Sending NMI from CPU 2 to CPUs 0:
02:56:21 kernel: watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [collectd:12623]
02:56:21 kernel: Modules linked in: ledtrig_netdev ledtrig_heartbeat ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_hashlimit xt_mark xt_policy xt_TCPMSS nf_nat_irc nf_conntrack_irc nf_nat_tftp nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat_h323 nf_conntrack_h323 xt_CT xt_helper nf_nat_sip nf_conntrack_sip xt_conntrack xt_comment ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG xt_limit iptable_raw iptable_mangle iptable_filter sch_fq_codel amd64_edac_mod edac_mce_amd kvm_amd kvm igb irqbypass crct10dif_pclmul crc32_pclmul sp5100_tco ghash_clmulni_intel ptp pps_core i2c_piix4 pcspkr dca ccp i2c_algo_bit fam15h_power k10temp i2c_core leds_gpio gpio_keys pinctrl_amd pcc_cpufreq acpi_cpufreq lp parport_pc parport video sdhci_pci sdhci mmc_core
02:56:21 kernel: CPU: 1 PID: 12623 Comm: collectd Tainted: G L 4.14.154-ipfire #1
02:56:21 kernel: Hardware name: PC Engines apu3/apu3, BIOS v4.10.0.3 11/07/2019
02:56:21 kernel: task: ffff8fb1681a2940 task.stack: ffffa0f40095c000
02:56:21 kernel: RIP: 0010:smp_call_function_single+0xce/0xf0
02:56:21 kernel: RSP: 0018:ffffa0f40095fd00 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff10
02:56:21 kernel: RAX: 0000000000000000 RBX: 00000006fb2851f6 RCX: 0000000000000000
02:56:21 kernel: RDX: 0000000000000001 RSI: 00000000000000fb RDI: 0000000000000246
02:56:21 kernel: RBP: ffffa0f40095fd58 R08: ffff8fb1673ecab8 R09: ffff8fb169b51e40
02:56:21 kernel: R10: 0000000000000004 R11: 0000000000000000 R12: ffff8fb168658000
02:56:21 kernel: R13: ffffffffa64d1600 R14: 0000000000000001 R15: ffff8fb168992980
02:56:21 kernel: FS: 00007499b5597700(0000) GS:ffff8fb16ec80000(0000) knlGS:0000000000000000
02:56:21 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
02:56:21 kernel: CR2: 00007b99454f5880 CR3: 0000000128524000 CR4: 00000000000406e0
02:56:21 kernel: Call Trace:
02:56:21 kernel: ? arch_smt_update.cold.6+0x7a/0x7a
02:56:21 kernel: ? ktime_get+0x36/0xa0
02:56:21 kernel: aperfmperf_snapshot_cpu+0x40/0x50
02:56:21 kernel: arch_freq_get_on_cpu+0x2c/0x70
02:56:21 kernel: show_scaling_cur_freq+0x15/0x80
02:56:21 kernel: show+0x33/0x50
02:56:21 kernel: sysfs_kf_seq_show+0x93/0x110
02:56:21 kernel: seq_read+0x16b/0x460
02:56:21 kernel: __vfs_read+0x36/0x1b0
02:56:21 kernel: vfs_read+0x91/0x140
02:56:21 kernel: SyS_read+0x57/0xd0
02:56:21 kernel: do_syscall_64+0x67/0x100
02:56:21 kernel: entry_SYSCALL_64_after_hwframe+0x3d/0xa2
02:56:21 kernel: RIP: 0033:0x7499b96422c4
02:56:21 kernel: RSP: 002b:00007499b55966a0 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
02:56:21 kernel: RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007499b96422c4
02:56:21 kernel: RDX: 0000000000001000 RSI: 00007499a80a59c0 RDI: 0000000000000004
02:56:21 kernel: RBP: 00007499a80a59c0 R08: 0000000000000000 R09: 00007499a80007f0
02:56:21 kernel: R10: 00007499a80008d0 R11: 0000000000000246 R12: 0000000000001000
02:56:21 kernel: R13: 00007499b9718340 R14: 000000000000000f R15: 00007499a8000b20
02:56:21 kernel: Code: 48 8b 4c 24 38 65 48 33 0c 25 28 00 00 00 75 34 c9 c3 48 89 d1 48 89 f2 48 89 e6 e8 7d fe ff ff 8b 54 24 18 83 e2 01 74 0b f3 90 <8b> 54 24 18 83 e2 01 75 f5 eb ca 8b 05 f9 bc 50 01 85 c0 75 88
02:56:49 kernel: watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [collectd:12623]
02:56:49 kernel: Modules linked in: ledtrig_netdev ledtrig_heartbeat ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_hashlimit xt_mark xt_policy xt_TCPMSS nf_nat_irc nf_conntrack_irc nf_nat_tftp nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat_h323 nf_conntrack_h323 xt_CT xt_helper nf_nat_sip nf_conntrack_sip xt_conntrack xt_comment ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG xt_limit iptable_raw iptable_mangle iptable_filter sch_fq_codel amd64_edac_mod edac_mce_amd kvm_amd kvm igb irqbypass crct10dif_pclmul crc32_pclmul sp5100_tco ghash_clmulni_intel ptp pps_core i2c_piix4 pcspkr dca ccp i2c_algo_bit fam15h_power k10temp i2c_core leds_gpio gpio_keys pinctrl_amd pcc_cpufreq acpi_cpufreq lp parport_pc parport video sdhci_pci sdhci mmc_core
02:56:49 kernel: CPU: 1 PID: 12623 Comm: collectd Tainted: G L 4.14.154-ipfire #1
02:56:49 kernel: Hardware name: PC Engines apu3/apu3, BIOS v4.10.0.3 11/07/2019
02:56:49 kernel: task: ffff8fb1681a2940 task.stack: ffffa0f40095c000
02:56:49 kernel: RIP: 0010:smp_call_function_single+0xce/0xf0
02:56:49 kernel: RSP: 0018:ffffa0f40095fd00 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff10
02:56:49 kernel: RAX: 0000000000000000 RBX: 00000006fb2851f6 RCX: 0000000000000000
02:56:49 kernel: RDX: 0000000000000001 RSI: 00000000000000fb RDI: 0000000000000246
02:56:49 kernel: RBP: ffffa0f40095fd58 R08: ffff8fb1673ecab8 R09: ffff8fb169b51e40
02:56:49 kernel: R10: 0000000000000004 R11: 0000000000000000 R12: ffff8fb168658000
02:56:49 kernel: R13: ffffffffa64d1600 R14: 0000000000000001 R15: ffff8fb168992980
02:56:49 kernel: FS: 00007499b5597700(0000) GS:ffff8fb16ec80000(0000) knlGS:0000000000000000
02:56:49 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
02:56:49 kernel: CR2: 00007b99454f5880 CR3: 0000000128524000 CR4: 00000000000406e0
02:56:49 kernel: Call Trace:
02:56:49 kernel: ? arch_smt_update.cold.6+0x7a/0x7a
02:56:49 kernel: ? ktime_get+0x36/0xa0
02:56:49 kernel: aperfmperf_snapshot_cpu+0x40/0x50
02:56:49 kernel: arch_freq_get_on_cpu+0x2c/0x70
02:56:49 kernel: show_scaling_cur_freq+0x15/0x80
02:56:49 kernel: show+0x33/0x50
02:56:49 kernel: sysfs_kf_seq_show+0x93/0x110
02:56:49 kernel: seq_read+0x16b/0x460
02:56:49 kernel: __vfs_read+0x36/0x1b0
02:56:49 kernel: vfs_read+0x91/0x140
02:56:49 kernel: SyS_read+0x57/0xd0
02:56:49 kernel: do_syscall_64+0x67/0x100
02:56:49 kernel: entry_SYSCALL_64_after_hwframe+0x3d/0xa2
02:56:49 kernel: RIP: 0033:0x7499b96422c4
02:56:49 kernel: RSP: 002b:00007499b55966a0 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
02:56:49 kernel: RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007499b96422c4
02:56:49 kernel: RDX: 0000000000001000 RSI: 00007499a80a59c0 RDI: 0000000000000004
02:56:49 kernel: RBP: 00007499a80a59c0 R08: 0000000000000000 R09: 00007499a80007f0
02:56:49 kernel: R10: 00007499a80008d0 R11: 0000000000000246 R12: 0000000000001000
02:56:49 kernel: R13: 00007499b9718340 R14: 000000000000000f R15: 00007499a8000b20
02:56:49 kernel: Code: 48 8b 4c 24 38 65 48 33 0c 25 28 00 00 00 75 34 c9 c3 48 89 d1 48 89 f2 48 89 e6 e8 7d fe ff ff 8b 54 24 18 83 e2 01 74 0b f3 90 <8b> 54 24 18 83 e2 01 75 f5 eb ca 8b 05 f9 bc 50 01 85 c0 75 88
02:57:09 kernel: watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [sshd:8871]
02:57:09 kernel: Modules linked in: ledtrig_netdev ledtrig_heartbeat ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_hashlimit xt_mark xt_policy xt_TCPMSS nf_nat_irc nf_conntrack_irc nf_nat_tftp nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat_h323 nf_conntrack_h323 xt_CT xt_helper nf_nat_sip nf_conntrack_sip xt_conntrack xt_comment ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG xt_limit iptable_raw iptable_mangle iptable_filter sch_fq_codel amd64_edac_mod edac_mce_amd kvm_amd kvm igb irqbypass crct10dif_pclmul crc32_pclmul sp5100_tco ghash_clmulni_intel ptp pps_core i2c_piix4 pcspkr dca ccp i2c_algo_bit fam15h_power k10temp i2c_core leds_gpio gpio_keys pinctrl_amd pcc_cpufreq acpi_cpufreq lp parport_pc parport video sdhci_pci sdhci mmc_core
02:57:09 kernel: CPU: 2 PID: 8871 Comm: sshd Tainted: G L 4.14.154-ipfire #1
02:57:09 kernel: Hardware name: PC Engines apu3/apu3, BIOS v4.10.0.3 11/07/2019
02:57:09 kernel: task: ffff8fb168e2b700 task.stack: ffffa0f4009f4000
02:57:09 kernel: RIP: 0010:smp_call_function_many+0x1cc/0x230
02:57:09 kernel: RSP: 0018:ffffa0f4009f7c30 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff10
02:57:09 kernel: RAX: 0000000000000000 RBX: ffff8fb16ed21fc0 RCX: ffff8fb16ec25480
02:57:09 kernel: RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8fb16ed21fc8
02:57:09 kernel: RBP: ffff8fb16ed21fc8 R08: ffffffffffffffff R09: 000000000000000b
02:57:09 kernel: R10: ffffeb5e049bca80 R11: ffffa0f4009f7d48 R12: ffffffffa5468210
02:57:09 kernel: R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000040
02:57:09 kernel: FS: 00007289dfb3c740(0000) GS:ffff8fb16ed00000(0000) knlGS:0000000000000000
02:57:09 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
02:57:09 kernel: CR2: 00007289dfc79e10 CR3: 0000000126f32000 CR4: 00000000000406e0
02:57:09 kernel: Call Trace:
02:57:09 kernel: ? load_new_mm_cr3+0xc0/0xc0
02:57:09 kernel: on_each_cpu+0x28/0x60
02:57:09 kernel: flush_tlb_kernel_range+0x6b/0xb0
02:57:09 kernel: __purge_vmap_area_lazy+0x4d/0xb0
02:57:09 kernel: vm_unmap_aliases+0xde/0x110
02:57:09 kernel: change_page_attr_set_clr+0x2a3/0x470
02:57:09 kernel: set_memory_ro+0x26/0x30
02:57:09 kernel: bpf_prog_select_runtime+0x4e/0xe0
02:57:09 kernel: bpf_prepare_filter+0x527/0x5a0
02:57:09 kernel: bpf_prog_create_from_user+0xbb/0x110
02:57:09 kernel: ? hardlockup_detector_perf_enable.cold.6+0x1b/0x1b
02:57:09 kernel: do_seccomp+0x256/0x6d0
02:57:09 kernel: SyS_prctl+0x25d/0x540
02:57:09 kernel: do_syscall_64+0x67/0x100
02:57:09 kernel: entry_SYSCALL_64_after_hwframe+0x3d/0xa2
02:57:09 kernel: RIP: 0033:0x7289dfc79e1a
02:57:09 kernel: RSP: 002b:00007ffc712c19a8 EFLAGS: 00000246 ORIG_RAX: 000000000000009d
02:57:09 kernel: RAX: ffffffffffffffda RBX: 00007ffc712c19b0 RCX: 00007289dfc79e1a
02:57:09 kernel: RDX: 000058cdba81c200 RSI: 0000000000000002 RDI: 0000000000000016
02:57:09 kernel: RBP: 000058cdbc045bd0 R08: 0000000000000000 R09: 00000000ffffffff
02:57:09 kernel: R10: 00007289dfc79e1a R11: 0000000000000246 R12: 000058cdbc03c710
02:57:09 kernel: R13: 0000000000000004 R14: 0000000000000000 R15: 000058cdba7d3f72
02:57:09 kernel: Code: 89 c7 e8 88 5f 67 00 3b 05 a6 94 01 01 0f 83 b7 fe ff ff 48 63 d0 48 8b 0b 48 03 0c d5 a0 a6 2e a6 8b 51 18 83 e2 01 74 0a f3 90 <8b> 51 18 83 e2 01 75 f6 eb c8 48 c7 c2 f0 fd 52 a6 48 89 ee 89
02:57:17 kernel: watchdog: BUG: soft lockup - CPU#1 stuck for 22s! [collectd:12623]
02:57:17 kernel: Modules linked in: ledtrig_netdev ledtrig_heartbeat ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_hashlimit xt_mark xt_policy xt_TCPMSS nf_nat_irc nf_conntrack_irc nf_nat_tftp nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat_h323 nf_conntrack_h323 xt_CT xt_helper nf_nat_sip nf_conntrack_sip xt_conntrack xt_comment ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG xt_limit iptable_raw iptable_mangle iptable_filter sch_fq_codel amd64_edac_mod edac_mce_amd kvm_amd kvm igb irqbypass crct10dif_pclmul crc32_pclmul sp5100_tco ghash_clmulni_intel ptp pps_core i2c_piix4 pcspkr dca ccp i2c_algo_bit fam15h_power k10temp i2c_core leds_gpio gpio_keys pinctrl_amd pcc_cpufreq acpi_cpufreq lp parport_pc parport video sdhci_pci sdhci mmc_core
02:57:17 kernel: CPU: 1 PID: 12623 Comm: collectd Tainted: G L 4.14.154-ipfire #1
02:57:17 kernel: Hardware name: PC Engines apu3/apu3, BIOS v4.10.0.3 11/07/2019
02:57:17 kernel: task: ffff8fb1681a2940 task.stack: ffffa0f40095c000
02:57:17 kernel: RIP: 0010:smp_call_function_single+0xce/0xf0
02:57:17 kernel: RSP: 0018:ffffa0f40095fd00 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff10
02:57:17 kernel: RAX: 0000000000000000 RBX: 00000006fb2851f6 RCX: 0000000000000000
02:57:17 kernel: RDX: 0000000000000001 RSI: 00000000000000fb RDI: 0000000000000246
02:57:17 kernel: RBP: ffffa0f40095fd58 R08: ffff8fb1673ecab8 R09: ffff8fb169b51e40
02:57:17 kernel: R10: 0000000000000004 R11: 0000000000000000 R12: ffff8fb168658000
02:57:17 kernel: R13: ffffffffa64d1600 R14: 0000000000000001 R15: ffff8fb168992980
02:57:17 kernel: FS: 00007499b5597700(0000) GS:ffff8fb16ec80000(0000) knlGS:0000000000000000
02:57:17 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
02:57:17 kernel: CR2: 00007b99454f5880 CR3: 0000000128524000 CR4: 00000000000406e0
02:57:17 kernel: Call Trace:
02:57:17 kernel: ? arch_smt_update.cold.6+0x7a/0x7a
02:57:17 kernel: ? ktime_get+0x36/0xa0
02:57:17 kernel: aperfmperf_snapshot_cpu+0x40/0x50
02:57:17 kernel: arch_freq_get_on_cpu+0x2c/0x70
02:57:17 kernel: show_scaling_cur_freq+0x15/0x80
02:57:17 kernel: show+0x33/0x50
02:57:17 kernel: sysfs_kf_seq_show+0x93/0x110
02:57:17 kernel: seq_read+0x16b/0x460
02:57:17 kernel: __vfs_read+0x36/0x1b0
02:57:17 kernel: vfs_read+0x91/0x140
02:57:17 kernel: SyS_read+0x57/0xd0
02:57:17 kernel: do_syscall_64+0x67/0x100
02:57:17 kernel: entry_SYSCALL_64_after_hwframe+0x3d/0xa2
02:57:17 kernel: RIP: 0033:0x7499b96422c4
02:57:17 kernel: RSP: 002b:00007499b55966a0 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
02:57:17 kernel: RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007499b96422c4
02:57:17 kernel: RDX: 0000000000001000 RSI: 00007499a80a59c0 RDI: 0000000000000004
02:57:17 kernel: RBP: 00007499a80a59c0 R08: 0000000000000000 R09: 00007499a80007f0
02:57:17 kernel: R10: 00007499a80008d0 R11: 0000000000000246 R12: 0000000000001000
02:57:17 kernel: R13: 00007499b9718340 R14: 000000000000000f R15: 00007499a8000b20
02:57:17 kernel: Code: 48 8b 4c 24 38 65 48 33 0c 25 28 00 00 00 75 34 c9 c3 48 89 d1 48 89 f2 48 89 e6 e8 7d fe ff ff 8b 54 24 18 83 e2 01 74 0b f3 90 <8b> 54 24 18 83 e2 01 75 f5 eb ca 8b 05 f9 bc 50 01 85 c0 75 88
02:57:37 kernel: watchdog: BUG: soft lockup - CPU#2 stuck for 23s! [sshd:8871]
02:57:37 kernel: Modules linked in: ledtrig_netdev ledtrig_heartbeat ipt_MASQUERADE nf_nat_masquerade_ipv4 xt_hashlimit xt_mark xt_policy xt_TCPMSS nf_nat_irc nf_conntrack_irc nf_nat_tftp nf_conntrack_tftp nf_nat_ftp nf_conntrack_ftp nf_nat_h323 nf_conntrack_h323 xt_CT xt_helper nf_nat_sip nf_conntrack_sip xt_conntrack xt_comment ipt_REJECT nf_reject_ipv4 nf_log_ipv4 nf_log_common xt_LOG xt_limit iptable_raw iptable_mangle iptable_filter sch_fq_codel amd64_edac_mod edac_mce_amd kvm_amd kvm igb irqbypass crct10dif_pclmul crc32_pclmul sp5100_tco ghash_clmulni_intel ptp pps_core i2c_piix4 pcspkr dca ccp i2c_algo_bit fam15h_power k10temp i2c_core leds_gpio gpio_keys pinctrl_amd pcc_cpufreq acpi_cpufreq lp parport_pc parport video sdhci_pci sdhci mmc_core
02:57:37 kernel: CPU: 2 PID: 8871 Comm: sshd Tainted: G L 4.14.154-ipfire #1
02:57:37 kernel: Hardware name: PC Engines apu3/apu3, BIOS v4.10.0.3 11/07/2019
02:57:37 kernel: task: ffff8fb168e2b700 task.stack: ffffa0f4009f4000
02:57:37 kernel: RIP: 0010:smp_call_function_many+0x1cc/0x230
02:57:37 kernel: RSP: 0018:ffffa0f4009f7c30 EFLAGS: 00000202 ORIG_RAX: ffffffffffffff10
02:57:37 kernel: RAX: 0000000000000000 RBX: ffff8fb16ed21fc0 RCX: ffff8fb16ec25480
02:57:37 kernel: RDX: 0000000000000001 RSI: 0000000000000000 RDI: ffff8fb16ed21fc8
02:57:37 kernel: RBP: ffff8fb16ed21fc8 R08: ffffffffffffffff R09: 000000000000000b
02:57:37 kernel: R10: ffffeb5e049bca80 R11: ffffa0f4009f7d48 R12: ffffffffa5468210
02:57:37 kernel: R13: 0000000000000000 R14: 0000000000000001 R15: 0000000000000040
02:57:37 kernel: FS: 00007289dfb3c740(0000) GS:ffff8fb16ed00000(0000) knlGS:0000000000000000
02:57:37 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
02:57:37 kernel: CR2: 00007289dfc79e10 CR3: 0000000126f32000 CR4: 00000000000406e0
02:57:37 kernel: Call Trace:
02:57:37 kernel: ? load_new_mm_cr3+0xc0/0xc0
02:57:37 kernel: on_each_cpu+0x28/0x60
02:57:37 kernel: flush_tlb_kernel_range+0x6b/0xb0
02:57:37 kernel: __purge_vmap_area_lazy+0x4d/0xb0
02:57:37 kernel: vm_unmap_aliases+0xde/0x110
02:57:37 kernel: change_page_attr_set_clr+0x2a3/0x470
02:57:37 kernel: set_memory_ro+0x26/0x30
02:57:37 kernel: bpf_prog_select_runtime+0x4e/0xe0
02:57:37 kernel: bpf_prepare_filter+0x527/0x5a0
02:57:37 kernel: bpf_prog_create_from_user+0xbb/0x110
02:57:37 kernel: ? hardlockup_detector_perf_enable.cold.6+0x1b/0x1b
02:57:37 kernel: do_seccomp+0x256/0x6d0
02:57:37 kernel: SyS_prctl+0x25d/0x540
02:57:37 kernel: do_syscall_64+0x67/0x100
02:57:37 kernel: entry_SYSCALL_64_after_hwframe+0x3d/0xa2
02:57:37 kernel: RIP: 0033:0x7289dfc79e1a
02:57:37 kernel: RSP: 002b:00007ffc712c19a8 EFLAGS: 00000246 ORIG_RAX: 000000000000009d
02:57:37 kernel: RAX: ffffffffffffffda RBX: 00007ffc712c19b0 RCX: 00007289dfc79e1a
08:50:03 kernel: igb 0000:03:00.0: eth2: PBA No: FFFFFF-0FF
08:50:03 kernel: igb 0000:03:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s)
08:50:03 kernel: Adding 1006496k swap on /dev/sda2. Priority:1 extents:1 across:1006496k SS
08:50:03 kernel: EXT4-fs (sda3): re-mounted. Opts: (null)
08:50:03 kernel: EXT4-fs (sda3): re-mounted. Opts: (null)
08:50:03 kernel: EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
08:50:03 kernel: EXT4-fs (sda4): mounted filesystem with ordered data mode. Opts: (null)
08:50:03 kernel: igb 0000:03:00.0 red0: renamed from eth2
08:50:03 kernel: igb 0000:01:00.0 green0: renamed from eth0
08:50:03 kernel: random: crng init done
08:50:03 kernel: random: 1 urandom warning(s) missed due to ratelimiting
08:50:08 kernel: igb 0000:01:00.0 green0: igb: green0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
08:50:09 kernel: igb 0000:03:00.0 red0: igb: red0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
09:09:41 kernel: pps pps1: new PPS source ptp1
09:09:41 kernel: igb 0000:02:00.0: added PHC on eth1
09:09:41 kernel: igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection
09:09:41 kernel: igb 0000:02:00.0: eth1: (PCIe:2.5Gb/s:Width x1) 00:0d:b9:49:da:29
09:09:41 kernel: igb 0000:02:00.0: eth1: PBA No: FFFFFF-0FF
09:09:41 kernel: igb 0000:02:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s)
09:09:41 kernel: pps pps2: new PPS source ptp2
09:09:41 kernel: igb 0000:03:00.0: added PHC on eth2
09:09:41 kernel: igb 0000:03:00.0: Intel(R) Gigabit Ethernet Network Connection
09:09:41 kernel: igb 0000:03:00.0: eth2: (PCIe:2.5Gb/s:Width x1) 00:0d:b9:49:da:2a
09:09:41 kernel: igb 0000:03:00.0: eth2: PBA No: FFFFFF-0FF
09:09:41 kernel: igb 0000:03:00.0: Using MSI-X interrupts. 2 rx queue(s), 2 tx queue(s)
09:09:41 kernel: Adding 1006496k swap on /dev/sda2. Priority:1 extents:1 across:1006496k SS
09:09:41 kernel: EXT4-fs (sda3): re-mounted. Opts: (null)
09:09:41 kernel: EXT4-fs (sda3): re-mounted. Opts: (null)
09:09:41 kernel: EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
09:09:41 kernel: EXT4-fs (sda4): mounted filesystem with ordered data mode. Opts: (null)
09:09:41 kernel: igb 0000:03:00.0 red0: renamed from eth2
09:09:41 kernel: igb 0000:01:00.0 green0: renamed from eth0
09:09:41 kernel: random: crng init done
09:09:41 kernel: random: 1 urandom warning(s) missed due to ratelimiting
09:09:46 kernel: igb 0000:01:00.0 green0: igb: green0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
09:09:46 kernel: igb 0000:03:00.0 red0: igb: red0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX
16:58:09 kernel: perf: interrupt took too long (2573 > 2500), lowering kernel.perf_event_max_sample_rate to 77700
20:40:00 kernel: perf: interrupt took too long (3254 > 3216), lowering kernel.perf_event_max_sample_rate to 61200
thanks for all help