CPU spike at 1:30am every morning

Installing & Running IPFire
1

There is a pretty large cpu spike about 1:30 every morning that lasts about 20 minutes or so. The network is quiet at this time. I am guessing it’s something running on a schedule since it is at the same time every night. I went through the system logs but could not find anything definitive. This is not an issue, just a curiosity. I want to know what is responsible. The cpu is a Celeron J3160. It is not super powerful so it does work hard during working hours when many users use it. This system is a “guest network” that 50+ employees connect their mobile devices or anything else we want to have limited access. Here is the system configuration:

Here is a screenshot of a day so you can see it up close:

cpuspike1
cpuspike1974×626 141 KB

And here is a screenshot of a week so you can see that it happens at the same time each morning (even on the weekend when no one is here):

cpuspike2
cpuspike2963×595 108 KB

Thanks in advance.

edit: Here is a bandwidth graph during the same time. You can see the network activity stops at 1:30am (when night shift ends) and is flat until a little after 6:00am when the next shift comes in.

bandwidthcpuspike
bandwidthcpuspike946×522 125 KB

2

First investigation in fcrontab shows, that at 1:25 the fcron.daily directory is run.
In my system this are ‘trim’ for SSDs, updatexlrator cleanup and urlfilter update.
Your system shows no traffic at this time, so no list updates are done.

What is the contents of your /etc/fcron.daily dir?

3

I would create a script

timeout 240s top -b -d 5 > top.log

which I would get fcron to run at say 1:28.

This runs top in batch mode for 4 minutes, so would cover the period 1:28 to 1:32 and would provide the top output every 5 seconds and pipe it to a log file.

For reference running the above command for 10 seconds with output every second (-d 1) gave a file of 129K so depending on your free disk space, you might want to send the output to a usb stick connected to IPFire with ExtraHD.

You can of course modify the command to change the frequency of checks and the length of time it runs for.

You could then look at the results and see what process(es) is/are using the cpu at 1:30

4

You could also make the command only use top and not require the timeout command

top -b -n 5 -d 1 > top.log

The -n 5 option tells top to run for 5 iterations with a delay of 1 second before stopping.

5

Contents of /etc/fcron.daily:

openvpn-crl-updater

scheduler_daily

trim

scheduler_daily is for @roberto report which I no longer use.

Roberto, I just tried to uninstall this, which your directions say:

  1. To uninstall:

./uninstall.sh

from /opt/pakfire/tmp. But when I go to this directory, it is empty. Any guidance on how to properly uninstall? And can I just delete scheduler_daily from the /etc/fcron.daily directory?

6

Hi,
this is the uninstall file. Copy it to the directory and run it.

#!/bin/bash

. /opt/pakfire/lib/functions.sh

#  delete License files.
/bin/rm --verbose --force /etc/fcron.hourly/scheduler_hourly
/bin/rm --verbose --force /etc/fcron.daily/scheduler_daily
/bin/rm --verbose --force /etc/fcron.weekly/scheduler_weekly
/bin/rm --verbose --force /etc/fcron.monthly/scheduler_monthly
/bin/rm --verbose --force /srv/web/ipfire/cgi-bin/reports.cgi
/bin/rm --verbose --force /var/ipfire/addon-lang/reports.es.pl
/bin/rm --verbose --force /var/ipfire/addon-lang/reports.en.pl
/bin/rm --verbose --force /var/ipfire/addon-lang/reports.de.pl
/bin/rm --verbose --force /var/ipfire/addon-lang/reports.it.pl
/bin/rm --verbose --force /var/ipfire/menu.d/EX-reports.menu
/bin/rm --verbose --force --recursive /var/ipfire/reports/

# Update Language cache
/usr/local/bin/update-lang-cache
7

The tmp directory is emptied periodically and definitely at a reboot.

You will need to untar the package to that tmp directory again and then you should have the uninstall script available.

Don’t forget to also remove the ROOTFILES file that you will have manually copied across in the install after running the uninstall script with the name of the package.

https://www.ipfire.org/docs/devel/ipfire-2-x/addon-howto#testing-the-install-uninstall-update-routines-and-add-on-itself

8

Okay, looking at what @tratru has provided, then there is no point in following the IPFire add-on instructions link I provided because the uninstall-sh does not follow the standard addon approach.

So ignore my post.

9

I just took the contents of the uninstall file

10

It looks like I could just manually run those commands too?

11

Hi,
as I said, I just copied and pasted the text from the file.
To run them manually, I think so.