Netdata addon by ummeegge

sec-con · 26 February 2023 14:46

Note to self:
Netdata flagged an update so I should change my setup from Siosios installation method to the default with the cron script.

sec-con · 26 February 2023 15:04

Yeh, the updater throws an error when installling:

: ERROR: netdata-updater.sh:  Unable to determine what type of auto-update scheduling to use. Auto-updates have NOT been enabled.

My init.d is empty.

I really suck at nano, but I assume I am supposed to copy that code

#!/bin/sh
# SPDX-License-Identifier: GPL-3.0-or-later
#
# netdata	Real-time performance monitoring, done right
# chkconfig: 345 99 01
# description:  Netdata is a daemon that collects data in real-time (per second)
#               and presents a web site to view and analyze them. The presentation
#               is also real-time and full of interactive charts that precisely
#               render all collected values.
# processname:	netdata

# Source functions
. /etc/sysconfig/rc
. $rc_functions

DAEMON="netdata"
DAEMON_PATH=/opt/netdata/bin
PIDFILE_PATH=/opt/netdata/var/run/netdata
PIDFILE=$PIDFILE_PATH/$DAEMON.pid
DAEMONOPTS="-P $PIDFILE"
STOP_TIMEOUT="60"

[ -e /etc/sysconfig/$DAEMON ] && . /etc/sysconfig/$DAEMON

LOCKFILE=/var/lock/subsys/$DAEMON

service_start()
{
	[ -x $DAEMON_PATH ] || exit 5
	[ ! -d $PIDFILE_PATH ] && mkdir -p $PIDFILE_PATH
	chown netdata:netdata $PIDFILE_PATH
	echo -n "Starting $DAEMON..."
	loadproc $DAEMON_PATH/$DAEMON $DAEMONOPTS
	RETVAL=$?
	echo
	[ $RETVAL -eq 0 ] && touch $LOCKFILE
	return $RETVAL
}

service_stop()
{
	printf "%-50s" "Stopping $DAEMON..."
	killproc -p ${PIDFILE} $DAEMON
	RETVAL=$?
	echo
	[ $RETVAL -eq 0 ] && rm -f ${PIDFILE} ${LOCKFILE}
	return $RETVAL
}

condrestart()
{
	if ! service_status > /dev/null; then
		RETVAL=$1
		return $RETVAL
	fi

	/etc/rc.d/init.d/netdata stop
	/etc/rc.d/init.d/netdata start
}

service_status()
{
	statusproc $DAEMON
}

service_status_quiet()
{
	statusproc $DAEMON >/dev/null 2>&1
}

case "$1" in
start)
	
	service_start
;;
stop)
	service_status_quiet || exit 0
	service_stop
;;
restart)
	service_stop
	service_start
;;
try-restart)
	condrestart 0
	;;
force-reload)
	condrestart 7
;;
status)
	service_status
;;
*)
	echo "Usage: $0 {start|stop|restart|try-restart|force-reload|status}"
	exit 3
esac

to an init.d file located in the /etc/rc.d/ directory?

Netdata works btw, using kickstarter.sh script.

sec-con · 1 March 2023 10:36

The thing about the code in init.d is actually a question.

Where do I put that code?

tofficap · 2 March 2023 05:19

For updating i use the following in my cron:

# Netdata täglich um 1:00 Uhr aktualisieren
0 1 * * *		nice wget -O /tmp/netdata-kickstart.sh https://my-netdata.io/kickstart.sh && sh /tmp/netdata-kickstart.sh --no-updates --stable-channel

sec-con · 2 March 2023 08:45

Sorry, don’t know where to put that. Cron is a mystery to me.

cfusco · 2 March 2023 10:20

to be precise, it’s fcron, a different implementation of the cron subsystem.
As in any cron-liike system, the syntax is divided in two parts, all those asterisks determine when the command will run, and next to it there is the command to be executed using the shell language, which for IPFire is Bash.

The specific line suggested to you says to run a nice command every one hour and 0 minutes.

Nice is a command to start a script specifying the priority for the kernel scheduler, here it is left to the default (which should be 0).

Nice calls wget, which will connect to https://my-netdata.io/kickstart.sh to download the script written over there and redirect it locally to a new file in the tmp directory with the name netdata-kickstart.sh (-O /tmp/netdata-kickstart.sh).

Next with the AND operator && having finished SUCCESSFULLY that tasks (if not, it will exit the shell with a non 0 errorlevel which means “not successful”) it executes the next command. It starts a subshell calling that script with the options --no-updates --stable-channel.

Now, to the question where to put the script in fcron. If as root you call the command fcrontab -e, it will open your default editor (likely nano) and show all the fcron table, where you can add the command. However, you should not do that for security reasons and instead create a specific user with root privileges to do that. You can find a detailed tutorial on how to do that in the wiki page dedicated to fcron.

siosios · 4 March 2023 18:21

I’ve reverted back to the package that works with ipfire. It doesn’t take long for me to update it and it works like I want it to, everyone has their own opinions and ways of doing things so I’ll leave this thread at that. If anyone cares to use the package I update, it will be on my git hub and updated as I usually do.

jiab77 · 6 March 2023 00:57

@siosios Sorry to be late in the party but I’ve read the whole thread and wonder what is your opinion between your method and the kickstart script provided by netdata?

To be honest, I’m not against keeping your method but maybe the install / update methods needs to be made simpler and the service being added to the service list as all other add-ons and your method is perfect to me.

I’ll try the kickstart method on my test VM and see from my own side which one is the most simpler and includes everything for being integrated in IPFire and let you know here.

jiab77 · 6 March 2023 01:01

@siosios, just saying, you should avoid using space and special chars for everything related to the web and if possible rename your folders with _ or - instead of space. It would make things more compatible for every OS’s.

[Edit]

Just to let you know that I’m preparing something for this project so stay tuned

[Edit 2]

First working version of the install script released! Please check the README before running it!

WARNING

Don’t run the following commands at the moment!

./install.sh -s add

./install.sh -s rm

These commands might break your IPFire installation, so please, really, don’t run them!

(Both exists in the script but they are not ready to use for the moment)

Now that the warning is given, here is how to use it

Install the add-on
Remove the add-on
Update the add-on (not tested completely yet)

Please let me know if you have any issues!

[Edit 3]

If you have installed the Netdata add-on within my script and the package built by @siosios, you can at least run this service related command but not the others!

./install.sh -s test

It will test and report if the Netdata service is running or not.

ummeegge · 6 March 2023 15:51

You guys are just awesome

siosios · 6 March 2023 19:44

wget -O /tmp/netdata-kickstart.sh https://my-netdata.io/kickstart.sh && sh /tmp/netdata-kickstart.sh --static-only --disable-telemetry --disable-cloud

this actually works to install and update but becuase ipfire does not use systemd the kickstart script can’t complete the install 100%. im going to try and rewrite the update script and change it to use systemv ( I think thats whats in use on ipfire https://qr.ae/prcEQF ) and see if I can update the functions file in init.d to add a little functionality

I still prefer the pakfire package but I’m willing to see if i can get the update script to work along with
the init.d/netdata script.

bonnietwin · 6 March 2023 20:28

That’s right, IPFire2.x is using SysV init scripts. IPFire3.x will be using Systemd.

jiab77 · 6 March 2023 21:46

Hahahaha thanks @ummeegge

Same here but it would be nice if we can handle both methods, pakfire package and the kickstart script. Please let me know if I can help you to write / edit the script.

Awesome, thanks a lot for your precious inputs @bonnietwin!

siosios · 6 March 2023 22:05

Thank you, I do appreciate positive input!

sec-con · 7 March 2023 06:48

Currently still on the kickstarter.sh but if this work provides pakfire plugin you would have my thanks.

I don’t know about you guys, but for me it is very important to keep an app updated. Not that I go all bananas over every single micro version, but in general the update functionality is recommended, either via Netdata itself or via Pakfire.

I still have not been able to look in to the cron commands to enable updating after kickstarter.sh, but no rush if something else is coming.

BTW is anyone using the Cloud functionality to collect Nodes? IPFire machine is often only one node.

jiab77 · 7 March 2023 18:12

I still need to run the tests to be able to handle the kickstart way to install Netdata with my script but I’m agree with you and is the reason why I’ve started to implement that update feature in the script.

As long as @siosios is providing updated packages regularly, the update feature in the script should do the job but again, the best would be to have it as an officially supported Pakfire add-on so that the Netdata add-on can be updated in the same way as all other add-ons.

I don’t remember what was the blocking point to have it as an official add-on but if I can help on that, please let me know.

For the moment, the script I’ve made will detect the current running IPFire version and patch and select the Netdata package version hardcoded in the script so as long as I’m specifying the latest version provided by @siosios, the script will update the existing Netdata add-on to the latest version as expected.

jiab77 · 7 March 2023 20:21

@siosios, I’ve just found and fixed a quite annoying issue regarding the Netdata plugins. Most of them could not run and almost nothing was printed except in the /opt/netdata/var/log/netdata/collector.log where you can see that most of the plugins gets “permission denied” when they are launched.

At least it was the case on my side. You can check on your side by running this command:

# cat /opt/netdata/var/log/netdata/collector.log | grep -i "permission denied"

If that’s the case on your side too, go to your install folder and run this command:

# ls -halF /opt/netdata/usr/libexec/netdata/plugins.d/

And get all the *.plugin like that or similar:

-rwxr-x--- 1 root 999 1.6M Feb 16 02:50 ebpf.plugin*
-rwxr-x--- 1 root 999  60M Feb 16 02:50 go.d.plugin*

Instead of:

-rwxr-x--- 1 root netdata 1.6M Feb 16 02:50 ebpf.plugin*
-rwxr-x--- 1 root netdata  60M Feb 16 02:50 go.d.plugin*

Then you have the same issue as I did until I could find a way to fix it.

Here is how you can fix it:

Search for the files that don’t have the right ownership:

# find /opt/netdata -group 999 -ls 2>/dev/null

Change 999 by the group id you have on your side.

You should see that most of the *.plugin files don’t have the right ownership (at least it was the case on my side)

Give the right ownership to the impacted files:

# find /opt/netdata -group 999 -exec chown -c root:netdata {} \;

Change 999 by the group id you have on your side.

Restart Netdata

# /etc/init.d/netdata restart

Reload Netdata dashboard

Once done, you should much more displayed data as shown here:

If you don’t have that on your side then you might have that permission issue too.

PS: You might not have the Unbound local and Sensors items on your side as I did some stuff to enable them.

[Edit 1]

Another way to see if you have that permission issue is with the command ps fax that way:

Permission issue not fixed:

# ps fax | grep netdata
12776 pts/0    S+     0:00          \_ grep netdata
22546 ?        S      0:00 /bin/sh /etc/init.d/netdata start
22548 ?        Sl    14:14  \_ /opt/netdata/usr/sbin/netdata -D -c /opt/netdata/etc/netdata/netdata.conf -i 192.168.210.3 -p 19999 -P /opt/netdata/var/run/netdata.pid
22550 ?        Sl     0:00      \_ /opt/netdata/usr/sbin/netdata --special-spawn-server
22780 ?        Sl    11:55      \_ /opt/netdata/usr/libexec/netdata/plugins.d/apps.plugin 1
 9872 ?        S      0:02      \_ bash /opt/netdata/usr/libexec/netdata/plugins.d/tc-qos-helper.sh 1

Permission issue fixed:

# ps fax | grep netdata
 3255 pts/5    S+     0:00          \_ grep netdata
 4375 pts/5    S      0:00 /bin/sh /etc/init.d/netdata start
 4377 pts/5    Sl     7:04  \_ /opt/netdata/usr/sbin/netdata -D -c /opt/netdata/etc/netdata/netdata.conf -i 192.168.100.1 -p 19999 -P /opt/netdata/var/run/netdata.pid
 4379 pts/5    Sl     0:00      \_ /opt/netdata/usr/sbin/netdata --special-spawn-server
 4612 pts/5    Sl     1:53      \_ /opt/netdata/usr/libexec/netdata/plugins.d/apps.plugin 1
 4615 pts/5    Sl     0:17      \_ /usr/bin/python3 /opt/netdata/usr/libexec/netdata/plugins.d/python.d.plugin 1
 4639 pts/5    S      0:26      \_ bash /opt/netdata/usr/libexec/netdata/plugins.d/charts.d.plugin 1
 4642 pts/5    Sl     0:26      \_ /opt/netdata/usr/libexec/netdata/plugins.d/go.d.plugin 1
31822 pts/5    S      0:02      \_ bash /opt/netdata/usr/libexec/netdata/plugins.d/tc-qos-helper.sh 1

From what I could see on my test VM, it’s apparently a bug in the package made by @siosios so I’ll implement a quick fix in my install.sh script so you will just have to run ./install.sh -f or ./install.sh --fix-perms to solve the issue until it gets fixed in the package itself.

In the meantime, the quickest fix is the following:

# Fix permissions
find /opt/netdata -group 999 -exec chown -c root:netdata {} \;

# Restart the service
/etc/init.d/netdata restart

I’ll rebuild my test VM and confirm here if the bug is coming from the IPFire package made by @siosios or not and let you know.

siosios · 8 March 2023 01:49

if you look at a netdata install on a centos/alma box you’ll find that the dirs/files are netdata:netdata, others are root:netdata or vise versa, and finally some are root:root. the chowns done in the install of the package were never changed from the original package files that i started with.

alma netdata:

  File: /etc/netdata
  Type: directory
  Size: 38.65 KiB    Blocks: 8          IO Block: 4096   directory
Device: fd00h/64768d Inode: 1075387877  Links: 10
 Attrs: --------------------
Access: (0755/drwxr-xr-x)  Uid: (0/root)   Gid: (0/root)

everything in the directory is root:root

  File: /usr/libexec/netdata
  Type: directory
  Size: 88.34 MiB    Blocks: 0          IO Block: 4096   directory
Device: fd00h/64768d Inode: 2151198070  Links: 5
 Attrs: --------------------
Access: (0755/drwxr-xr-x)  Uid: (0/root)   Gid: (0/root)

  File: /usr/libexec/netdata/plugins.d
  Type: directory
  Size: 86.9 MiB     Blocks: 8          IO Block: 4096   directory
Device: fd00h/64768d Inode: 3222555922  Links: 3
 Attrs: --------------------
Access: (0755/drwxr-xr-x)  Uid: (0/root)   Gid: (0/root)

File: /usr/lib/netdata
  Type: directory
  Size: 758.33 KiB   Blocks: 0          IO Block: 4096   directory
Device: fd00h/64768d Inode: 5870045     Links: 4
 Attrs: --------------------
Access: (0775/drwxrwxr-x)  Uid: (0/root)   Gid: (0/root)

  File: /usr/share/netdata
  Type: directory
  Size: 44.82 MiB    Blocks: 0          IO Block: 4096   directory
Device: fd00h/64768d Inode: 1075387641  Links: 3
 Attrs: --------------------
Access: (0775/drwxrwxr-x)  Uid: (0/root)   Gid: (0/root)

  File: /var/lib/netdata
  Type: directory
  Size: 45.6 KiB     Blocks: 0          IO Block: 4096   directory
Device: fd02h/64770d Inode: 1074952368  Links: 6
 Attrs: --------------------
Access: (0775/drwxrwxr-x)  Uid: (969/netdata)   Gid: (968/netdata)

 File: /var/log/netdata
  Type: directory
  Size: 291.21 MiB   Blocks: 0          IO Block: 4096   directory
Device: fd02h/64770d Inode: 178689      Links: 2
 Attrs: --------------------
Access: (0755/drwxr-xr-x)  Uid: (969/netdata)   Gid: (0/root)

but the log files are

  File: /var/log/netdata/collector.log
  Type: UTF-8 Unicode text, with very long lines
  Size: 330.13 KiB   Blocks: 664        IO Block: 4096   regular file
Device: fd02h/64770d Inode: 180497      Links: 1
 Attrs: --------------------
Access: (0644/-rw-r--r--)  Uid: (969/netdata)   Gid: (968/netdata)

the chowns in the install script:

# Set permissions
chown -R root:netdata /opt/netdata/var/lib/netdata
chown -R netdata:netdata /opt/netdata/var/cache/netdata/
chown -R root:netdata /opt/netdata/usr/lib/netdata/
chown -R netdata:netdata /opt/netdata/usr/share/netdata
chown -R netdata:netdata /opt/netdata/var/log/netdata
chown -R root:netdata /opt/netdata/etc/netdata
chown root:root /opt/netdata/etc/netdata/netdata.conf

# Enable application groups
chown root:netdata /opt/netdata/usr/libexec/netdata/plugins.d/apps.plugin
chmod 4750 /opt/netdata/usr/libexec/netdata/plugins.d/apps.plugin

which ill go file by file from an original kickstart.sh install on a regular linux box to correct them

jiab77 · 8 March 2023 03:05

@siosios I was not saying that you did something wrong, just that I’ve found an issue regarding the permissions that are not correct on some files (321 files to be precise) after or during the install process that are currently blocking Netdata from working properly.

I’ve documented the whole debugging process I’ve followed so that you can see by yourself the result when it’s patched with my updated install script.

I did the tests on my dev and production deployment of IPFire and it have resolved all the issues I had on my production deployment. I could then revert the changes I’ve made on the unbound config.

So please don’t take it as if I’m criticizing your work, it’s absolutely not my goal and on contrary, I really like and enjoy all the work you’ve done on this package!

I just know Netdata very well as I’ve contributed to this project in the past and keep using it within the metrics streaming feature configured on many hosts.

Now and with your great work on Netdata, I can see how Unbound is performing on my IPFire deployment!

For those who want to see the temperature, voltage and so on of their IPFire deployment displayed on Netdata, just check the sensors page and how to enable them.

Basically, you just have to run the following commands:

# cd /opt/netdata/etc/netdata
# ./edit-config charts.d.conf

and un-comment the line # sensors=force at the end of the file. You should get this result:

# example=force
sensors=force

Then restart Netdata with:

# /etc/init.d/netdata restart

Once done, just reload the dashboard and you should see this new graph:

And this or similar when clicked:

That’s all from me for today Enjoy guys!

PS: One last thing, for those who want to reset their Netdata installation with my script, just run this command: ./install.sh -R. It will backup your existing config files, remove, reinstall, fix the permissions and restore your config files so you should not lose any changes by running the reset command.

sec-con · 8 March 2023 08:34

I was pondering, and yes I may of course be over simplifying this due to ignorance, but… well, here goes…

We have a custom script using wget that is a way to install Netdata on IPFire.

We have the kickstarter.sh that - a tad unexpectedly - is also a way to install Netdata on IPFire .

The main issue with both methods above is the lack of an UI and accessibility to general users in IPFire. And of course a way to highlight and enable some relevant options.

We have Pakfire which is a way to customize addons for IPFire and adding their installation to the interface. wiki.ipfire.org - Pakfire
From what I can read you could configure Pakfire to use any of above methods but I do not find what more it takes to actually build a Pakfire Package that will show up in the Pakfire list of Addons.

So would the latter not be a relevant target to aim for, once you sorted out other obstacles you stumbled over?

After some searching I found this: wiki.ipfire.org - IPFire 2.x - Building Addons

@bonnietwin Maybe that article could be linked in to the Main Addon / Pakfire articles on the Wiki?