Unbound (and other programs ) logging

Generally spoken, if you have new ideas about logging ( not yet implemented since the start of this thread ), feel free to join the development community. How this can be done is documented in the wiki.

Just my 2cts:
Almost each core update I’ve to adopt /srv/web/ipfire/cgi-bin/logs.cgi/log.dat to get the logs entries, that NUT, precisely upsmon, writes into messages. Maybe worth to change the coding?

Additionally, I want to see logs that HAProxy and Deyhdrated create, unfortunately both do not write into /var/log/messages though. They use their own files, perhaps one time those packages get their own menu item below Log?

1 Like

Raise a bug for that, then the issue is recorded and can be worked on when resources are available.

Raise a separate bug for each add-on that is doing it differently.

1 Like

I certainly would like to do, if not Bugzilla has that ugly and unfortunately not user friendly interface and missing data to file a detailed and appropriate ticket. :sleepy:

As my post above, is not really a bug, but a feature request, I would like to file as such in Bugzilla. But, I’m completely lost while searching for the correct component as nut nor upsmon are both not listed there. So I’ve searched the list for words log and messages to no avail either.

So please advice on how to file such a request and on how to choose the correct component. :smiley:

1 Like

Components are only selected if you are entering a bug for IPFire 3.x

Select the entry at the top of the drop down list which is three dashes and the wording in the box indicates that is the entry to use for all IPFire2.x entries.

If you are unsure about an entry then just leave it at unknown or the default setting. We can modify it afterwards if it should be different.

For NUT I would classify that as a bug as the log data is put into /var/log/messages but there is no filter in the system logs.

HAProxy and Dehydrated might be more feature related than bug but the standard approach for IPFire is to use the system logs entry so I would also consider them as bugs as they are not following the standard approach for the logs.

Either way, just put the entries into bugzilla and we will manage to work with them. I will have a look through afterwards and see if anything needs to be tuned in the entries.

2 Likes

Just a further comment about working with bugs / feature requests.
To follow such things you need to be informed about every change. If you do this with the community board, you will be flooded with announcements. So the real information is lost .
Chosing bugzilla or the development mail list narrows this amount massively.
Bugzilla has the advantage, that the devs can easily search for unsolved tickets.

I don’t think the structure of bugzilla is too complicated. Classifying a request helps the requestor also to clarify his problem.
Because there is no automatism in working with bugzilla, each ticket is read and probably corrected as @bonnietwin stated. The ticket opener is informed about these modificatios.

1 Like

Done for the bug of package NUT.

And a second one as a feature request for HAProxy and Dehydrated.

Nevertheless, since HAProxy and Dehydrated use their own or no standard logging at all, I doubt to see the logs in the WebIf anytime soon :thinking:

1 Like

NUT should be no problem.

Looking at HAProxy then it looks like you can set it up to feed all output into a syslog system so it should be feasible to make work but it can also be changed in the conf file to go to quite different log files. Will need to try it out to see if it is possible but I am hopeful.

Having looked at Dehydrated I am less hopeful. It is not a running daemon but a script that is run when you need it, or it can be run periodically via cron. There is no logging setup at all. To get all the output going to a log system would require a modification of the script and the challenge would be to have the output going to a log file and at the same time to the screen because usually you will be interested in seeing the result when you run the script.
Not sure now that this will be feasible.

Will work on NUT first and then have a look at HAProxy and then based on how those went will decide about trying Dehydrated or not.

3 Likes

Maybe something like this for a script:

theScript | logger -t dehydrated 2>&1

I don’t know if tee would work with this…

That’s how you would do it manually when running the script but then if you hadn’t read the addon page and didn’t run that compound command but just the dehydrated command then nothing would be in the logs but you would still have a Dehydrated entry in the system logs drop down box which would show an empty output.

I was hoping to think of a way where someone has to actively change things somewhere from the default to prevent the data getting into the logfile.

The file redirection command you show could be put into every line that outputs stuff in the script but that would be a lot of update editing when the Dehydrated script is updated.
You could certainly add into the script the commands to feed the whole output of the script to a log file although I am not sure that the logger command then works in terms of adding the title for every entry.

I think it could need a lot of experimentation to confirm a solution that is built into the script so that when people use the dehydrated command it automatically feeds the output to a log file and the screen and puts a dehydrated title on each line in the logfile.

I am open to further ideas to try out when I get to working on the Dehydrated situation.

I knew it wouldn’t be easy since Dehydrated is not a service/daemon/process that delivers standard logs to messages but using it’s own format.

The same applies for HAproxy, too, as already mentioned in the bug/feature request.

At the end, I would like to see at least NUT logs in the WebIF :wink: I’ve done this already by modifying the appropriate source.

If the other packages will find their way into the WebIF some time, too, it would be great but this is certainly not highest priority of course.

Thanks!

Currently my script for Dehydrated looks like this

#!/usr/bin/env bash

cd /etc/dehydrated 

#Remove previous log file
mv -f renewal.log renewal.log.bak

#Renew certificates forcing renewal, clean up old certs moving to archive folder
date >> renewal.log
/usr/bin/dehydrated -gc >> renewal.log
/usr/bin/dehydrated -c -g -x >> renewal.log

#Merge certificates
/etc/dehydrated/merge_pem.sh

#Reload haproxy
/etc/init.d/haproxy reload >> renewal.log

##Write metrics
cd /root/metrics
python3 /root/metrics/system_certs.py

All output from Dehydrated goes into a text file, nothing fancy, but for debugging issues enough detail.

Question and suggestion.
HAproxy can log to a syslog server, can IPFire act as a syslog server and probably receive HAProxy logs, if configured?

Though I don’t know if a syslog server itself writes into /var/log/messages.

IPFire is an open source project. So everyone can easily find the function

#
# log (“message”) use default ‘ipcop’ tag
# log (“tag”,“message”) use your tag
#
sub log
{
my $tag=‘ipfire’;
$tag = shift if (@>1);
my $logmessage = $
[0];
$logmessage =~ /([\w\W]*)/;
$logmessage = $1;
&system(‘logger’, ‘-t’, $tag, $logmessage);
}

in file /var/ipfire/general-functions.pl.
Looking at this code, you can see that the software writes the systemlog using the Linux utility logger.
Man page about logger reads:

DESCRIPTION

logger makes entries in the system log. When the optional message argument is present, it is written to the log. If it is not present, and the -f option is not given either, then standard input is logged.

To answer your question:
yes, IPFire contains a local syslog server; if HAProxy is able to send to a local syslog server, the messages are written to /var/log/messages. These should be tagged with ‘HAProxy’. Then it is possible to filter them to an own section of the Logs part of the WebGUI.

EDIT: I’ve just scrolled a bit through this thread and followed the links to the ‘uncomfortable’ :wink: Bugzilla.
I think your questions are answered before and there. Adolf is working at this.

1 Like

I know Adolf is already taking some action on my tickets, at least for nut logging and filtering. HAProxy and Deyhdrated are IMO, more complicated regarding the possibilities of writing to the system log.

So, I just wanted to give Adolf some possible help for HAProxy logging, nothing else :slightly_smiling_face:

I am open to any help available. I will remember to come back here when I get to looking at HAProxy and Dehydrated.

Thanks for all the input. :+1:

1 Like

FYI, I’ve just updated the bugzilla ticket with some information on how to let HAproxy log into /var/log/messages.

Maybe it’s of any help for you and others: Add a WebIF option for logs originated from HAProxy

Now, I just need an entry for HAProxy in the WebIf to filter for those logs :wink:

3 Likes

Just did some tests again and re-checked /var/log/messages. I noticed that HAProxy now writes proper logs into the syslog file.

Don’t ask me how this happened, I do not know. I did not change anything on my side, not even restarted IPFire, HAproxy or anything else.

An entry in messages now looks like. You should notice that localhost.localdomain disappeared and now reads localhost.

Sep 19 19:30:55 localhost haproxy[13742]: 100.20.194.139:46494 http_https~ collab_server/collab01 207 1821 --VN “PROPFI
ND /dav.php/calendars/family/holiday/ HTTP/1.1”

Wait a minute: I’ve written here that the file /etc/hosts looks like:

127.0.0.1 localhost.localdomain localhost
172.17.0.1 gateway

I had a look again and now there is a totally different content!

127.0.0.1 localhost
192.168.0.1 ipfire.ourhome ipfire
172.17.0.1 gateway

So how on earth did this content found it’s way into this file?

Ah!!! A few days ago I’ve edited the hosts list in WebIf, inserted one single host. I assume that this action did re-write the file /etc/hosts, too. Presumably with the correct content! I do not know, so devs please check this yourself if necessary.

But why has there been some unusual content before?

1 Like

My hosts file looked like your original with just the two lines.

I then added my IPFire hostname to the Hosts page and my hosts file now looks like your second table but with my IPFire hostname and domain name shown.

I then deleted the IPFire host entry in the Hosts page and checked the hosts file and it has stayed with the three lines.
I still need to check what happens when I reboot.
Don’t fully understand what is happening.

2 Likes

Have confirmed that when you doi a reboot the /etc/hosts file goes back to the original default setting.

3 Likes