Monit config for checking log messages

Add-Ons
1

I was recently trying to troubleshoot a cabling issue with my red0 connection and found that setting up an interface monitor for a failed link didn’t actually help me.
Eg.

check network red0 with interface red0 
    if failed link then alert

This would only alert if there were download errors detected – which I’m still not actually sure what that means. However, IPFire was always logging a red0: carrier lost when I’d lose connection. So I wrote a bash script that will check the messages log for keywords and exit accordingly to alert monit for problems.

Here’s how I handle the monit config:

check program check_carrier_lost with path "/usr/local/bin/check_carrier_lost.sh"
    if status != 0 then alert

Here’s the wrapper for the generic monitor log that checks for the carrier lost:

#!/bin/bash
# Wrapper for Monit - hard-coded parameters to avoid quoting hell

/usr/local/bin/monitor_log.sh \
    "/var/log/messages" \
    "red0: carrier lost" \
    "/var/run/carrier_lost.state"

And here’s the actual monitor script:

#!/bin/bash

# =============================================
# Log Monitor Script
# Usage: ./monitor_log.sh <logfile> <search_pattern> [state_file]
#
# Example:
#   ./monitor_log.sh /var/log/messages "carrier lost" /var/run/carrier_lost.state
# =============================================

set -euo pipefail

LOGFILE="${1:-}"
PATTERN="${2:-}"
STATEFILE="${3:-/var/run/log_monitor.state}"

if [ -z "$LOGFILE" ] || [ -z "$PATTERN" ]; then
    echo "Usage: $0 <logfile> <search_pattern> [state_file]" >&2
    exit 2
fi

if [ ! -f "$LOGFILE" ]; then
    echo "Error: Log file not found: $LOGFILE" >&2
    exit 2
fi

# Find the most recent matching line
LAST_MATCH=$(grep -a -- "$PATTERN" "$LOGFILE" | tail -n 1)

if [ -z "$LAST_MATCH" ]; then
    exit 0
fi

# Extract timestamp (first 3 fields: Month Day Time)
# Example: Jun 21 17:59:54
TS_RAW=$(echo "$LAST_MATCH" | awk '{print $1 " " $2 " " $3}')

# Convert to Unix timestamp (epoch seconds). Uses current year if none specified.
CURRENT_EPOCH=$(date -d "$TS_RAW" +%s 2>/dev/null) || {
    echo "Error: Failed to parse timestamp: $TS_RAW" >&2
    exit 2
}

# Load previous timestamp (if any)
if [ -f "$STATEFILE" ]; then
    LAST_EPOCH=$(cat "$STATEFILE" 2>/dev/null || echo 0)
else
    LAST_EPOCH=0
fi

# Compare
if [ "$CURRENT_EPOCH" -gt "$LAST_EPOCH" ]; then
    # Newer match found → update state and alert
    echo "$CURRENT_EPOCH" > "$STATEFILE"
    echo "ALERT: NEW MATCH DETECTED" >&2
    echo "Pattern: $PATTERN" >&2
    echo "Line: $LAST_MATCH" >&2
    exit 1   # Non-zero = error / alert
else
    exit 0
fi
2

You may want to try a monit FILE CONTENT TEST:

FILE CONTENT TEST
The content statement can be used to incrementally test the content of a text file by using regular expressions.

It can search for the PATTERN you have like this:

#	review messages log
check file messageLog path /var/log/messages
	ignore content = "monit"

#	review messages log for carrier lost
	if content = "red0: carrier lost"
		then alert

It doesn’t do everything in your script but it does detect the needed PATTERN.

3

Does it search the whole file and continue to alert if it finds any instance of the pattern? Basically the log file would have to roll over for the alert to clear?

4

Any new instance would cause an alert. Not old instances.

5

Ok, I’ll try it out and see how it works.

6

I actually like your solution better than mine. It actually provides better total alert tracking with less code.