Sendmail Segfault from Swatch

I am using swatch to monitor /var/log/suricata/fast.log. When swatch detects that an entry has been written to fast.log, an email message is sent via sendmail.
This all works fine when I have started swatch from an ssh session. It also works if swatch is started via fcron.

However when swatch is started at boot out of /etc/sysconfig/rc.local, sendmail fails with a segfault.

Here’s the swatch startup …

/usr/bin/swatch --daemon --config-file=/etc/swatch.conf --tail-file=/var/log/suricata/fast.log --pid-file=/var/run/swatch.pid

Here’s the segfault in /var/log/messages

[root@ipfire ~]# grep sendmail /var/log/messages
Nov 28 05:45:34 ipfire kernel: sendmail[15365]: segfault at c08d1af8 ip 000076cfc07d2776 sp 00007ffc274a5148 error 4 in libc-2.29.so[76cfc074a000+170000]
Nov 28 07:05:17 ipfire kernel: sendmail[3434]: segfault at b3559af8 ip 00007026b345a776 sp 00007ffd3d461b98 error 4 in libc-2.29.so[7026b33d2000+170000]
Nov 29 16:00:58 ipfire kernel: sendmail[3219]: segfault at a61e9af8 ip 00007f1ca60ea776 sp 00007fff2befdcd8 error 4 in libc-2.29.so[7f1ca6062000+170000]

Would any of you have a clue as to what the heck is going on here?

[edit] I also tried having swatch started via /etc/rc.d/rc3.d/ – with the same result, segfault upon sending mail via sendmail.

img

Hey,

When did you install this system and for how long have you been updating this?

We will need to check if upstream has released a new version of swatch where this bug could have been fixed already.

I did a fresh install a few revs back – don’t recall which one.
Currently updated to IPFire 2.23 (x86_64) - core138
Swatch version is 3.2.3

It’s weird that I can start swatch as root via ssh or via fcron and all works just dandy, but when swatch is started at boot via rc.local or init.d, the sendmail gets a segfault. The sendmail segfault doesn’t seem to impact swatch as it continues running and continues invoking sendmail as new log monitor events trigger.

Thanks for looking into this,
Charles

Oh you are right. I missed that. It is sendmail.

Are you using Postfix or DMA?

I don’t recall making a choice between Postfix or DMA
I configured the mail service via the WUI on the System-> Mail Service page.

[edit 1] Doing a little grepping, it would appear to be using dma
[edit 2] Just did another fresh install on another box and getting the exact same behavior.
img

Okay, can you find out what command swatch executes including arguments? I guess that is how we can find out why dma doesn’t like it.

When I start swatch from ssh session (a works okay scenario), a .swatch_script.<some pid> gets generated in /root Here’s the contents from the last time I start swatch:

#
#    swatch: The Simple WATCHdog
#    Copyright (C) 1993-2006 E. Todd Atkins
#
#    This program is free software; you can redistribute it and/or modify
#    it under the terms of the GNU General Public License as published by
#    the Free Software Foundation; either version 2 of the License, or
#    any later version.
#
#    This program is distributed in the hope that it will be useful,
#    but WITHOUT ANY WARRANTY; without even the implied warranty of
#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#    GNU General Public License for more details.
#
#    You should have received a copy of the GNU General Public License
#    along with this program; if not, write to the Free Software
#    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
#

use strict;
use FileHandle;
use POSIX ":sys_wait_h";

## User supplied modules and their locations
use lib qw();


use Swatch::Actions;
use Swatch::Throttle;
use Swatch::Threshold;

use vars qw/
  %Msg_Rec
  $Fh
/;

$SIG{'TERM'} = $SIG{'HUP'} = 'goodbye';
$SIG{'CHLD'} = 'DEFAULT';

## Constants
(my $Me = $0) =~ s%.*/%%;
my $BELL   = " ";
$/ = "
";
my $swatch_flush_interval = 300;
my $swatch_last_flush = time;
my $tail_pid = -1;

use IO::Handle;
STDOUT->autoflush(1);

sub goodbye {
  $| = 0;
  if( $tail_pid != -1 ) {
    kill('TERM', $tail_pid);
  }

  &Swatch::Actions::close_pipe_if_open();

  exit(0);
}

#
# write_pid_file(file_name) - writes a one line file that contains
# the current process id.
#
sub write_pid_file {
  my $name = shift;
  my $fh = new FileHandle "$name", "w";
 
  if (defined($fh)) {
    print $fh "$$\n";
    $fh->close;
  } else {
    warn "$Me: cannot write pid file named $name: $!\n";
  }
}
  
my $pid = fork;
exit if $pid;
die "Couldn't fork: $!" unless defined($pid);
# dissociate from the controlling terminal
POSIX::setsid() or die "Can't start new session: $!"; 
# set our named to 'swatch' so that rc scripts can 
# figure out who we are.
$0="/usr/bin/swatch --daemon --config-file=/etc/swatch.conf --tail-file=/var/log/suricata/fast.log --pid-file=/var/run/swatch.pid";
	       write_pid_file("/var/run/swatch.pid"); 

my $filename = '/var/log/suricata/fast.log';
$tail_pid = open(TAIL, "/usr/bin/tail -n 0 -F $filename|");
if (not $tail_pid) {
    die "/usr/bin/swatch: cannot read run \"/usr/bin/tail -n 0 -F $filename\": $!\n";
}

LOOP: while (<TAIL>) {

    chomp;
    my $S_ = $_;
    @_ = split;
    
    ### quote all special shell chars ###
    $S_ =~ s/([;&\(\)\|\^><\$`'\\])/\\$1/g;
    my @S_ = split(/\s+/, $S_);

    if (/\[\*\*\]/) {
	&Swatch::Actions::send_email('SUBJECT' => "Firewall Rule Fired", 'MESSAGE' => "$_", 'ADDRESSES' => "cab_77573\@yahoo.com", );
      next;
    }

}
## TODO: Add close !!!

I’m not sure where to look for the same sort of .swatch_script. file when swatch gets started from rc.local

Changed back to having swatch started in rc.local and found .swatch_script.3050 in the / directory.
Here’s the contents

#
#    swatch: The Simple WATCHdog
#    Copyright (C) 1993-2006 E. Todd Atkins
#
#    This program is free software; you can redistribute it and/or modify
#    it under the terms of the GNU General Public License as published by
#    the Free Software Foundation; either version 2 of the License, or
#    any later version.
#
#    This program is distributed in the hope that it will be useful,
#    but WITHOUT ANY WARRANTY; without even the implied warranty of
#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#    GNU General Public License for more details.
#
#    You should have received a copy of the GNU General Public License
#    along with this program; if not, write to the Free Software
#    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
#

use strict;
use FileHandle;
use POSIX ":sys_wait_h";

## User supplied modules and their locations
use lib qw();


use Swatch::Actions;
use Swatch::Throttle;
use Swatch::Threshold;

use vars qw/
  %Msg_Rec
  $Fh
/;

$SIG{'TERM'} = $SIG{'HUP'} = 'goodbye';
$SIG{'CHLD'} = 'DEFAULT';

## Constants
(my $Me = $0) =~ s%.*/%%;
my $BELL   = " ";
$/ = "
";
my $swatch_flush_interval = 300;
my $swatch_last_flush = time;
my $tail_pid = -1;

use IO::Handle;
STDOUT->autoflush(1);

sub goodbye {
  $| = 0;
  if( $tail_pid != -1 ) {
    kill('TERM', $tail_pid);
  }

  &Swatch::Actions::close_pipe_if_open();

  exit(0);
}

#
# write_pid_file(file_name) - writes a one line file that contains
# the current process id.
#
sub write_pid_file {
  my $name = shift;
  my $fh = new FileHandle "$name", "w";
 
  if (defined($fh)) {
    print $fh "$$\n";
    $fh->close;
  } else {
    warn "$Me: cannot write pid file named $name: $!\n";
  }
}
  
my $pid = fork;
exit if $pid;
die "Couldn't fork: $!" unless defined($pid);
# dissociate from the controlling terminal
POSIX::setsid() or die "Can't start new session: $!"; 
# set our named to 'swatch' so that rc scripts can 
# figure out who we are.
$0="/usr/bin/swatch --daemon --config-file=/etc/swatch.conf --tail-file=/var/log/suricata/fast.log --pid-file=/var/run/swatch.pid";
	       write_pid_file("/var/run/swatch.pid"); 

my $filename = '/var/log/suricata/fast.log';
$tail_pid = open(TAIL, "/usr/bin/tail -n 0 -F $filename|");
if (not $tail_pid) {
    die "/usr/bin/swatch: cannot read run \"/usr/bin/tail -n 0 -F $filename\": $!\n";
}

LOOP: while (<TAIL>) {

    chomp;
    my $S_ = $_;
    @_ = split;
    
    ### quote all special shell chars ###
    $S_ =~ s/([;&\(\)\|\^><\$`'\\])/\\$1/g;
    my @S_ = split(/\s+/, $S_);

    if (/\[\*\*\]/) {
	&Swatch::Actions::send_email('SUBJECT' => "Firewall Rule Fired", 'MESSAGE' => "$_", 'ADDRESSES' => "cab_77573\@yahoo.com", );
      next;
    }

}
## TODO: Add close !!!

Surely the issue must be with some environment setting difference(s) between running at boot out of rc.local vs running from an ssh root login or running from roots fcrontab

Not really Solved but working okay by launching swatch out of root’s fcron with @reboot

@reboot /etc/rc.d/init.d/swatch start

Okay, I found a cause – it was my permissions on the the /var/ipfire/dma/auth.conf
I still don’t understand why daemons running as root, started from rc.local or init.d script behaved differently than when started by root from an ssh or console shell. This seems to have something to do with the permissions and/or ownership on /var/ipfire/mail config files and perhaps the setuid settings on sendmail.dma

Here’s the ownership and permissions for dma mail transfer agent:

[root@ipfire dma]# ls -l /usr/sbin/*dma
-rwxr-sr-x 1 root mail 68848 Apr 9 2019 /usr/sbin/dma
lrwxrwxrwx 1 root root 3 Apr 9 2019 /usr/sbin/sendmail.dma -> dma

… and here’s the working ownership and permissions on my mail config files:

[root@ipfire dma]# ls -l /var/ipfire/dma
-rw-r----- 1 root mail 43 Sep 8 10:37 auth.conf
-rw-r–r-- 1 nobody nobody 189 Dec 2 19:42 dma.conf
-rw-r–r-- 1 nobody nobody 72 Sep 8 10:37 mail.conf

Specifically, one may note the root:mail ownership of auth.conf