Pcregrep on ipFire

Howzit all and all

Not sure where this would fit best, ended up in this backyard… feel free to move it.

I’m trying to get the IP Bulk Reporter from AbuseIPdb to work on a ipFire setup. It however requires pcregrep to work.
So far I have found a way to add the tool, nor pcre-tools which contains it.
Has anyone tried this before?

Or would I be better off trying to figure how to do the script section with something alternative?

pcregrep -o1 -o2 -o3 --om-separator="$unit_sep" -e '([a-zA-Z]+ [0-9]+ [0-9]+:[0-9]+:[0-9]+) .* (Invalid user [a-zA-Z0-9]+ from (([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})) port [0-9]+)' $secureLogFile > matches.txt

Never worked with it, so I am groping at things in the dark…
Thanks

Hi Andreas Otto and welcome here :slightly_smiling_face: .
As far as i can see is pcregrep in the ROOTFILE commented -->
https://git.ipfire.org/?p=ipfire-2.x.git;a=blob;f=config/rootfiles/common/pcre;h=6db5724fadf58bc04c83c3ad5de0b2d4132653d4;hb=refs/heads/core137#l2
and therefore not available in the core system.

Best,

Erik

If you send a patch, we can ship this with the next update…

Patch has been send to dev mailinglist --> https://patchwork.ipfire.org/patch/2598/ .

Best,

Erik

Tag Erik

Thanks for the help. I didn’t expect there would be much interest in getting this working, most appreciated.

Cheers
Andreas

Jo gerne,
should we mark this one as solved ?

Cheers,

Erik

Core 138 installed. I have not got it working yet, still shows
“Command ‘pcregrep’ required, but is not installed. Aborting.”

I assume the patch will only be included in the next update.
I’ll be patient :smile:

Yes your right. The update cycle for Core 138 was at that time closed and only for testing. As far as i can see is the patch until now not merged but i think it will soon.

If you do not want to wait the whole time until the next release, you can find in here --> https://people.ipfire.org/~ummeegge/pcregrep/ a 64bit binary of pcregrep which is located under /usr/bin .

Best,

Erik

Thanks most appreciated, all done. Everything is working.
Will add the the script to run every 24hrs, should be ample.

For those interested in running the abuseipdb, I placed the parse-logs.sh the site provides into /mnt/mo_scripts/, as I have a few scripts there my-sysstat, etc. You will need to chmod +x /mnt/mo_scripts/*.sh
To run the script, you will need to change the sites recommended log location.
/mnt/mo_scripts/parse-logs.sh /var/log/messages <your API Key>

There is a rate limit on requests, not for submitting. I would suggest to fcrontab -e and set the script to run around midnight your local time zone. There is no cutting/bleeding edge advantage to do it more often. Your submitted offending IP(s) will be added to a database pool. Submitting it more often from the same API does nothing to flag it as serious, only resets the date/time stamp on their side.
IPs only get flagged by having multiple sources report it, get others involved.

The site only mentions Fail2Ban using it’s db. Which means to me that rules.emergingthreats.net, lists.blocklist.de, abuse.ch and possibly iplists.firehol.org will have a simlar list going.
All this however only becomes interesting if you run IPSET with an update script.

Wish you all a wonderful day.
Cheers

Hi,
nice to hear that it works as expected :slightly_smiling_face: .

firehol is a nice platform which i use here also. Even the setup is not in there, but if someone is interested in an updater script, in here a FhBl one is findable --> https://gitlab.com/ummeegge/scripts/blob/master/ipset_FhBl_updater.sh or an older one for hardcoded BLs --> https://gitlab.com/ummeegge/scripts/blob/master/ipset_update.sh .

Beneath info, the last script currently do includes a for loop to add the new lists which can be made much faster if you have vast lists. The trick to speed this process significantly up is to use ‘ipset restore’ whereby the format from ‘ipset save’ can be used.

Did there a fast check via time and perl, sed and awk which looked like this:

+ perl -pe 'chomp; $_ = "add ipset_setname $_ -exist\n"' ip

real	0m0.071s
user	0m0.060s
sys	0m0.010s
+ sed -e 's/.*/add ipset_name & -exist/' ip

real	0m0.151s
user	0m0.129s
sys	0m0.012s
+ awk '{ print "add ipset_name "$0" -exist" }' ip

real	0m0.073s
user	0m0.059s
sys	0m0.014s
+ echo

++ wc -l ip
+ echo 'Number of IPs: 18985'
Number of IPs: 18985
+ exit 0

As little beneath one.

Best,

Erik

Ref.: https://lists.ipfire.org/pipermail/development/2019-September/006381.html