Recommended way to install filebeat

Hello,

What would be the recommended way to install and keep up to date filebeat for shipping the logs to logstash?

My guess is git pull and build, but keen to see if anyone else is doing something different.

You are test this?

Yah I have logstash already running for other stuff, now I am trying to get filebeat on ipfire -> logstash sending at a minimum Suricata logs.

Not having wget or apt means I am making my own git repo and putting stuff in there and then pulling that repo down to install the dependencies like golang etc.

I am still making this now, but interested to see what other people are doing because maybe there is easier way then I can think of.

If anyone comes across this and is interested, I have filebeat shipping suricata logs to logstash.

I am using a docker image to compile filebeat into an armv7 compiled binary. The specific docker command I use is:

docker run -v $(pwd):/build -e BEATS_VERSION=7.11 andig/beats4pi That’ll pull the image from dockerhub and run it.

Change 7.11 to whatever the version of filebeat is you want to build (7.11 is latest at this time), it spits out a binary named filebeat that you can then ship to ipfire and run.

Here is github repo containing the build script (build.sh) for people who have zero trust environments and want to make their own docker image to compile. Obviously if you are not running on arm things are different.

This is a significant improvement over using the inbuilt remote syslog sending in ipfire because you can enable TLS for encryption and authentication by logstash which I have done.

I don’t know what the criterion is for something becoming a pakfire? But I’d reckon filebeat would be a pretty good candidate for a pakfire.

I can and likely will use logstash to break up the message payload into fields, as I do with nginx logs.

Here is sample suricata event of mine that was sent to logstash->elastic from ipfire

Hi @anon45931963,

If you are building filebeat from source and not installing the binary then the approach for how to build an add-on is defined here:-

https://wiki.ipfire.org/devel/ipfire-2-x/build-howto
https://wiki.ipfire.org/devel/ipfire-2-x/addon-howto

Following this you can build the add-on and install it as a local add-on to confirm that everything is working.
Then to get it included in the IPFire add-on set you need to join the development mailing list and provide a patch for the add-on.

https://wiki.ipfire.org/devel/submit-patches

go is included in the buildbase of IPFire2.x

If you create an add-on and your patch is accepted then the expectation is that you will maintain that add-on as new versions of filebeat are released.

If you run into any problems in following this then the best place to ask is on the development mailing list.

1 Like

Hello @bonnietwin ,

Yes ok I will look at submitting filebeat as a pakfire then, I’m happy to maintain it as I’ll be using it myself and want it updated also.

go is included in the buildbase of IPFire2.x

How do I get it to work? Typing go version just gives me the response: -bash: go: command not found

Getting go to work for me would be needed for me to have the source compiled on device.

go is not on the firewall itself. When you create your build system (see the links) then go is available in the base build of that.

So go is only available while building and not in the firewall itself.

1 Like

Ah ok gotcha! But I am confused about one thing, if I build the pakfire in the base build, how can I ensure it is multi-architecture? For e.g. if my build system is amd64 and I build in there, it will not be armv7 compatible for example. This can’t be right so I must have some wrong thinking on how it works?

If the build process is the same for all architectures then when you build it on your system the build covers any cross compile requirements.

If the build needs to be different in some way for different architectures then you can specify those in some of the files. The kernel requirements fit into that category.
However I don’t know which files and that is something I have not had to do myself.

That would be the sort of question to ask on the development mailing list.

I must admit I have just built the add-ons I am installing or updating on my x86_64 system and submitted the patches and everything has been okay.

2 Likes

Ah I think I understand now. I’m submitting the build process to a pipeline that’s going to compile for all architectures. Yep I see how that works, yea should be no issue then as I believe the build process is the same across architectures.

Good luck in your build and submit process and welcome to the add-on development team.

1 Like

That is the basic process of cross-compiling.
And concerning go, this is a compiler for the go language. Compilers are contained in the build package, not the production package.

Yes this is where I went wrong, I thought by submitting the build process as the add-on, I was essentially submitting instructions for the ipfire device to build the pakfire at add time, which of course it cannot do withou the go compiler. But now I can see that it is instead cross-compiled when received by me and then distributed.

There is another problem with compiling on the fly. IPFire modules are signed. This signing is done at the system build process. If I remember right the signing key is thrown away after the build. Therefore you cannot add an additional module, you don’t know the key of the system.

Ah yea that makes sense.

I’m sure there is some kind of gpg based pki system in place, likely the certificate containing pubKey is in the production ipfire image, and wherever is doing the compiling and packaging is signing with the private key corresponding to that cert. Same thing as for iOS and iPhone apps.

For kernel module signing see Michaels blog article .

pakfires ain’t kernel modules tho, 100% userland I’d be reckoning.

Seems like devs would prefer to upgrade the syslog used in ipfire instead of using filebeat, which I’m happy if it does TLS encryption and authentication to logstash. So holding off making pakfire now as devs have signalled their intent to instead upgrade syslog see here Shipping logs to logstash - #14 by ms

Now he says they have no roadmap to do it… I do not understand this guy, he comes and says there is a better way to do it but then says probably they wont do it… well which is it? Shipping logs to logstash - #17 by ms

So it is likely I will not bother with a pakfire, I wanted to contribute to the project but seems not possible, I get that there is a better way to do it, but if you’re not going to do it then how is it better? My way is better than no way at all (or completely insecure as it is currently).

What to do? I don’t care I have it working for me, I just wanted to give back to the community with a pakfire for others who want to ship logs to a remote SOC, currently not possible with ipfire unless you want to send logs plain text which I imagine no one would want to do unless it was in their own network, or over a VPN as was suggested by someone else. I feel maintaining a VPN connection just for shipping logs is very overkill.

I would hate to see you not contribute this Plugin.
Ipfire is truly a very small team of developers.
It would be sad to see someone like your self leave this community.
Please do not feel that your contribution is not wanted.
I Do not think this is the case.