Pmacct: Memory WARN messages

Continuing the discussion from Pmacct - "Unknown key: interface" , "no such column: vlan_in":


@tphz, @ummeegge, and everyone else!

Has anyone run into these memory WARN messages related to pmacct?

I see LOTS of these messages from the memory plugin:

Jun 28 13:58:42 ipfire pmacctd[4228]: WARN ( plugin1/memory ): Failed during write: Resource temporarily unavailable 
. . .
Jun 28 13:58:53 ipfire pmacctd[4232]: WARN ( plugin1/memory ): Missing data detected (plugin_buffer_size=102400 plugin_pipe_size=102400000). 
Jun 28 13:58:53 ipfire pmacctd[4232]: WARN ( plugin1/memory ): Increase values or look for plugin_buffer_size, plugin_pipe_size in CONFIG-KEYS document.  

.

If I enable debug in the config file then I see these messages related to the config file:

Jun 28 14:02:09 ipfire pmacctd[19511]: INFO ( plugin1/memory ): plugin_pipe_size=10240000 bytes plugin_buffer_size=10240 bytes
Jun 28 14:02:09 ipfire pmacctd[19511]: INFO ( plugin1/memory ): ctrl channel: obtained=212992 bytes target=8000 bytes 

.
My current config is this:

!----------------------------  memory  ------------------------------

!
! "plugin1" plugin configuration
!
!plugins: memory[plugin1]

plugin_buffer_size[plugin1]: 10240
plugin_pipe_size[plugin1]:   10240000

imt_path[plugin1]: /var/spool/pmacct/plugin1.pipe

aggregate[plugin1]: src_host, src_mac, dst_host, dst_mac
aggregate_filter[plugin1]: ip

I did find this:

But it did not help me… And I’ve tried many “guesses” for plugin_pipe_size and plugin_buffer_size.

Were you able to fix this?

Hello John,
have no warnings in debug mode like this therefor have not increased the buffer/pipe size. What configuration (in what) environment are you running Pmacct ?

Best,

Erik

The IPFire environment.


For what it is worth…

It looks like there was another pmacct change with the new pmacct 1.7.8 version. After CU 175 these WARNs started:

for logf in $(ls -1tr /var/log/messages*) ; do zgrep --color -aie "CORE UPGR: Upgrading from release.*" -e "Failed during write:" $logf ; done
. . .
Dec 30 12:16:18 ipfire pakfire: CORE UPGR: Upgrading from release 171 to 172
Feb 28 12:01:02 ipfire pakfire: CORE UPGR: Upgrading from release 172 to 173
Apr 19 21:31:21 ipfire pakfire: CORE UPGR: Upgrading from release 173 to 174
Jun 13 22:21:09 ipfire pakfire: CORE UPGR: Upgrading from release 174 to 175
Jun 25 13:31:20 ipfire pmacctd[21940]: WARN ( plugin1/memory ): Failed during write: Connection refused 
Jun 25 15:06:17 ipfire pmacctd[32658]: WARN ( plugin1/memory ): Failed during write: Resource temporarily unavailable 
Jun 25 15:06:18 ipfire pmacctd[32658]: WARN ( plugin1/memory ): Failed during write: Resource temporarily unavailable 
Jun 25 15:06:18 ipfire pmacctd[32658]: WARN ( plugin1/memory ): Failed during write: Resource temporarily unavailable 

But it took a few days to find it…

Erik - It is nice to see you around!

One thing to point out in my post directly above. It took about 12 days after my CU 175 update for the WARN messages to appear. I rebooted yesterday and only one WARN message has appeared so far. So this seems to be an issue that takes a few days to make itself known. Doesn’t make it easy to debug! But when it does happen it seems to explode and I get 100’s of messages.

EDIT: WARN in messages log:

-rw-rw-r-- 1 root syslogd 977084243 Jun 29 14:56 /var/log/messages
Failed during write                      = 66346
Missing data detected                    = 12222
Increase values or look . . .            = 12222

When you were building pmacct for ipfire, did you ever do a build with ZeroMQ?

I ask because a some of the threads with similar errors as mine (above) mention ZeroMQ. But there is no “SOLVED” type response so I don’t know if it works or not.


EDIT2:
Might be time to open another Issue report with pmacct GitHub.

1 Like

Good morning Jon,

the building time is long ago but at that time i used also Ntopng which needed ZeroMQ and also nDPI so i used both for Pmacct since they where at that time simply available in my building environment but i left them behind since both packages are not part of IPFire systems.

This might be the best, even if ZeroMQ would solve this problem, which i do not think, this would be only a shifting of a not solved problem.

Will have an eye on it if this problem comes by time also here around.

Best,

Erik

Done!

1 Like

Thinking out loud: I wonder if these WARN ( plugin1/memory ) lines are caused by enabling debug in the config file ?!?
:thinking:

FYI from pmacct concerning issue:

EDIT: added response.

Hi Jon,

As indicated in the quickstart guide, i’d totally discourage you to use the native version of internal queueing (what you are using right now) in favor of ZMQ. Please see the X. Internal buffering and queueing section of the QUICKSTART document at this propo: pmacct/QUICKSTART at master · pmacct/pmacct · GitHub

Paolo

https://github.com/pmacct/pmacct/issues/701#issuecomment-1615310861

1 Like

Hi Jon,
so ZeroMQ should be the way to go ? As far as i can see in IPFire building logs only ‘frr-8.0.1’ checks via configure openly for libzmq but beneath also Pmacct may there is more ? Currently not sure about this. What want you to do ?

Best,

Erik

This is above my skill level… I am not sure what this all means. Maybe Adolf @bonnietwin can assist?!?

FYI - I do get a ton more WARNs after enabling debug in the config file debug: true. I would expect a ton more DEBUG messages (and I get those) but not more WARNs.


EDIT: I’ve been reading through some of the zeromq docs and have a little (very little) idea of what is does.

Yes, I believe it is!

I see the libzmq references in the GitHub for frr but I don’t see any libzmq in IPFire. So I don’t understand the question.

If this can be added to the pmacct build that would be great! But I have to go by your recommendations.

Good morning Jon,

Yes as before written ZeroMQ (like nDPI) was and is not part of IPFire. Was digging a little through the build logs to get an idea which software can participate also from a asynchronous messaging library and frr was the only one which showed up but this does not mean that others does not have a compile time option for libzmq. This might be the first step to decide →

  1. if it might be a good idea to integrate a new library into IPFire (merge request) to find good arguments.
  2. On which place should ZeroMQ be placed in make.sh .
  3. Should it be part of the core system or delivered as a package (Pakfire).
  4. If checked which already running software can participate from ZeroMQ, adapt the appropriate LFS file.

This should then be the first steps before asking in the developer mailinglist in my opinion.

Best,

Erik

First thing that I found when looking at the zmq github page is that the last official release was in Jan 2017. However large numbers of commits have been done in the meantime with many raised issues being fixed.
I have found this with at least one other Open Source project that IPFire uses for an addon (lcdproc) where the last official tarball was in 2017 and you have to decide to do an extraction from github at a certain commit to use the most up to date option. I had to do that with lcdproc as a user had experienced a bug which had been fixed but only as a commit.

This is really not an ideal situation as you have to choose the commit carefully to make sure you have a self consistent set of changes.

I believe that if it is needed for pmacct then it should be installed with that package not as a core program.

As a dependency of pmacct place it just before pmacct in make.sh would be my choice.

Why would we want to do that. Is there a problem in the communication between different packages currently. If we do that are we sure that it doesn’t bring any security vulnerabilities or weaknesses.
Anyway, my view is that it should be used with pmacct if that needs to use it and nothing else currently.

If there is some generic structural benefit in using zmq for IPFire, then that should be considered for IPFire3.x

The last one from Github is from 2021 → Release libzmq 4.3.4 · zeromq/libzmq · GitHub but in principal you are right two years have a lot of stuff in it since the last commit is two weeks ago and the project looks quite busy.
For the rest do it like you want, am anyways good with it also without ZeroMQ since i do not have the problems which Jon have. According to your security concerns, i think ZeroMQ uses a good way in cryptography and security → Tinkering with ZeroMQ security - Y Soft Engineering Blog .
The principle of ZeroMQ should not be a problem solver with communication between different packages i think it should be a booster for them but as before, a good more fundamental understanding of what ZeroMQ is in general should be made IMHO.

Best,

Erik