Pmacct: Memory WARN messages

jon · 28 June 2023 21:21

Continuing the discussion from Pmacct - "Unknown key: interface" , "no such column: vlan_in":

@tphz, @ummeegge, and everyone else!

Has anyone run into these memory WARN messages related to pmacct?

I see LOTS of these messages from the memory plugin:

Jun 28 13:58:42 ipfire pmacctd[4228]: WARN ( plugin1/memory ): Failed during write: Resource temporarily unavailable 
. . .
Jun 28 13:58:53 ipfire pmacctd[4232]: WARN ( plugin1/memory ): Missing data detected (plugin_buffer_size=102400 plugin_pipe_size=102400000). 
Jun 28 13:58:53 ipfire pmacctd[4232]: WARN ( plugin1/memory ): Increase values or look for plugin_buffer_size, plugin_pipe_size in CONFIG-KEYS document.

.

If I enable debug in the config file then I see these messages related to the config file:

Jun 28 14:02:09 ipfire pmacctd[19511]: INFO ( plugin1/memory ): plugin_pipe_size=10240000 bytes plugin_buffer_size=10240 bytes
Jun 28 14:02:09 ipfire pmacctd[19511]: INFO ( plugin1/memory ): ctrl channel: obtained=212992 bytes target=8000 bytes

.
My current config is this:

!----------------------------  memory  ------------------------------

!
! "plugin1" plugin configuration
!
!plugins: memory[plugin1]

plugin_buffer_size[plugin1]: 10240
plugin_pipe_size[plugin1]:   10240000

imt_path[plugin1]: /var/spool/pmacct/plugin1.pipe

aggregate[plugin1]: src_host, src_mac, dst_host, dst_mac
aggregate_filter[plugin1]: ip

I did find this:

github.com

pmacct/pmacct/blob/master/CONFIG-KEYS#L324-L354


      
          KEY:		plugin_pipe_size
          DESC:		Core Process and each of the plugin instances are run into different processes. To
          		exchange data, they set up a circular queue (home-grown implementation, referred to
          		as 'pipe') and highlighted below with "XXXX":
          
          
							      XXXX
          		[network] ----> [kernel] ----> [core process] ----> [plugin] ----> [backend]
          					       [__________pmacct___________]
          
          
		This directive sets the total size, in bytes, of such queue. Its default size is set
          		to 4MB. Whenever facing heavy traffic loads, this size can be adjusted to hold more
          		data. In the following example, the queue between the Core process and the plugin
          		'test' is set to 10MB:
          
          
		...
          		plugins: memory[test]
          		plugin_pipe_size[test]: 10240000 
          		...
          
          
		When enabling debug, log messages about obtained and target pipe sizes are printed.

This file has been truncated. show original

But it did not help me… And I’ve tried many “guesses” for plugin_pipe_size and plugin_buffer_size.

Were you able to fix this?

ummeegge · 29 June 2023 19:13

Hello John,
have no warnings in debug mode like this therefor have not increased the buffer/pipe size. What configuration (in what) environment are you running Pmacct ?

Best,

Erik

jon · 29 June 2023 19:29

The IPFire environment.

For what it is worth…

It looks like there was another pmacct change with the new pmacct 1.7.8 version. After CU 175 these WARNs started:

for logf in $(ls -1tr /var/log/messages*) ; do zgrep --color -aie "CORE UPGR: Upgrading from release.*" -e "Failed during write:" $logf ; done
. . .
Dec 30 12:16:18 ipfire pakfire: CORE UPGR: Upgrading from release 171 to 172
Feb 28 12:01:02 ipfire pakfire: CORE UPGR: Upgrading from release 172 to 173
Apr 19 21:31:21 ipfire pakfire: CORE UPGR: Upgrading from release 173 to 174
Jun 13 22:21:09 ipfire pakfire: CORE UPGR: Upgrading from release 174 to 175
Jun 25 13:31:20 ipfire pmacctd[21940]: WARN ( plugin1/memory ): Failed during write: Connection refused 
Jun 25 15:06:17 ipfire pmacctd[32658]: WARN ( plugin1/memory ): Failed during write: Resource temporarily unavailable 
Jun 25 15:06:18 ipfire pmacctd[32658]: WARN ( plugin1/memory ): Failed during write: Resource temporarily unavailable 
Jun 25 15:06:18 ipfire pmacctd[32658]: WARN ( plugin1/memory ): Failed during write: Resource temporarily unavailable

But it took a few days to find it…

jon · 29 June 2023 19:47

Erik - It is nice to see you around!

One thing to point out in my post directly above. It took about 12 days after my CU 175 update for the WARN messages to appear. I rebooted yesterday and only one WARN message has appeared so far. So this seems to be an issue that takes a few days to make itself known. Doesn’t make it easy to debug! But when it does happen it seems to explode and I get 100’s of messages.

EDIT: WARN in messages log:

-rw-rw-r-- 1 root syslogd 977084243 Jun 29 14:56 /var/log/messages
Failed during write                      = 66346
Missing data detected                    = 12222
Increase values or look . . .            = 12222

When you were building pmacct for ipfire, did you ever do a build with ZeroMQ?

I ask because a some of the threads with similar errors as mine (above) mention ZeroMQ. But there is no “SOLVED” type response so I don’t know if it works or not.

EDIT2:
Might be time to open another Issue report with pmacct GitHub.

ummeegge · 30 June 2023 08:55

Good morning Jon,

the building time is long ago but at that time i used also Ntopng which needed ZeroMQ and also nDPI so i used both for Pmacct since they where at that time simply available in my building environment but i left them behind since both packages are not part of IPFire systems.

This might be the best, even if ZeroMQ would solve this problem, which i do not think, this would be only a shifting of a not solved problem.

Will have an eye on it if this problem comes by time also here around.

Best,

Erik

jon · 30 June 2023 16:21

Done!

github.com/pmacct/pmacct

pmacct 1.7.8 - Failed during write: Resource temporarily unavailable

opened 04:19PM - 30 Jun 23 UTC

closed 10:10PM - 19 Jul 23 UTC

JonMurphy

question

- [ ] **Description** I see LOTS of these messages from the **memory** plugin a…fter updating from pmacct version 1.7.6 to 1.7.8. ``` Jun 28 13:58:42 ipfire pmacctd[4228]: WARN ( plugin1/memory ): Failed during write: Resource temporarily unavailable . . . Jun 28 13:58:53 ipfire pmacctd[4232]: WARN ( plugin1/memory ): Missing data detected (plugin_buffer_size=102400 plugin_pipe_size=102400000). Jun 28 13:58:53 ipfire pmacctd[4232]: WARN ( plugin1/memory ): Increase values or look for plugin_buffer_size, plugin_pipe_size in CONFIG-KEYS document. ``` If I enable debug in the config file then I see these messages: ``` Jun 28 14:02:09 ipfire pmacctd[19511]: INFO ( plugin1/memory ): plugin_pipe_size=10240000 bytes plugin_buffer_size=10240 bytes Jun 28 14:02:09 ipfire pmacctd[19511]: INFO ( plugin1/memory ): ctrl channel: obtained=212992 bytes target=8000 bytes ``` My config is this. Please see: https://github.com/pmacct/pmacct/issues/696#issuecomment-1611655481 I did find this: https://github.com/pmacct/pmacct/blob/master/CONFIG-KEYS#L324-L354 But it did not help me... And I've tried many "guesses" for plugin_pipe_size and plugin_buffer_size. One thing to point out - It took about 12 days after the update for the pmacct messages to appear. So this seems to be an issue that takes a few days to make itself known. And it doesn't make it easy to debug! But when it does happen it seems to explode and I get 100's of messages. pmacct message count in log starting June 13: ``` -rw-rw-r-- 1 root syslogd 977084243 Jun 29 14:56 /var/log/messages Failed during write (count) = 66346 Missing data detected (count) = 12222 Increase values or look . . . (count) = 12222 ``` <br> **Version** Provide the version in use. An output of -V is typically good, ie. `nfacctd -V` ``` [**root@ipfire** ~] # pmacct -V pmacct IMT plugin client, pmacct 1.7.8-git (RELEASE) ```

jon · 1 July 2023 00:43

Thinking out loud: I wonder if these WARN ( plugin1/memory ) lines are caused by enabling debug in the config file ?!?

jon · 1 July 2023 15:03

FYI from pmacct concerning issue:

EDIT: added response.

Hi Jon,

As indicated in the quickstart guide, i’d totally discourage you to use the native version of internal queueing (what you are using right now) in favor of ZMQ. Please see the X. Internal buffering and queueing section of the QUICKSTART document at this propo: https://github.com/pmacct/pmacct/blob/master/QUICKSTART

Paolo

https://github.com/pmacct/pmacct/issues/701#issuecomment-1615310861

ummeegge · 2 July 2023 17:56

Hi Jon,
so ZeroMQ should be the way to go ? As far as i can see in IPFire building logs only ‘frr-8.0.1’ checks via configure openly for libzmq but beneath also Pmacct may there is more ? Currently not sure about this. What want you to do ?

Best,

Erik

jon · 2 July 2023 18:20

This is above my skill level… I am not sure what this all means. Maybe Adolf @bonnietwin can assist?!?

FYI - I do get a ton more WARNs after enabling debug in the config file debug: true. I would expect a ton more DEBUG messages (and I get those) but not more WARNs.

EDIT: I’ve been reading through some of the zeromq docs and have a little (very little) idea of what is does.

Yes, I believe it is!

I see the libzmq references in the GitHub for frr but I don’t see any libzmq in IPFire. So I don’t understand the question.

If this can be added to the pmacct build that would be great! But I have to go by your recommendations.

ummeegge · 3 July 2023 06:00

Good morning Jon,

Yes as before written ZeroMQ (like nDPI) was and is not part of IPFire. Was digging a little through the build logs to get an idea which software can participate also from a asynchronous messaging library and frr was the only one which showed up but this does not mean that others does not have a compile time option for libzmq. This might be the first step to decide →

if it might be a good idea to integrate a new library into IPFire (merge request) to find good arguments.
On which place should ZeroMQ be placed in make.sh .
Should it be part of the core system or delivered as a package (Pakfire).
If checked which already running software can participate from ZeroMQ, adapt the appropriate LFS file.

This should then be the first steps before asking in the developer mailinglist in my opinion.

Best,

Erik

bonnietwin · 3 July 2023 07:45

First thing that I found when looking at the zmq github page is that the last official release was in Jan 2017. However large numbers of commits have been done in the meantime with many raised issues being fixed.
I have found this with at least one other Open Source project that IPFire uses for an addon (lcdproc) where the last official tarball was in 2017 and you have to decide to do an extraction from github at a certain commit to use the most up to date option. I had to do that with lcdproc as a user had experienced a bug which had been fixed but only as a commit.

This is really not an ideal situation as you have to choose the commit carefully to make sure you have a self consistent set of changes.

I believe that if it is needed for pmacct then it should be installed with that package not as a core program.

As a dependency of pmacct place it just before pmacct in make.sh would be my choice.

Why would we want to do that. Is there a problem in the communication between different packages currently. If we do that are we sure that it doesn’t bring any security vulnerabilities or weaknesses.
Anyway, my view is that it should be used with pmacct if that needs to use it and nothing else currently.

If there is some generic structural benefit in using zmq for IPFire, then that should be considered for IPFire3.x

ummeegge · 3 July 2023 08:16

The last one from Github is from 2021 → Release libzmq 4.3.4 · zeromq/libzmq · GitHub but in principal you are right two years have a lot of stuff in it since the last commit is two weeks ago and the project looks quite busy.
For the rest do it like you want, am anyways good with it also without ZeroMQ since i do not have the problems which Jon have. According to your security concerns, i think ZeroMQ uses a good way in cryptography and security → Tinkering with ZeroMQ security - Y Soft Engineering Blog .
The principle of ZeroMQ should not be a problem solver with communication between different packages i think it should be a booster for them but as before, a good more fundamental understanding of what ZeroMQ is in general should be made IMHO.

Best,

Erik