Data usage monitoring

Community
#1

I’ve done some searching and I see some discussion on this in the past. I’m about to switch my Comcast cable account over to a program that has 1TB a month data cap.

I want to on a monthly basis track current data usage per device and a monthly grand total.

Now I’ve seen some discussion of ntopng and iptraf.

I could use ntopng and do a port mirroring on my switch to another machine… However I would like to know if anyone else has any other solutions.

thanks,

#2

There is general collection of the data that is being passed on each interface.

However, there is no automatic deactivation of the Internet connection after a quota has been reached.

#3

I just want to monitor,
Is there an internal database that shows per device data usage that I can sum up? Basically where is the general collection kept at ?

#4

I use Comcast and I am looking for the same thing:

IPFire (old) Forum - Help Monitoring Data Caps

IPFire (old) Forum - Traffic accounting

 

The closest I came to monitoring each device was using pmacct (and with a ton of help from Erik @ummeegge!)

But I had to post process everything in Excel using pivot table to get the info I wanted. And I could not determine how to make that happen via a script.

IPFire (new) Community - Pmacct - A lightweight passive network monitoring tool

#5

Did you try sarg addon? Shows hourly, daily, weekly and monthly data usage per device so in my mind.

#6

Doesn’t sarg require everything to go through the squid proxy? Meaning all the bittoreent / https traffic wouldn’t get counted?

Also I don’t see sarg in the firepak list of available packages.

#7

This is very interesting!!!
I work for Cisco specifically in the in the Cloud Ops department so doing queries in sql is no problem. This seems like a great starting point.

Thanks I’m going to start working on this and see where I end up.

#8

For me Sarg did not give me the correct info. For all of November 2019:

  • Comcast = 411 GB
    • from the Comcast website
  • IPFire = 413.36 GiB
    • from IPFire WUI menu Status > Net-Traffic
  • SARG = 110.44 G Bytes
    • from the Squid Analysis Report Generator

Maybe SARG is only looking at non-SSL type data? (e.g., HTTP and not HTTPS). I am guessing!

Which user (machine) is using all the bandwidth
#9

Having someone that does SQL queries is worth gold! Please share what you come up with. I’d be happy to help test.

#10

You are right. Squid is non-TLS/SSL only.
Without breaking the certificate chain of Internet data stream squid is not able to analyze all traffic.
But that’s another topic/thread.

#11

If you want you can send me via a link a dump of your sql database and I can write some queries for you with sql and /or make sql queries in python. I do that all the time for work.

#12

What instructions did you use to compile pmacct for ipfire?

#13

Hi all,

we used here different versions with Sqlite only but also with libmaxminddb and rabbitmq --> https://people.ipfire.org/~ummeegge/pmacct/ where some versions needed additional packages. In the old forum beneath a lot more tests, versions with support for MySQL and PostgreSQL has also been build.
Some potential build instructions can be found e.g. in here --> https://git.ipfire.org/?p=people/ummeegge/ipfire-2.x.git;a=blob;f=lfs/pmacct;h=01a120f4ead2f86e80c50380772f81a1705d9321;hb=7756aac6bec7fb8944aab1cf1d83ff32e9fd90d5 which is not the current status (–enable-zmq was also enabled) but i think it should work without other dependencies.

In general, this is not the currently available test version since for example the PAKS scripts are meanwhile adapted for Pakfire this process is not well checked at the moment (have currently not that much time for this project).

Please feel free to test extend and share new knowledge again --> https://git.ipfire.org/?p=people/ummeegge/ipfire-2.x.git;a=commit;h=7756aac6bec7fb8944aab1cf1d83ff32e9fd90d5 .

Best,

Erik

EDIT:

  • If you want to use Pmacctd with Sqlite you need to adapt the configuration since the pushed one is held as simple as possible (potential Addon version) .
    OT
  • May you know it already but if you administer Cisco machines there seems to be a problem with the certificates
    " Self-signed X.509 PKI certificates (SSC) that were generated on devices that run affected Cisco IOS® or Cisco IOS XE software releases expire on 2020-01-01 00:00:00 UTC. New self-signed certificates cannot be created on affected devices after 2020-01-01 00:00:00 UTC . Any service that relies on these self-signed certificates to establish or terminate a secure connection might not work after the certificate expires."
    just as a beneath info :slightly_smiling_face:
1 Like
#14

May you know it already but if you administer Cisco machines there seems to be a problem with the certificates
" Self-signed X.509 PKI certificates (SSC) that were generated on devices that run affected Cisco IOS® or Cisco IOS XE software releases expire on 2020-01-01 00:00:00 UTC. New self-signed certificates cannot be created on affected devices after 2020-01-01 00:00:00 UTC . Any service that relies on these self-signed certificates to establish or terminate a secure connection might not work after the certificate expires."
just as a beneath info :slightly_smiling_face:

idiots in another division… I’m in the collaboration group. We do the webex Teams. Specifically Logging Metrics and analytics.