Routers including IPfire go down over massive NAT

[bassie (Bas Heijermans)]

Hi all,

I have been searching for more then 1 year to solve my problem.

It started with me running an NTP-server behind NAT.

All of a sudden I noticed that my home-network started to act very weird.

I was using a Fritzbox 7590, so I bought and 5690Pro and returned the first to AVM to have it repaired (also because of a Wifi-issue).

Thinking the 5690Pro is a brand new box, it can handle the load…nope it can’t.

Ok, no problem, my ISP was using Draytek for big clients, so buy a Draytek, it can handle 60K NAT…nope can’t.

Bought me a MikroTek, all said it can do it…no again!

No problem, I build my own router with IPfire…what?? It fails too!!!

After a lot of testing I found that the NAT-tables got filled when using 50 packets/sec, but those are 50 different IP’s, so within a few hours NAT stops working properly, then DNS quits and in the end you can only unplug it.

The problem is in the kernel parameters, they are not modified to handle such.

The solution is simple, change /etc/sysctl.conf and add:

net.netfilter.nf_conntrack_udp_timeout = 15 
net.core.netdev_max_backlog = 4000
net.netfilter.nf_conntrack_max = 1048576

UDP needs 15 sec else VOIP will do strange things.

See what happens after the change:

Peaks of 1300 packets/second, all different IP’s. And it keeps being fast.

So I ask you, please adapt the kernel-parameters to avoid problems with massive NAT’ing going wrong.

If you want to test yourself. Start an NTP-server yourself, add it to the NTPpool.org and set the speed setting to max 3GB/sec and see your box go down after a few hours.

With my settings it keeps humming like nothing happens.

My 2cts, for this fantastic piece of software.

Bas.

[bassie (Bas Heijermans)]

Is it possible to have firewall rules with an extra marking, where you can set No-state-tracking?

As for NTP packets we do not need state tracing as after the time is send the requester won’t be back for hours or more.

So to keep the state is a waste of resources.

Thanks.

[ms (Michael Tremer)]

Hello Bas,

@bassie

No problem, I build my own router with IPfire…what?? It fails too!!!

Oh yes it can. We are also running a public NTP/NTS instance inside the IPFire infrastructure and the firewall for that is of course IPFire.

@bassie

net.netfilter.nf_conntrack_max = 1048576

The value for this is automatically determined by the kernel and depends on how much memory your system has. It is a simple protection against the system opening too many connections at the same time so it won’t run out of memory.

You can safely tune this if you have sufficient system memory.

@bassie

Is it possible to have firewall rules with an extra marking, where you can set No-state-tracking?

No, this is not supported, but you can create your own firewall rules in the CUSTOMINPUT/CUSTOMFORWARD/CUSTOMOUTPUT chains.

The reason why we cannot easily support this is that a lot of state is stored with the connection. For example QoS markers, whether the packet has been seen by the IPS and so on.

And as mentioned before, we don’t seem to need any of this on our node.

We are also running some large DNS infrastructure that uses a lot of UDP datagrams and we don’t have any problems there either.

The 15s timeout should mean that you see about 1300 packets/s * 15 seconds = 19,500 entries in the connection tracking table. Your limit is over a million, so you are good. NTP actually transmits a couple of packets which will re-use the same connection, so this is at least halved… Connection tracking should not be a problem here.

[dr_techno (Dave Mikeska)]

Problem is consumer connections have too high of an up traffic ping time to host a public ntp server and even then, you wouldn’t want to run it on a computer in the first place. I would run GPS + oven controlled crystal oscillator NTP/PTP server. Which is in my list of network upgrades since I’m just currently running a GPS style NTP and want PTP master clock for my network audio. But I wouldn’t host it for public.

If you want to host a public NTP then you will need to load balance at least 5 outside ip addresses with at least a public dns cluster of at least 5 name servers. Otherwise it would get overloaded with 1 connection any ways when you get a lot of ntp clients requesting updates.