Example Customized QoS - Update

[bloater99 (Tim Zakharov)]

I did one of these posts many years ago in 2015. The internet speeds back then were much slower. The line I presented back then was 10Mbps/2Mbps. Today I wanted to post my current home setup. The ISP says this is a 250/25 connection. In reality, actual speeds are closer to 300/30 when the network is quiet. Sometimes even faster. Note the downlink (22ms) and uplink (18ms) Pings are higher than idle (12ms) ping.

I’m going to discuss a couple of things that break the standard rules for QoS (such as the rule to reduce your ISP speeds by 3%). This, in part, is due to the firewall cpu’s power or lack thereof, as well as normal variation of internet connections.

First, when you set your ISP speeds at the top of the QoS page, behind the scenes there is a 10% reduction which happens automatically in the settings file. If your network and buffers can handle it, you can “overclock” these numbers so that after a 10% reduction, your speeds are where you want them. The settings file is in /var/ipfire/qos. Here is the relevant bit from my file:

INC_SPD=350000 <-what you input in the QoS page
OUT_SPD=30000 <-what you input in the QoS page
DEF_INC_SPD=315000 <-this is generated automatically in this file
DEF_OUT_SPD=27000 <-this is generated automatically in this file

You can see that my inputted values of 350000 and 30000 are already reduced to 315000/27000. In reality, as long as these numbers are not lower than any of your fastest class speeds, you can make them as high as you want. Mine end up at 315/27.

Now, when you start adjusting the Guaranteed and Maximum bandwidths in each of your classes, this is where you want to make sure you do not exceed any buffers. Here is what my classes file at /var/ipfire/qos looks like:

imq0;200;1;1000;2000;;;;Ping;
imq0;202;2;1000;2000;;;;DNS;
imq0;203;3;225000;300000;;;;Web;
imq0;204;4;10000;26500;;;;VPN;
imq0;205;5;36500;200000;;;;Email;
imq0;210;6;26500;250000;;;;Default;
red0;101;1;1000;2000;;;;Ping;
red0;102;2;1000;5000;;;;DNS;
red0;103;3;10000;27000;;;;Web;
red0;104;4;10000;26500;;;;VPN;
red0;105;5;4000;25000;;;;Email;
red0;110;6;1000;25000;;;;Default;

A couple of things to note:
-make sure that the sum of your guaranteed bandwidths in each class does not exceed your maximum bandwidth. You can see in the Classes file above that the sum of my downlink guaranteed bandwidths is 300000 and the sum of the guaranteed uplink bandwidths is 27000. Classes 103/203 (Web) have maximum bandwidths set to 27000/300000 respectively. Bingo!
-use a bufferbloat testing site to fine-tune these values. When you get the best combination of speeds and low latency, save it and run it awhile. Test again during peak network busyness to see if latency remains low. Also look at the gateway graph to see if spikes in gateway latency are reduced or eliminated. This is a good indication that QoS is set up well.

Regarding the CPU in your IPFire, if it is too slow, it won’t be able to keep up with the speeds you set. Most CPUs should be able to maintain at least 100Mbps speeds with QoS enabled. If your internet speeds are higher than that, then you are more likely to have computational slowdowns. The only way to know for sure is trial and error. For example, I tested an Intel J3160 cpu and it had trouble maintaining speeds as I increased the maximal QoS downlink speeds. What you can try in this situation is additional “overclocking”. Increase your QoS bandwidths beyond the actual maximum speeds so that the end result after cpu slowdown is where you’d like it to be. For example, you might try setting your downlink speed to 300Mbps to achieve a real-world speed of 250Mbps. Again, lots of trial and error will be involved, but it can pay off.

Lastly (and this is outside of what most of this post is about), you can customize what types of network traffic are shaped within the portconfig and level7config files. Here are mine:

portconfig
101;red0;icmp;;;;;
102;red0;tcp;;;;53;
102;red0;tcp;;;;853;
102;red0;udp;;;;53;
102;red0;udp;;;;853;
103;red0;tcp;;;;443;
103;red0;tcp;;;;80;
104;red0;esp;;;;;
104;red0;tcp;;1194;;;
104;red0;tcp;;;;1194;
104;red0;udp;;1194;;;
104;red0;udp;;4500;;4500;
104;red0;udp;;500;;500;
104;red0;udp;;;;1194;
105;red0;tcp;;25;;;
105;red0;tcp;;465;;;
105;red0;tcp;;587;;;
105;red0;tcp;;;;25;
105;red0;tcp;;;;465;
105;red0;tcp;;;;587;
105;red0;udp;;25;;;
105;red0;udp;;465;;;
105;red0;udp;;587;;;
105;red0;udp;;;;25;
105;red0;udp;;;;465;
105;red0;udp;;;;587;
200;imq0;icmp;;;;;
202;imq0;tcp;;53;;;
202;imq0;tcp;;853;;;
202;imq0;udp;;53;;;
202;imq0;udp;;853;;;
203;imq0;tcp;;443;;;
203;imq0;tcp;;80;;;
204;imq0;esp;;;;;
204;imq0;tcp;;1194;;;
204;imq0;tcp;;;;1194;
204;imq0;udp;;1194;;;
204;imq0;udp;;4500;;4500;
204;imq0;udp;;500;;500;
204;imq0;udp;;;;1194;
205;imq0;tcp;;110;;;
205;imq0;tcp;;993;;;
205;imq0;tcp;;995;;;
205;imq0;tcp;;;;110;
205;imq0;tcp;;;;993;
205;imq0;tcp;;;;995;
205;imq0;udp;;110;;;
205;imq0;udp;;993;;;
205;imq0;udp;;995;;;
205;imq0;udp;;;;110;
205;imq0;udp;;;;993;
205;imq0;udp;;;;995;

level7config
102;red0;dns;;;
102;red0;rtp;;;
102;red0;skypetoskype;;;
103;red0;http;;;
103;red0;ssl;;;
104;red0;rdp;;;
104;red0;ssh;;;
104;red0;vnc;;;
105;red0;imap;;;
105;red0;smtp;;;
202;imq0;dns;;;
202;imq0;rtp;;;
202;imq0;skypetoskype;;;
203;imq0;http;;;
203;imq0;ssl;;;
204;imq0;rdp;;;
204;imq0;ssh;;;
204;imq0;vnc;;;
205;imq0;imap;;;
205;imq0;pop3;;;

Finally, here is a look at the UI page in total:
ipfire.home - QoS.pdf (416.9 KB)

And here are speedtest results:
Speedtest.net (note the download and upload latencies being lower than idle latency)

Cloudflare Speedtest:

Waveform Bufferbloat:

For comparison, here are speedtest results with QoS DISabled:

Speedtest.net:

Cloudflare:

Waveform:

Note the increased latency during downloads and uploads. Also, in the Cloudflare result, the 50Mbps upload is mostly fake. Our ISP has a temporary speed that is quickly saturated, then drops back to ~29Mbps.

[ms (Michael Tremer)]

Hello,

thanks for giving IPFire’s QoS a good test. It looks like it is doing well…

@bloater99

I’m going to discuss a couple of things that break the standard rules for QoS (such as the rule to reduce your ISP speeds by 3%). This, in part, is due to the firewall cpu’s power or lack thereof, as well as normal variation of internet connections.

The reason why there is this recommendation to take off a few percent is simply packet header overhead. It has nothing to do with the CPU performance of the firewall.

Historically, ISPs gave you a line. Whatever technology was chosen, you could transfer data at line speed. There was no throttling. In the commercial and heavily contested world that we are in today, ISPs are slicing those lines into smaller chunks and you book a plan that usually has some throttling. But how is that throttling done?

A 1 GBit/s line transfers exactly that amount of data. That includes everything. The IP packet, Ethernet header and usually there is more encapsulation like PPP headers and so on. The QoS works on the Ethernet packet basis, because in case of a DSL connection, we don’t know the headers that the modem will add to the packet. We only know what amount of data we are sending to the modem.

Know, people probably complained, that when they booked a 100 MBit/s plan, they only got 95 MBit/s in a speed test. Because overhead and what not. So ISPs will normally give you a little bit extra so that you will at least see the number that you are paying for. You might not be able to go over the contracted speed for too long, because they could treat this as a burst to make speed tests work; you might also have an ISP that does not care and will simply give you the extra bandwidth all of the time.

The QoS is configured for the most pessimistic (or sometimes we call it ‘conversative’) scenario. The burst is nice, but we cannot keep sending packets at that rate for ever. If we do, we will pile up packets in the DSL modem/DSLAM/cable modem, or somewhere else on the way, where the throttling is being applied. That is what we call buffer bloat. The buffer fills up and then packets are queued.

@bloater99

First, when you set your ISP speeds at the top of the QoS page, behind the scenes there is a 10% reduction which happens automatically in the settings file. If your network and buffers can handle it, you can “overclock” these numbers so that after a 10% reduction, your speeds are where you want them.

So this overclocking will only work if your ISP is permanently giving you more bandwidth than you contracted for. I approve of your method to simply test this, but that is a lot of effort and I think the “take a few percent off” rule came from that place of laziness.

@bloater99

Test again during peak network busyness to see if latency remains low.

This is however a dangerous part: If latency goes up at peak times, that means there is some congestion inside the ISP’s network or even later on. The QoS cannot control this and therefore you should only manage the immediate queue(s) - i.e. the one where your ISP throttles your connection. That could even be IPFire itself if you have a 100 MBit/s fibre connection, but your LAN is connected with the usual Gigabit infrastructure.

@bloater99

Regarding the CPU in your IPFire, if it is too slow, it won’t be able to keep up with the speeds you set. Most CPUs should be able to maintain at least 100Mbps speeds with QoS enabled. If your internet speeds are higher than that, then you are more likely to have computational slowdowns. The only way to know for sure is trial and error. For example, I tested an Intel J3160 cpu and it had trouble maintaining speeds as I increased the maximal QoS downlink speeds. What you can try in this situation is additional “overclocking”.

I don’t understand this part. The work the CPU has to do is always the same. Each packet has to be accounted for and QoS needs to decide which packet to send next. If you have more bandwidth, there are obviously more packets and that means more work. But a higher limit does not change anything here.

What am I missing?

[bloater99 (Tim Zakharov)]

@ms

The reason why there is this recommendation to take off a few percent is simply packet header overhead.

I understand where the 3% comes from as well. It’s an effort to keep buffers from queuing.

@ms

It has nothing to do with the CPU performance of the firewall.

What I am getting at is, in the thread I link to later about the Intel J3160 cpu, Arne determined that with certain CPUs, Cake can bottleneck QoS set to higher link speeds. Also, I did some extensive testing showing how this bottlenecking can look with the J3160. What this means is, with some cpus and higher link speeds, QoS speed settings may have to be set higher in order to obtain lower speeds. If you look at the chart I link to above(the link “I did some extensive testing”), it will become more clear. Also, if you look at my results at the bottom of my opening post, you will see that I am setting QoS speeds to 300Mbps but only getting a maximum of ~275Mbps. When I disable QoS, speeds increase to ~300Mbps. This is where my term “overclocking” comes to play.

@ms

So this overclocking will only work if your ISP is permanently giving you more bandwidth than you contracted for.

In this case, I am explaining about something IPFire’s QoS does in the file at /var/ipfire/qos/settings. When you set your bandwidth at the top of the QoS page (in my case it was 350000/30000), IPFire reduces these numbers by 10% in the settings file. See the lines
DEF_INC_SPD=
DEF_OUT_SPD=
in my original post above. I believe this is a more stringent interpretation of the 3% reduction we discussed above, that is already happening in IPFire’s QoS. My “overclocking” comment here is basically to set your ISP’s speeds to 10% higher in the UI so that after IPFire reduces those values by 10%, you will still have room to maximize your speeds in each individual Class without automatically losing 10%.

I have found that you can set your ISP speeds at the top of the QoS page to 100 times higher than actual and it won’t matter, as long as each Class’s maximum bandwidth does not exceed your network buffers. I believe the purpose of the ISP speeds at the top of the page is for setting accurate automatic settings when you click the Preset button after setting ISP speeds. If you know what you are doing and are customizing each Class anyway, this won’t matter.

@ms

This is however a dangerous part: If latency goes up at peak times, that means there is some congestion inside the ISP’s network or even later on.

In my experience of using IPFire’s QoS for 10 years, a correctly configured QoS will maintain low latency even during high network congestion. That is part of its purpose, and it can be seen clearly on the gateway graph. If you run IPFire for a week without QoS, the gateway graph will have numerous spikes during high activity times. Turn on a correctly configured QoS and after a week, the graph will look much smoother with almost no spikes. This is because IPFire is not allowing data transmission to ever exceed any system buffers.

@ms

I don’t understand this part. The work the CPU has to do is always the same.

See my explanation above to your first question.

Cheers!

[ms (Michael Tremer)]

@bloater99

What I am getting at is, in the thread I link to later about the Intel J3160 cpu, Arne determined that with certain CPUs, Cake can bottleneck QoS set to higher link speeds. Also, I did some extensive testing showing how this bottlenecking can look with the J3160. What this means is, with some cpus and higher link speeds, QoS speed settings may have to be set higher in order to obtain lower speeds. If you look at the chart I link to above(the link “I did some extensive testing”), it will become more clear. Also, if you look at my results at the bottom of my opening post, you will see that I am setting QoS speeds to 300Mbps but only getting a maximum of ~275Mbps. When I disable QoS, speeds increase to ~300Mbps. This is where my term “overclocking” comes to play.

Okay, I see your results, but I don’t have an explanation for this.

Why do you think this could be?

I would regard the “overclocking” problem as a different problem. When I set it to a value X, I expect the QoS to output X bytes per second at a maximum. QoS is not heavily CPU-bound, so there should not be a strong connection to general CPU performance.

It might be a different problem to find the maximum value that your ISP allows you to transfer.

[bloater99 (Tim Zakharov)]

@ms

Okay, I see your results, but I don’t have an explanation for this.

Why do you think this could be?

I was hoping @arne_f would know. His tests in that same thread I linked to match my results and at one point he said the bottleneck is Cake

Regardless, my first post in this thread says to test for this when setting up your maximal bandwidth, and that lots of trial and error will be involved in fine-tuning QoS.

@ms

When I set it to a value X, I expect the QoS to output X bytes per second at a maximum. QoS is not heavily CPU-bound, so there should not be a strong connection to general CPU performance.

Arne’s post here suggests otherwise.

To clarify, my opening post in this topic is not asking for any solutions. It is simply to share my experiences with fine-tuning QoS in IPFire so that others may benefit.