QoS: Warning about AQM codel not being enabled

dnl · 9 May 2021 06:08

I’m seeing the following warning about AQM codel in messages when restarting QoS. I have bridging enabled on GREEN only.

May  9 16:01:25 ipfire root: Could not find a bridged zone for imq0
May  9 16:01:25 ipfire codel: Codel AQM could not be enabled on 'imq0'. Error code: 2
May  9 16:01:25 ipfire ipfire: QoS started
May  9 16:01:25 ipfire vnstatd[2306]: Interface "imq0" enabled.

Running qosctrl status |grep -i codel returns multiple lines, but I’m unclear if the error related to AQM is a problem.

In Question about QoS in 137 release - #7 by iotapi322 @ms mentioned an in Core Update 137, which was some time ago.

Is this something I should investigate or just a warning I can ignore please?

Thank you in advance.

jon · 10 May 2021 16:47

I have QOS enabled. I looked through my messages log and there is nothing for codel or AQM.

I’ll move your question to a new thread.

dnl · 10 May 2021 20:45

Please don’t. I’d rather repost it myself

jon · 11 May 2021 03:07

it can all be easily deleted

dnl · 11 May 2021 07:18

Sorry @jon it’s OK. It was someone else who shut a thread on me before I had a chance to respond.
I would have re-worded things and written a clear explanation of the problem so that it wasn’t out of context. It’s fine though.

EDIT: I’ve edited the first post above now. Thanks

dnl · 12 June 2021 11:10

I’m still getting codel: Codel AQM could not be enabled on 'imq0'. Error code: 2 in Syslog every time QoS is started. Does anyone know what I can investigate?

@ms mentioned a fix way back in Core Update 137. Is it possible there’s a related bug? Do I need to adjust my configuration somehow?

Thank you in advance

dnl · 12 June 2021 11:21

I found the error in the source code ipfire-2.x/enable_codel at 59ebcaedbd5a1cc2a451c80460f674c312edbcd8 · ipfire/ipfire-2.x · GitHub and sure enough when I run it manually I get the same result:

[root@ipfire ~]# /sbin/tc qdisc add root dev imq0 fq_codel
RTNETLINK answers: File exists
[root@ipfire ~]# echo $?
2

Does this imply that there is a problem when stopping and starting QoS? The “queuing discipline” cannot be added to the device because it’s already defined?

EDIT:

[root@ipfire ~]# /sbin/tc qdisc list root dev imq0 
qdisc htb 2: root refcnt 2 r2q 10 default 0x210 direct_packets_stat 0 direct_qlen 32
[root@ipfire ~]#

If I’ve done that properly, my link isn’t using CODEL but instead the “HTB” “Classfull QDISC”, which seems to be a mistake, given that IPFire promotes it specifically uses fq_codel.

I cannot reboot my router at this point, but when I can I’ll see if this is reported on boot.

dnl · 12 June 2021 11:41

It is happening on boot - fq_codel is not loaded, instead rqr is used for imq0.

I can export my QoS config, but I’m pretty sure there’s nothing unusual in it.

dnl · 12 June 2021 11:48

I’m really confused. Could a developer or expert please help?

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: red0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc htb state UP group default qlen 1000
    link/ether XX:XX:31:08:a4:d8 brd ff:ff:ff:ff:ff:ff
    inet xxx.xxx.xxx.xxx/xx brd xxx.xxx.xxx.xxx scope global dynamic noprefixroute red0
       valid_lft 446sec preferred_lft 371sec
3: orange0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether XX:XX:31:08:a4:d9 brd ff:ff:ff:ff:ff:ff
    inet 10.0.3.1/24 scope global orange0
       valid_lft forever preferred_lft forever
4: blue0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether XX:XX:31:08:a4:da brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.1/24 scope global blue0
       valid_lft forever preferred_lft forever
5: green0p0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master green0 state UP group default qlen 1000
    link/ether XX:XX:31:08:a4:db brd ff:ff:ff:ff:ff:ff
6: green0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 02:44:02:7e:db:a4 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.1/24 scope global green0
       valid_lft forever preferred_lft forever
7: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master green0 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:d6:1a:15 brd ff:ff:ff:ff:ff:ff
8: ifb0: <BROADCAST,NOARP> mtu 1500 qdisc fq_codel state DOWN group default qlen 32
    link/ether f2:91:48:e6:a8:00 brd ff:ff:ff:ff:ff:ff
9: ifb1: <BROADCAST,NOARP> mtu 1500 qdisc fq_codel state DOWN group default qlen 32
    link/ether fa:5f:4f:b9:24:df brd ff:ff:ff:ff:ff:ff
10: imq0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc htb state UNKNOWN group default qlen 32
    link/ether aa:92:ba:d7:ee:3e brd ff:ff:ff:ff:ff:ff

See that htb is active on imq0 however I have an ifb0 and an ifb1 interface with fq_codel set but they are down. Also QoS configuration specifically mentions imq0 not these other interfaces.

Please help! I’m out of my depth!

Here’s more context from the messages log after boot.

Jun 12 21:40:00 ipfire dhcpcd[4442]: DUID 00:04:03:00:02:00:04:00:05:00:00:06:00:07:00:08:00:09
Jun 12 21:40:00 ipfire dhcpcd[4442]: red0: IAID 31:08:a4:d8
Jun 12 21:40:00 ipfire dhcpcd[4442]: red0: adding address fe80::4262:31ff:fe08:a4d8
Jun 12 21:40:00 ipfire dhcpcd[4442]: ipv6_addaddr1: Permission denied
Jun 12 21:40:01 ipfire dhcpcd[4442]: red0: soliciting an IPv6 router
Jun 12 21:40:01 ipfire dhcpcd[4442]: red0: soliciting a DHCP lease
Jun 12 21:40:01 ipfire dhcpcd[4442]: red0: offered xxx.xxx.xxx.107 from xxx.xxx.xxx.1
Jun 12 21:40:02 ipfire dhcpcd[4442]: red0: probing address xxx.xxx.xxx.107/24
Jun 12 21:40:07 ipfire dhcpcd[4442]: red0: leased xxx.xxx.xxx.107 for 600 seconds
Jun 12 21:40:07 ipfire dhcpcd[4442]: red0: adding route to xxx.xxx.xxx.0/24
Jun 12 21:40:07 ipfire dhcpcd[4442]: red0: adding default route via xxx.xxx.xxx.1
Jun 12 21:40:07 ipfire dhcpcd.exe[4548]: red0 has been (re)configured with IP=xxx.xxx.xxx.107
Jun 12 21:40:07 ipfire unbound: [2367:0] info: generate keytag query _ta-4a5c-4f66. NULL IN
Jun 12 21:40:08 ipfire ntpd[2721]: Listen normally on 6 red0 xxx.xxx.xxx.107:123
Jun 12 21:40:08 ipfire ntpd[2721]: new interface(s) found: waking up resolver
Jun 12 21:40:11 ipfire kernel: DROP_INPUT IN=red0 OUT= MAC=XX:XX:31:08:a4:d8:3c:fd:fe:bd:a1:fa:08:00 SRC=92.63.197.94 DST=xxx.xxx.xxx.107 LEN=40 TOS=0x00 PREC=0x00 TTL=248 ID=51647 PROTO=TCP SPT=48703 DPT=46434 WINDOW=1024 RES=0x00 SYN URGP=0 
Jun 12 21:40:15 ipfire suricata: This is Suricata version 5.0.6 RELEASE running in SYSTEM mode
Jun 12 21:40:16 ipfire suricata: [ERRCODE: SC_WARN_NO_STATS_LOGGERS(261)] - stats are enabled but no loggers are active
Jun 12 21:40:16 ipfire suricata: all 4 packet processing threads, 2 management threads initialized, engine started.
Jun 12 21:40:16 ipfire suricata: rule reload starting
Jun 12 21:40:16 ipfire kernel: HTB: quantum of class 10001 is big. Consider r2q change.
Jun 12 21:40:17 ipfire kernel: u32 classifier
Jun 12 21:40:17 ipfire kernel:     Performance counters on
Jun 12 21:40:17 ipfire kernel:     input device check on
Jun 12 21:40:17 ipfire kernel:     Actions configured
Jun 12 21:40:17 ipfire kernel: Mirror/redirect action on
Jun 12 21:40:17 ipfire kernel: HTB: quantum of class 20001 is big. Consider r2q change.
Jun 12 21:40:17 ipfire kernel: HTB: quantum of class 20203 is big. Consider r2q change.
Jun 12 21:40:17 ipfire kernel: HTB: quantum of class 20204 is big. Consider r2q change.
Jun 12 21:40:17 ipfire root: Could not find a bridged zone for ifb0
Jun 12 21:40:17 ipfire codel: Codel AQM has been enabled on 'ifb0'.
Jun 12 21:40:17 ipfire root: Could not find a bridged zone for ifb1
Jun 12 21:40:17 ipfire root: Could not find a bridged zone for imq0
Jun 12 21:40:17 ipfire codel: Codel AQM has been enabled on 'ifb1'.
Jun 12 21:40:17 ipfire codel: Codel AQM could not be enabled on 'imq0'. Error code: 2
Jun 12 21:40:18 ipfire unbound: [2367:0] info: service stopped (unbound 1.13.1).

dnl · 13 June 2021 05:57

I can manually get the desired result by running:

/sbin/tc qdisc replace root dev imq0 fq_codel
/sbin/tc qdisc replace root dev red0 fq_codel

Every time after QoS is started - including after a fresh reboot.

I’ve looked through IPFire’s complex network scripts but have not yet found where the /lib/udev/enable_codel shell script is called from. (EDIT: I’m unclear on how udev is involved)
That script only caters for adding and removing a “qdisc”. It doesn’t check if one was already applied for some reason and try to instead replace it.

Surely other people are affected by this?

dnl · 13 June 2021 06:08

When manually changing to fq_codel I’m getting slightly better “bufferbloat” results from Speed test - how fast is your internet? | DSLReports, ISP Information?

dnl · 13 June 2021 06:41

So when I manually replace the htb qdisc with fq_codel the rrdtool graphs on the QoS page stop registering any traffic. This is not a good sign.

If that is only a problem with the graphs (and not some side-effect I don’t understand) then it might be best to modify the /lib/udev/enable_codel script so that the word “add” is changed to “replace” as the tc manual states it will create if it cannot replace:

       replace
              Performs a nearly atomic remove/add on an existing node id. If the node does not exist yet it is created.

dnl · 13 June 2021 06:54

Could someone else who is using QoS please post the output of ip a from a shell on their IPFire server?

bbitsch · 13 June 2021 08:17

Thats’s my output:

 ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: green0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 1000
    link/ether 00:0d:b9:12:d0:30 brd ff:ff:ff:ff:ff:ff
    inet 192.168.10.254/24 scope global green0
       valid_lft forever preferred_lft forever
3: red0: <BROADCAST,UP,LOWER_UP> mtu 1500 qdisc htb state UNKNOWN group default qlen 1000
    link/ether yy:zz:aa:12:d0:31 brd ff:ff:ff:ff:ff:ff
    inet xx.xx.xx.xx22 brd 255.255.255.255 scope global dynamic noprefixroute red0
       valid_lft 4651sec preferred_lft 3751sec
4: blue0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:80:48:6d:22:ff brd ff:ff:ff:ff:ff:ff
    inet 192.168.20.1/24 scope global blue0
       valid_lft forever preferred_lft forever
5: ifb0: <BROADCAST,NOARP> mtu 1500 qdisc fq_codel state DOWN group default qlen 32
    link/ether c2:09:63:04:bd:a4 brd ff:ff:ff:ff:ff:ff
6: ifb1: <BROADCAST,NOARP> mtu 1500 qdisc fq_codel state DOWN group default qlen 32
    link/ether 7e:15:e5:ff:a6:8a brd ff:ff:ff:ff:ff:ff
7: imq0: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc htb state UNKNOWN group default qlen 32
    link/ether 6e:eb:ad:5a:d2:f0 brd ff:ff:ff:ff:ff:ff

dnl · 13 June 2021 10:11

Thank you @bbitsch !
You might want to edit your post and change your red0 IP address to xxx.xxx.xxx.xxx for privacy reasons.

I’ve been doing more testing and when I manually set the qdisc to fq_codel on red0 and imq0 I seem to have more bufferbloat than when I leave it with IPFire’s default. That is, when it reports the error I started this thread with and actually sets a qdisc of htb not fq_codel.

There are three possibilities:

@bbitsch and I have a random failure and everyone else using QoS has fq_codel set and working rrdtool graphs.
There is more to setting the QoS algorithm that I don’t understand (likely!) and this is in fact using the correct algorithm and fq_codel is actually used somewhere else “under the hood”
There is a bug in IPFire where the queuing discipline (qdisc) is not being set to the one which was advertised (fq_codel). The error message I have discussed here is the first indication of this problem which has been around some time.

bbitsch · 13 June 2021 10:25

Thx. It’s never good to do things to fast.

bbitsch · 13 June 2021 10:29

I did some testing also.
I removed (most) of redirection of errors to /dev/null in generating qos script and starting and stopping.
There were no errors reported. Strange.

dnl · 13 June 2021 11:49

What output do you get if you run:
/sbin/tc qdisc add root dev imq0 fq_codel
in a shell?

bbitsch · 13 June 2021 12:55

RTNETLINK answers: File exists

dnl · 14 June 2021 10:09

Thank you. That’s the same result I have.

The other way to confirm that fq_codel isn’t used is to run
/sbin/tc qdisc list root dev imq0
/sbin/tc qdisc list root dev red0
and then note what comes after qdisc.