My outbound to tun0 interface has been limited to 330Mbps by carrier company.
I want to keep OSPF/BGP packet when the interface is under heavy load. So I wrote this policy, and it result a commit error.
vyos@bsp-asbr2-cm# show qos policy shaper test { bandwidth 330mbit class 10 { bandwidth 320mbit match ospf { ip { protocol ospf } } queue-type fair-queue } default { bandwidth 300mbit queue-type fair-queue } } [edit] vyos@bsp-asbr2-cm# set qos int tun0 eg test [edit] vyos@bsp-asbr2-cm# commit [ qos ] VyOS had an issue completing a command. Report time: 2023-03-03 10:31:49 Image version: VyOS 1.4-rolling-202302150317 Release train: current Built by: autobuild@vyos.net Built on: Wed 15 Feb 2023 03:17 UTC Build UUID: e62b2d4d-c09c-4dd6-a722-884b782e4d13 Build commit ID: 5207b6f510d677 Architecture: x86_64 Boot via: installed image System type: VMware guest Hardware vendor: VMware, Inc. Hardware model: VMware20,1 Hardware S/N: VMware-56 4d fb 13 43 c4 e5 63-b0 9e 49 a8 b7 33 09 da Hardware UUID: 13fb4d56-c443-63e5-b09e-49a8b73309da Traceback (most recent call last): File "/usr/libexec/vyos/conf_mode/qos.py", line 255, in <module> apply(c) File "/usr/libexec/vyos/conf_mode/qos.py", line 246, in apply tmp.update(shaper_config, direction) File "/usr/lib/python3/dist-packages/vyos/qos/trafficshaper.py", line 100, in update super().update(config, direction) File "/usr/lib/python3/dist-packages/vyos/qos/base.py", line 246, in update self._cmd(filter_cmd) File "/usr/lib/python3/dist-packages/vyos/qos/base.py", line 36, in _cmd return cmd(command) File "/usr/lib/python3/dist-packages/vyos/util.py", line 161, in cmd raise OSError(code, feedback) PermissionError: [Errno 1] failed to run command: tc filter replace dev tun0 parent 1: protocol all u32 match ip protocol ospf 0xff action police rate 320000000 burst 15k flowid 1:a returned: exit code: 1 noteworthy: cmd 'tc qdisc del dev lo parent ffff:' returned (out): returned (err): Error: Cannot find specified qdisc on specified device. cmd 'tc filter replace dev tun0 parent 1: protocol all u32 match ip protocol ospf 0xff action police rate 320000000 burst 15k flowid 1:a' returned (out): returned (err): Illegal "match" [[qos]] failed Commit failed [edit] vyos@bsp-asbr2-cm#
Btw, when I enable a 300mbit/s shaper on tunnel, the actual limit is around 140-150mbit/s, iperf3 said lots of retry.
vyos@bsp-asbr2-cm# show qos interface tun0 { egress test } policy { shaper test { bandwidth 330mbit default { bandwidth 300mbit queue-type fair-queue } } } [edit] vyos@bsp-asbr2-cm#
hikari@mynas:~$ iperf3 -c 198.18.0.10 -t 100 Connecting to host 198.18.0.10, port 5201 [ 5] local 10.96.4.101 port 49502 connected to 198.18.0.10 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 19.5 MBytes 164 Mbits/sec 2459 1.51 MBytes [ 5] 1.00-2.00 sec 16.2 MBytes 136 Mbits/sec 1891 1.16 MBytes [ 5] 2.00-3.00 sec 17.5 MBytes 147 Mbits/sec 660 1.13 MBytes [ 5] 3.00-4.00 sec 17.5 MBytes 147 Mbits/sec 1263 1.22 MBytes [ 5] 4.00-5.00 sec 16.2 MBytes 136 Mbits/sec 1423 1.20 MBytes [ 5] 5.00-6.00 sec 17.5 MBytes 147 Mbits/sec 1561 1.21 MBytes [ 5] 6.00-7.00 sec 17.5 MBytes 147 Mbits/sec 1588 1.25 MBytes [ 5] 7.00-8.00 sec 17.5 MBytes 147 Mbits/sec 1616 1.27 MBytes [ 5] 8.00-9.00 sec 17.5 MBytes 147 Mbits/sec 2112 1.28 MBytes [ 5] 9.00-10.00 sec 17.5 MBytes 147 Mbits/sec 2206 1.31 MBytes [ 5] 10.00-11.00 sec 12.5 MBytes 105 Mbits/sec 1133 1.18 MBytes ^C[ 5] 11.00-11.60 sec 10.0 MBytes 140 Mbits/sec 770 1.28 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-11.60 sec 197 MBytes 142 Mbits/sec 18682 sender [ 5] 0.00-11.60 sec 0.00 Bytes 0.00 bits/sec receiver iperf3: interrupt - the client has terminated hikari@mynas:~$
Iperf3 result when I disabled this qos rule:
hikari@mynas:~$ iperf3 -c 198.18.0.10 -t 100 Connecting to host 198.18.0.10, port 5201 [ 5] local 10.96.4.101 port 49762 connected to 198.18.0.10 port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 35.8 MBytes 300 Mbits/sec 0 4.92 MBytes [ 5] 1.00-2.00 sec 42.5 MBytes 357 Mbits/sec 1 4.92 MBytes [ 5] 2.00-3.00 sec 42.5 MBytes 357 Mbits/sec 0 4.92 MBytes [ 5] 3.00-4.00 sec 38.8 MBytes 325 Mbits/sec 0 4.92 MBytes [ 5] 4.00-5.00 sec 45.0 MBytes 378 Mbits/sec 0 4.92 MBytes [ 5] 5.00-6.00 sec 42.5 MBytes 357 Mbits/sec 0 4.92 MBytes [ 5] 6.00-7.00 sec 42.5 MBytes 357 Mbits/sec 0 4.92 MBytes [ 5] 7.00-8.00 sec 42.5 MBytes 357 Mbits/sec 0 4.92 MBytes [ 5] 8.00-9.00 sec 42.5 MBytes 357 Mbits/sec 0 4.92 MBytes [ 5] 9.00-10.00 sec 42.5 MBytes 357 Mbits/sec 0 4.92 MBytes [ 5] 10.00-11.00 sec 32.5 MBytes 273 Mbits/sec 0 5.14 MBytes ^C[ 5] 11.00-11.30 sec 12.5 MBytes 353 Mbits/sec 0 5.14 MBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-11.30 sec 462 MBytes 343 Mbits/sec 1 sender [ 5] 0.00-11.30 sec 0.00 Bytes 0.00 bits/sec receiver iperf3: interrupt - the client has terminated hikari@mynas:~$
My hardware platform is Supermicro's X10SDV with Intel Xeon D1521 CPU, hypervisor configuration for this instance is 2vCPU+1GB of RAM.
Because the IPsec performance in normal setting is too poor and I changed kernel cmdline to this:
vyos@bsp-asbr2-cm:~$ cat /proc/cmdline BOOT_IMAGE=/boot/1.4-rolling-202302150317/vmlinuz boot=live rootdelay=5 noautologin net.ifnames=0 biosdevname=0 vyos-union=/boot/1.4-rolling-202302150317 console=ttyS0,115200 console=tty0 mitigations=off vyos@bsp-asbr2-cm:~$
Btw, if I manually set shaper rule by hand, it works correctly:
sudo tc qdisc add root dev tun0 handle 1: htb default 2 sudo tc class add dev tun0 parent 1: classid 1:1 htb rate 330mbit sudo tc class add dev tun0 parent 1:1 classid 1:2 htb rate 300mbit sudo tc class add dev tun0 parent 1:1 classid 1:3 htb rate 320mbit