Page MenuHomeVyOS Platform
Create Task
Maniphest T5489

Change to BBR as TCP congestion control, or at least make it an config option
Closed, ResolvedPublicFEATURE REQUEST
Actions

Description

VyOS 1.4-rolling is currently using default TCP congestion control:

vyos@vyos:~$ sysctl net.core.default_qdisc
net.core.default_qdisc = pfifo_fast
vyos@vyos:~$ sysctl net.ipv4.tcp_congestion_control
net.ipv4.tcp_congestion_control = cubic

Which equals to:

sysctl -w net.core.default_qdisc=pfifo_fast
sysctl -w net.ipv4.tcp_congestion_control=cubic

There is a new cool kid in town named BBR (bottleneck bandwidth and round-trip propagation time) which seems to work better when congestion occurs compared to the legacy default methods.

I would like to propose to change the TCP congestion control in VyOS to use BBR:

sysctl -w net.core.default_qdisc=fq
sysctl -w net.ipv4.tcp_congestion_control=bbr

And if thats not possible then at least make it available as an config option, something like:

set system option tcp-congestion-control bbr

Available methods:

vyos@vyos:~$ sysctl net.ipv4.tcp_available_congestion_control 
net.ipv4.tcp_available_congestion_control = reno cubic bbr

NOTE! It seems like BBR isnt compiled in the current linux kernel (6.1.45) used by VyOS, but its available as a module:

vyos@vyos:~$ cat /proc/sys/net/ipv4/tcp_available_congestion_control 
reno cubic
vyos@vyos:~$ grep -i bbr /boot/config-6.1.45-amd64-vyos 
CONFIG_TCP_CONG_BBR=m

Workaround is to put this in /config/scripts/vyos-preconfig-bootup.script:

/sbin/modprobe tcp_bbr

Along with following in config:

set system sysctl parameter net.core.default_qdisc value 'fq'
set system sysctl parameter net.ipv4.tcp_congestion_control value 'bbr'

According to https://github.com/google/bbr/blob/master/Documentation/bbr-quick-start.md there is no longer a strict requirement to install the "fq" qdisc to use BBR but "fq" is claimed to perform better during high load.

Ref:

https://cloud.google.com/blog/products/networking/tcp-bbr-congestion-control-comes-to-gcp-your-internet-just-got-faster

https://blog.apnic.net/2017/05/09/bbr-new-kid-tcp-block/

https://www.cyberciti.biz/cloud-computing/increase-your-linux-server-internet-speed-with-tcp-bbr-congestion-control/

https://wiki.defect.ch/os/linux/kernel-tuning

https://www.kernel.org/doc/html/v6.1/admin-guide/sysctl/net.html#default-qdisc

https://www.kernel.org/doc/html/v6.1/networking/ip-sysctl.html?highlight=tcp_congestion_control

Details

Difficulty level
Unknown (require assessment)
Version
-
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Feature (new functionality)

Related Objects

Event Timeline

Apachez created this task.Aug 17 2023, 5:50 PM
Apachez updated the task description. (Show Details)Aug 17 2023, 5:52 PM
pasik added a subscriber: pasik.Aug 17 2023, 7:35 PM
Viacheslav added a subscriber: Viacheslav.Sep 5 2023, 1:43 PM

PR https://github.com/vyos/vyos-1x/pull/2205

Viacheslav changed the task status from Open to In progress.Sep 5 2023, 1:43 PM
Viacheslav claimed this task.
Apachez added a comment.Sep 5 2023, 5:24 PM

Note that regarding qdisc it seems that qdisc=fq_codel is nowadays supported according to:

https://groups.google.com/g/bbr-dev/c/4jL4ropdOV8

For kernels v4.20 and later, BBR will use the Linux TCP-layer pacing if the connection notices that there is no qdisc on the sending host implementing pacing. So one reasonable configuration for v4.20 and later is BBR for congestion control and the fq_codel qdisc for queuing (though hopefully BBR would not typically create queues long enough for fq_codel to start dropping...).

However according to the current sourcecode:

https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/net/ipv4/tcp_bbr.c?h=v6.1.51

* NOTE: BBR might be used with the fq qdisc ("man tc-fq") with pacing enabled,
* otherwise TCP stack falls back to an internal pacing using one high
* resolution timer per TCP socket and may use more resources.

https://www.bufferbloat.net/projects/codel/wiki/ claims that:

All modern linux distros now ship with fq_codel.

For servers with tcp-only workloads, particularly at 10GigE speeds, for queue management, we recomend sch_fq instead of fq_codel.

Either qdisc can be enabled by default via a single sysctl option in /etc/sysctl.conf:

net.core.default_qdisc = fq_codel - best general purpose qdisc

net.core.default_qdisc = fq - for fat servers, fq_codel for routers.

Note that in a virtualized environment the underlying server IS a router, the guest VMs are hosts and we don’t have a lot of data regarding either qdiscs’ performance at 10gigE speeds on vms - and early data on fq shows fq_codel’s lowered quantums of benefit at 1GigE and below. We certainly expect fq to continue to improve on hosts and servers… and we expect fq_codel to improve also.

I would prefer that the default will be as PR2205 that is using qdisc=fq.

One could manually override this with:

set system sysctl parameter net.core.default_qdisc value 'fq_codel'

Following commands can be used to verify which qdisc is configured:

tc -d qdisc
tc -s qdisc

Also note that according to https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/configuring_and_managing_networking/linux-traffic-control_configuring-and-managing-networking changing the sysctl isnt enough, one must also reload the network driver:

4 - To apply the changes, reload the network driver:

# modprobe -r NETWORKDRIVERNAME
# modprobe NETWORKDRIVERNAME

5 - Start the network interface:

# ip link set enp0s1 up
Viacheslav added a comment.Sep 5 2023, 5:48 PM

To check if it works just use ss -tin, for example:

vyos@r14:~$ ss -tin
State                    Recv-Q                    Send-Q                                        Local Address:Port                                        Peer Address:Port                     Process                                                                                                                                                                                          
ESTAB                    0                         0                                            192.168.122.14:22                                         192.168.122.1:45446                    
	 bbr wscale:7,7 rto:201 rtt:0.103/0.048 ato:40 mss:1448 pmtu:1500 rcvmss:1448 advmss:1448 cwnd:140 ssthresh:138 bytes_sent:126383 bytes_acked:126383 bytes_received:20969 segs_out:2460 segs_in:2592 data_segs_out:2445 data_segs_in:341 bbr:(bw:503652048bps,mrtt:0.018,pacing_gain:1.25,cwnd_gain:2) send 15745242718bps lastsnd:3003449 lastrcv:3005252 lastack:3003448 pacing_rate 623269416bps delivery_rate 503652168bps delivered:2446 app_limited busy:2029ms rcv_space:14600 rcv_ssthresh:64076 minrtt:0.018 snd_wnd:64128
ESTAB                    0                         36                                           192.168.122.14:22                                         192.168.122.1:51320                    
	 bbr wscale:7,7 rto:201 rtt:0.427/0.555 ato:40 mss:1448 pmtu:1500 rcvmss:1448 advmss:1448 cwnd:148 ssthresh:136 bytes_sent:1779503 bytes_retrans:68 bytes_acked:1779399 bytes_received:167865 segs_out:12061 segs_in:12088 data_segs_out:11899 data_segs_in:3996 bbr:(bw:530564808bps,mrtt:0.068,pacing_gain:1.25,cwnd_gain:2) send 4015063232bps lastsnd:30 lastrcv:31 lastack:31 pacing_rate 525259160bps delivery_rate 458772272bps delivered:11899 app_limited busy:27667ms unacked:1 retrans:0/1 dsack_dups:1 rcv_rtt:392993 rcv_space:65229 rcv_ssthresh:84348 minrtt:0.024 snd_wnd:1344128
vyos@r14:~$
Viacheslav changed the task status from In progress to Needs testing.Sep 6 2023, 6:26 PM
Apachez closed this task as Resolved.Sep 7 2023, 6:30 PM

Using VyOS 1.4-rolling-202309070021.

Looks good to me:

root@vyos:/home/vyos# ss -tin
State              Recv-Q               Send-Q                                 Local Address:Port                             Peer Address:Port               Process              
ESTAB              0                    0                                  192.168.56.2%MGMT:22                               192.168.56.1:57050              
	 bbr wscale:7,9 rto:203 rtt:2.657/4.322 ato:44 mss:1448 pmtu:1500 rcvmss:1448 advmss:1448 cwnd:90 bytes_sent:7267 bytes_acked:7267 bytes_received:5785 segs_out:87 segs_in:104 data_segs_out:80 data_segs_in:43 bbr:(bw:80444136bps,mrtt:0.178,pacing_gain:2.88672,cwnd_gain:2.88672) send 392382386bps lastsnd:39 lastrcv:41 lastack:39 pacing_rate 1253989424bps delivery_rate 74256408bps delivered:81 app_limited busy:285ms rcv_space:14600 rcv_ssthresh:64076 minrtt:0.178 snd_wnd:64128
root@vyos:/home/vyos# tc -d qdisc
qdisc noqueue 0: dev lo root refcnt 2 
qdisc fq 0: dev eth0 root refcnt 2 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop 
qdisc fq 0: dev eth1 root refcnt 2 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop 
qdisc fq 0: dev eth2 root refcnt 2 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop 
qdisc fq 0: dev eth3 root refcnt 2 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop 
qdisc noqueue 0: dev INTERNET root refcnt 2 
qdisc noqueue 0: dev MGMT root refcnt 2
root@vyos:/home/vyos# tc -s qdisc
qdisc noqueue 0: dev lo root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc fq 0: dev eth0 root refcnt 2 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop 
 Sent 20348 bytes 131 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  flows 5 (inactive 4 throttled 0)
  gc 0 highprio 6 throttled 0
qdisc fq 0: dev eth1 root refcnt 2 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop 
 Sent 1360 bytes 14 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  flows 7 (inactive 6 throttled 0)
  gc 0 highprio 6 throttled 0
qdisc fq 0: dev eth2 root refcnt 2 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop 
 Sent 826 bytes 7 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  flows 1 (inactive 0 throttled 0)
  gc 0 highprio 6 throttled 0
qdisc fq 0: dev eth3 root refcnt 2 limit 10000p flow_limit 100p buckets 1024 orphan_mask 1023 quantum 3028b initial_quantum 15140b low_rate_threshold 550Kbit refill_delay 40ms timer_slack 10us horizon 10s horizon_drop 
 Sent 826 bytes 7 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
  flows 1 (inactive 0 throttled 0)
  gc 0 highprio 6 throttled 0
qdisc noqueue 0: dev INTERNET root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0
qdisc noqueue 0: dev MGMT root refcnt 2 
 Sent 0 bytes 0 pkt (dropped 0, overlimits 0 requeues 0) 
 backlog 0b 0p requeues 0

Also verified that downloads works.

So this can be set to resolved.

Viacheslav added a project: VyOS 1.5 Circinus.Sep 8 2023, 11:50 AM
Viacheslav moved this task from Need Triage to Backport Candidates on the VyOS 1.5 Circinus board.
Viacheslav added a comment.Sep 8 2023, 12:10 PM

PR for 1.4 https://github.com/vyos/vyos-1x/pull/2224

Viacheslav moved this task from Backport Candidates to Finished on the VyOS 1.5 Circinus board.Sep 9 2023, 11:37 AM
Viacheslav moved this task from Need Triage to Finished on the VyOS 1.4 Sagitta board.
rea13 added a subscriber: rea13.Sep 19 2023, 8:09 PM
Apachez added a comment.Oct 8 2023, 5:43 PM

PR created: https://github.com/vyos/vyos-1x/pull/2349

Apachez reopened this task as Open.Oct 8 2023, 5:59 PM
Apachez claimed this task.Oct 8 2023, 8:34 PM
Apachez closed this task as Resolved.Oct 10 2023, 5:03 AM

Verified in VyOS 1.5-rolling-202310090023:

fq_codel properly added as default qdisc:

vyos@vyos:~$ sudo sysctl -a | grep -i net.core.default_qdisc
net.core.default_qdisc = fq_codel
vyos@vyos:~$ sudo sysctl -a | grep -i net.ipv4.tcp_congestion_control
net.ipv4.tcp_congestion_control = bbr

Also verified by looking in the file /etc/sysctl.d/30-vyos-router.conf.

Task can be set to resolved.

Viacheslav mentioned this in T2229: PPPOE Default Queue type selection.Dec 21 2023, 8:25 AM