Page MenuHomeVyOS Platform

FRR: `router bgp <asn>` re-leaks all VRF paths, commits take minutes with a full table in a VRF
Open, NormalPublicBUG

Description

Description

Component: FRR / BGP / VRF. Affects 1.5.1-S1 (FRR 10.5.5) and rolling (FRR stable/10.7, same code).

With a full table in a VRF (about 1.08M IPv4 + 253k IPv6 routes) and import vrf between the default VRF and the VRF, every router bgp <asn> on the default instance costs about 40 s of bgpd CPU. frr-reload.py enters the default instance once per changed line, so a commit with 11 changed BGP blocks takes about 8 minutes. During that time vtysh and the commit are blocked.

Cause (FRR bgpd/bgp_vty.c): router_bgp() calls vpn_leak_postchange_all() on every entry into the default instance, which re-leaks every path in both directions even though nothing changed.

Upstream issue: https://github.com/FRRouting/frr/issues/24488
Upstream fix: https://github.com/FRRouting/frr/pull/24489

Lab result with the fix on 10.5.5: 1.0 - 1.4 s -> 0.02 - 0.03 s at 200k routes. FRR topotests for VRF leak, L3VPN, EVPN and SRv6 pass.

Workaround until fixed: none in the config. Commit time scales with the number of BGP blocks that frr-reload.py has to change.

Plan: carry the upstream fix as a patch in vyos-build (packages/frr patches) until FRR has it in a release the VyOS build uses.

Details

Version
1.5.1
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Event Timeline

Sorry it was a long night...
Rolling already carries the upstream fix as vyos-build patch 0007-bgpd-Remove-extra-leak-processing-on-router-bgp.patch (T9451, FRRouting/frr#24297, b7f9a22c56).

1.5.x (FRR 10.5.5) does not have it. Request: please backport this patch to 1.5.x.

Impact on 1.5.x: router bgp <asn> on the default instance re-leaks all VRF paths on every entry. With a full table in a VRF (about 1.08M IPv4 + 253k IPv6 routes, import vrf between the default VRF and the VRF) it takes about 40 s of bgpd CPU per entry. frr-reload.py enters the default instance once per changed line, so a commit with 11
changed BGP blocks takes about 8 minutes.

Tested on FRR 10.5.5 with the upstream commit applied (lab, 200k routes in a VRF, import vrf in both directions): router bgp <asn> 0.02 s instead of 1.0 - 1.4 s, router bgp <asn> vrf <name> 0.04 s, leaking complete after restart (200008 routes).
Upstream backport request to FRR stable/10.5 - 10.7: https://github.com/FRRouting/frr/issues/24488

Patch backported to 10.5.5

Viacheslav triaged this task as Normal priority.Thu, Nov 19, 1:58 PM