Page MenuHomeVyOS Platform

VPP linux-cp: a VLAN that comes up during the netlink overflow resync is pushed back to admin-down
Open, HighPublicBUG

Description

After a netlink overflow (nl/nl: Error polling netlink socket), linux-cp closes its socket, waits 2 s and resyncs from kernel dumps. A VLAN created before that resync and brought up just after the link dump ends up admin-down in VPP and DOWN in the kernel, without its IPv6 link-local, and stays so. On vyos this is hit when a bond is rebuilt (hash-policy, MAC or mode change) with a full table routed over the bond: in some trials one VIF stays stuck, with T8913 and T9243 applied.

Root cause (FDio 744d3c7 + vyos-vpp-patches, src/plugins/linux-cp/)

  1. During the 2 s wait the socket is closed (lcp_nl.c:645, reopened :680). vyos re-creates the bond's LCP pair then; lcp_nl_pair_add_cb (:813-818) → lcp_nl_drain_messages (:792-799) fails with -NLE_BAD_SOCK and silently signals NL_EVENT_ERR.
  2. The first wait after the link dump takes that event (:720-729): the resync aborts and the queued messages are discarded, including a kernel UP sent after the dump. Signature: a second Wait before next synchronization attempt within 1 ms of Processed N dump replies of type link, no End synchronization of interface configurations, no error line.
  3. The VLAN was imported while DOWN. vyos's MTU set calls lcp_itf_mtu_change (lcp_interface_sync.c:249-289) → lcp_itf_pair_sync_state (:59-149), which copies VPP's admin-down to the kernel. The next resync keeps it.
  4. A second path needs no error event: any VPP→kernel sync_state between the link dump and the processing of the queue pushes the stale state. A large route dump makes this window seconds long.

Reproduction
With a full table (the failure is intermittent):

  1. vyos rolling with VPP; interfaces vpp bonding vppbond0 (802.3ad, two members) with several VLANs, each with IPv4 and IPv6 addresses; net.core.rmem_max at its default.
  2. iBGP over one of those VLANs, receiving a full table (about 1M IPv4 and 250k IPv6 prefixes) installed by zebra.
  3. Change a bond setting that rebuilds the bond and commit, e.g. set interfaces vpp bonding vppbond0 hash-policy layer2+3 (or a MAC or mode change); revert and repeat.
  4. After each commit check the VLANs: ip link show vppbond0.<vif> and vppctl show interface. A stuck VLAN is <BROADCAST,MULTICAST> state DOWN in the kernel, admin down in VPP, and has no fe80. The journal shows nl/nl: Error polling netlink socket about 1-2 s after the commit; with linux-cp/nl debug logging the signature above.

Some rebuilds leave one VLAN stuck. With 100k IPv6 routes the overflow still happened but no VLAN got stuck.

Deterministic, without routes: Any vyos with VPP and an LCP-paired bond (vppbond0 ↔ BondEthernet0); VLAN 3999 is not in the vyos config. vppctl lcp resync starts the same close / 2 s wait / dump sequence as an overflow. Run as root from a script (timing to ~10 ms):

vppctl set logging class linux-cp level debug syslog-level debug
vppctl set logging class nl level debug syslog-level debug
vppctl lcp resync &                                        # t0; link dump at about t0+2.0 s
sleep 1.0                                                  # t0+1.0: a pair add during the wait (raises the stale error)
vppctl create loopback interface instance 3999
vppctl lcp create loop3999 host-if lcptest0
sleep 0.9                                                  # t0+1.9: VLAN created, still DOWN
ip link add link vppbond0 name vppbond0.3999 type vlan id 3999
sleep 0.2                                                  # t0+2.1: UP 100 ms after the dump
ip link set vppbond0.3999 up
sleep 0.1                                                  # t0+2.2: VPP-side MTU change (what vyos does on a new VIF)
vppctl set interface mtu packet 1496 BondEthernet0.3999
sleep 10
ip link show vppbond0.3999                                 # stuck: <BROADCAST,MULTICAST> state DOWN
vppctl show interface BondEthernet0.3999                   # stuck: admin down

Stuck every time on rolling, on rolling with T9243, and on 1.5.1.

Fix directions
Both parts are needed; stopping the spurious error alone leaves the second path open.

  1. No spurious error from a pair add while the socket is closed: in lcp_nl_pair_add_cb, skip the drain when there is no socket.
  2. No stale VPP→kernel state push during a resync, either way:
    • defer lcp_itf_pair_sync_state while a resync runs and replay it once the queued messages are applied (VPP-side changes reach the kernel later during a resync), or
    • make the MTU-change callback sync only the MTU, not the admin state (narrower; MTU changes then no longer re-add IPv6 addresses, and other VPP-side admin changes during a resync are not covered).

Local patches for 1 + either variant of 2 removed the stuck state in the deterministic tests without affecting live bond rebuilds.

Details

Version
1.5.1, 1.5-rolling-202610018
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Event Timeline

@a.kudientsov thanks for the detailed report and the reproducer. You wrote that you have local patches for both parts (skipping the drain in lcp_nl_pair_add_cb when
there is no socket, and the state sync during the resync window) that remove the stuck state in your tests.

Since this is a change in the linux-cp plugin, it would go into vyos/vyos-vpp-patches (patches/vpp/), next to the existing LCP patches. Could you send them there as a pull request? If you want, I can test them on a VyOS VM with VPP and report back, using the vppctl lcp resync reproducer from the ticket.

Two questions that might help the review:

  • For the second part you mention two variants (defer the pair state sync during a resync, or sync only the MTU on an MTU change). Do both give the same result in your tests, and do you have a preference?
  • Does the stuck VLAN also happen without the 256 MiB buffer problem from T9484, for example with vppctl lcp resync alone? That would tell us whether T9484 only makes it less likely or is a precondition.

https://github.com/vyos/vyos-vpp-patches/pull/60

@rherold
Same result: no stuck VLAN in any test, including 15 full-table bond rebuilds each. The PR now uses the MTU-only variant: deferring the sync lost MTU changes made in VPP when several resync rounds failed in a row, and the MTU-only change kept them.
Yes: vppctl lcp resync alone, with no routes and the full buffer, left the VLAN stuck every time. T9484 only makes the overflow less likely.