After a netlink overflow (nl/nl: Error polling netlink socket), linux-cp closes its socket, waits 2 s and resyncs from kernel dumps. A VLAN created before that resync and brought up just after the link dump ends up admin-down in VPP and DOWN in the kernel, without its IPv6 link-local, and stays so. On vyos this is hit when a bond is rebuilt (hash-policy, MAC or mode change) with a full table routed over the bond: in some trials one VIF stays stuck, with T8913 and T9243 applied.
Root cause (FDio 744d3c7 + vyos-vpp-patches, src/plugins/linux-cp/)
- During the 2 s wait the socket is closed (lcp_nl.c:645, reopened :680). vyos re-creates the bond's LCP pair then; lcp_nl_pair_add_cb (:813-818) → lcp_nl_drain_messages (:792-799) fails with -NLE_BAD_SOCK and silently signals NL_EVENT_ERR.
- The first wait after the link dump takes that event (:720-729): the resync aborts and the queued messages are discarded, including a kernel UP sent after the dump. Signature: a second Wait before next synchronization attempt within 1 ms of Processed N dump replies of type link, no End synchronization of interface configurations, no error line.
- The VLAN was imported while DOWN. vyos's MTU set calls lcp_itf_mtu_change (lcp_interface_sync.c:249-289) → lcp_itf_pair_sync_state (:59-149), which copies VPP's admin-down to the kernel. The next resync keeps it.
- A second path needs no error event: any VPP→kernel sync_state between the link dump and the processing of the queue pushes the stale state. A large route dump makes this window seconds long.
Reproduction
With a full table (the failure is intermittent):
- vyos rolling with VPP; interfaces vpp bonding vppbond0 (802.3ad, two members) with several VLANs, each with IPv4 and IPv6 addresses; net.core.rmem_max at its default.
- iBGP over one of those VLANs, receiving a full table (about 1M IPv4 and 250k IPv6 prefixes) installed by zebra.
- Change a bond setting that rebuilds the bond and commit, e.g. set interfaces vpp bonding vppbond0 hash-policy layer2+3 (or a MAC or mode change); revert and repeat.
- After each commit check the VLANs: ip link show vppbond0.<vif> and vppctl show interface. A stuck VLAN is <BROADCAST,MULTICAST> state DOWN in the kernel, admin down in VPP, and has no fe80. The journal shows nl/nl: Error polling netlink socket about 1-2 s after the commit; with linux-cp/nl debug logging the signature above.
Some rebuilds leave one VLAN stuck. With 100k IPv6 routes the overflow still happened but no VLAN got stuck.
Deterministic, without routes: Any vyos with VPP and an LCP-paired bond (vppbond0 ↔ BondEthernet0); VLAN 3999 is not in the vyos config. vppctl lcp resync starts the same close / 2 s wait / dump sequence as an overflow. Run as root from a script (timing to ~10 ms):
vppctl set logging class linux-cp level debug syslog-level debug vppctl set logging class nl level debug syslog-level debug vppctl lcp resync & # t0; link dump at about t0+2.0 s sleep 1.0 # t0+1.0: a pair add during the wait (raises the stale error) vppctl create loopback interface instance 3999 vppctl lcp create loop3999 host-if lcptest0 sleep 0.9 # t0+1.9: VLAN created, still DOWN ip link add link vppbond0 name vppbond0.3999 type vlan id 3999 sleep 0.2 # t0+2.1: UP 100 ms after the dump ip link set vppbond0.3999 up sleep 0.1 # t0+2.2: VPP-side MTU change (what vyos does on a new VIF) vppctl set interface mtu packet 1496 BondEthernet0.3999 sleep 10 ip link show vppbond0.3999 # stuck: <BROADCAST,MULTICAST> state DOWN vppctl show interface BondEthernet0.3999 # stuck: admin down
Stuck every time on rolling, on rolling with T9243, and on 1.5.1.
Fix directions
Both parts are needed; stopping the spurious error alone leaves the second path open.
- No spurious error from a pair add while the socket is closed: in lcp_nl_pair_add_cb, skip the drain when there is no socket.
- No stale VPP→kernel state push during a resync, either way:
- defer lcp_itf_pair_sync_state while a resync runs and replay it once the queued messages are applied (VPP-side changes reach the kernel later during a resync), or
- make the MTU-change callback sync only the MTU, not the admin state (narrower; MTU changes then no longer re-add IPv6 addresses, and other VPP-side admin changes during a resync are not covered).
Local patches for 1 + either variant of 2 removed the stuck state in the deterministic tests without affecting live bond rebuilds.