Page MenuHomeVyOS Platform

vyos-netlinkd: stale RTM_NEWLINK DOWN from netlink backlog stops dhclient on an interface that is enabled and up
Closed, ResolvedPublicBUG

Description

Summary

During config commits, vyos-netlinkd logs Config commit in progress, skipping netlink events — but the events are not discarded consistently. After back-to-back commits (disable WAN interfaces, then re-enable them ~20s later, driven by VRRP transition scripts at boot), netlinkd processed a stale RTM_NEWLINK state=DOWN from the earlier disable 45+ seconds after the last commit finished, and stopped dhclient@eth0 — on an interface that was by then admin-enabled in config, link UP, with a freshly started dhclient mid-DISCOVER. The matching link-UP event was apparently swallowed by the skip window, so nothing restarted the client. The DHCP WAN stayed dead ~8 minutes until a manual link bounce/dhclient restart.

Timeline (journal, VRRP preferred-master node, boot)

01:28:17 keepalived-fifo: GROUP SYNC -> BACKUP; transition script disables eth0/eth8/wg0 (commit)
01:28:36 keepalived: SYNC -> MASTER (preferred master preempts 19s later; transition script re-enables)
01:28:36..01:29:25 vyos-netlinkd: "Config commit in progress, skipping netlink events" (continuous, both commits)
01:29:17 systemd: Stopping DHCP client on eth0... (enable transition restarts dhclient)
01:29:18 dhclient[8445]: new client starts, DHCPDISCOVER on eth0
01:29:43 vyos-netlinkd: RTM_NEWLINK -> eth8, state=DOWN  -> Stopping dhclient@eth8.service + dhcp6c@eth8
01:30:14 vyos-netlinkd: RTM_NEWLINK -> eth0, state=DOWN  -> Stopping dhclient@eth0.service   <-- stale; eth0 enabled+up
   (no further netlinkd activity on eth0; WAN dead)
01:38:18 manual recovery: link bounce -> netlinkd sees DOWN then UP -> Restarting dhclient@eth0.service
01:38:20 dhclient re-binds

Why the DOWN at 01:30:14 must be stale

  • The last disable of eth0 was the backup transition commit (~01:28:26). The master transition re-enabled it and dhclient 8445 was DISCOVERing from 01:29:18 onward.
  • At 01:30:14 there was no operator action, no commit in progress, and the link was up — the client was killed by netlinkd's systemctl stop, not by link loss.
  • The DOWN events for eth8 (01:29:43) and eth0 (01:30:14) arrive long after the skip window closes, in the order the disables happened — consistent with a buffered netlink backlog being drained late.

Impact

Any workflow that disables and quickly re-enables a DHCP interface (HA transition scripts around VRRP, revert-style automation commits) can have its dhclient/dhcp6c killed after the re-enable, leaving an interface that the config says is enabled with no DHCP client and no address until manual intervention. On an HA master this is a silent WAN outage with nothing to self-heal it (we added an external watchdog that restarts dhclient after 15 minutes down as a workaround).

Suggested fix

Before acting on a queued RTM_NEWLINK, validate against current state (re-read operstate and/or config intent for the interface) instead of trusting the event payload; or flush/discard the socket backlog when a commit-skip window closes; or make the skip window symmetric (if DOWN events survive the window, UP events must too).

Environment

  • VyOS 1.5 Circinus rolling 2026.07.11-0033-rolling
  • Proxmox VM, virtio NICs
  • keepalived sync-group transition scripts enable/disable WAN interfaces on VRRP state change (backup disables, master enables ~20s apart at boot of the preferred master)

Workaround

External watchdog restarting dhclient@<iface> after sustained down; booting the preferred master with WANs disabled in the saved config to remove the boot-time disable→enable churn.

Details

Version
VyOS 1.5 Circinus rolling 2026.07.11-0033-rolling
Is it a breaking change?
Perfectly compatible
Issue type
Bug (incorrect behavior)