Page MenuHomeVyOS Platform

VRRP over VPP: the new VIP stops answering ARP after failover
Open, NormalPublicBUG

Description

After a VRRP failover, the new master's VIP stops answering ARP, not because of anything wrong
with the election, but because VPP's own FIB keeps
a stale neighbor entry from before the failover and lets it win over the now-local VIP address.

Topology

Pair, eth1 VPP-bound on both nodes, VRRP1 vrid 1, VIP 192.0.2.100/24, priority 200 (r1) / 100 (r2),
hello-source-address set on both

Steps to reproduce

On r1:

set vpp settings interface eth1
set interfaces ethernet eth1 address '192.0.2.1/24'
commit

On r2:

set vpp settings interface eth1
set interfaces ethernet eth1 address '192.0.2.2/24'
commit

Configure the VRRP group

On r1:

set high-availability vrrp group VRRP1 interface 'eth1'
set high-availability vrrp group VRRP1 vrid '1'
set high-availability vrrp group VRRP1 priority '200'
set high-availability vrrp group VRRP1 address '192.0.2.100/24'
set high-availability vrrp group VRRP1 hello-source-address '192.0.2.1'
commit

On r2:

set high-availability vrrp group VRRP1 interface 'eth1'
set high-availability vrrp group VRRP1 vrid '1'
set high-availability vrrp group VRRP1 priority '100'
set high-availability vrrp group VRRP1 address '192.0.2.100/24'
set high-availability vrrp group VRRP1 hello-source-address '192.0.2.2'
commit

r1 (priority 200) should show VRRP1 eth1 1 MASTER, r2 VRRP1 eth1 1 BACKUP

On r2:

ping 192.0.2.100 count 5

Expect 0% packet loss.

Then:

ip neigh show 192.0.2.100 dev eth1

Expect 192.0.2.100 lladdr <r1 MAC> REACHABLE r2 now has r1's MAC cached for the VIP.

trigger a failover

set high-availability vrrp group VRRP1 disable
commit

confirm r2 took over

show vrrp

Expect VRRP1 eth1 1 MASTER

check the stale neighbor survived the transition.** On r2 (now master):

vppctl show ip neighbors

Expect to still see:

192.0.2.100   D   52:54:00:01:01:00   eth1

try to reach the VIP from the old master.

ping 192.0.2.100

Expected

Once the VIP is local to the new master, ARP requests for it should be answered normally, the
election is healthy, VIP should be fully reachable.

Actual

PING 192.0.2.100 ...
From 192.0.2.1 icmp_seq=1 Destination Host Unreachable
From 192.0.2.1 icmp_seq=2 Destination Host Unreachable
From 192.0.2.1 icmp_seq=3 Destination Host Unreachable
3 packets transmitted, 0 received, +3 errors, 100% packet loss

ip neigh show 192.0.2.100 dev eth1 on the old master: 192.0.2.100 FAILED.

show interfaces
eth1         192.0.2.2/24       52:54:00:02:01:00  default   1500  u/u
             192.0.2.100/24

Confirms the VIP is correctly assigned to r2

vppctl show ip neighbors
    Age                       IP                    Flags      Ethernet              Interface
226049.7132                192.0.2.1                  D    52:54:00:01:01:00 eth1
226018.1804               192.0.2.100                 D    52:54:00:01:01:00 eth1
vppctl show ip fib 192.0.2.100/32
192.0.2.100/32 fib:0 index:15 locks:3
  interface refs:1 entry-flags:connected,local, src-flags:added,contributing,active, cover:9
    path-list:[29] locks:2 flags:local, uPRF-list:21 len:0 itfs:[]
      path:[37] pl-index:29 ip4 weight=1 pref=0 receive:  oper-flags:resolved, cfg-flags:local,
        [@0]: dpo-receive: 192.0.2.100 on eth1

  adjacency refs:1 entry-flags:attached, src-flags:added, cover:-1
    path-list:[26] locks:1 uPRF-list:18 len:1 itfs:[1, ]
      path:[29] pl-index:26 ip4 weight=1 pref=0 attached-nexthop:  oper-flags:resolved,
        192.0.2.100 eth1
      [@0]: ipv4 via 192.0.2.100 eth1: mtu:1500 next:5 flags:[] 5254000101005254000201000800
    Extensions:
     path:29 adj-flags:[refines-cover]

 forwarding:   unicast-ip4-chain
  [@0]: dpo-load-balance: [proto:ip4 index:16 buckets:1 uRPF:21 to:[0:0]]
    [0] [@12]: dpo-receive: 192.0.2.100 on eth1

Both sources are present at once for the same /32:

  • interface (local/receive) installed because r2 is now master and the VIP is its own address.
  • adjacency the stale entry pointing at r1's MAC, left over from steps before

Details

Version
VyOS 1.5-rolling-202610191432
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Event Timeline

I could reproduce this on a lab VM pair (rolling 2026.10.16-0024, VPP bound to eth0 on the backup node, the master node without VPP, VRRP1 vrid 1, VIP 198.51.100.100/24, priorities 200 / 100).

What happens, step by step:

  1. The VPP node is BACKUP and pings the VIP. Linux learns 198.51.100.100 lladdr <master MAC> REACHABLE, and linux-cp (lcp-sync on) copies it into VPP: vppctl show ip neighbors has 198.51.100.100 D <master MAC> eth0.
  2. The master is disabled, the VPP node becomes MASTER and keepalived adds the VIP to eth0. The Linux neighbor entry stays (it only turns STALE, the kernel does not remove a neighbor entry when the same address becomes a local address). So the VPP entry stays too.
  3. vppctl show ip fib table 0 198.51.100.100/32 now has both sources, receive from the interface address and the adjacency with the old master MAC, as in the report. The ping from the old master fails with Destination Host Unreachable, its neighbor entry is FAILED.

What fixes it, each tested on its own:

  • Deleting only the stale VPP neighbor (vppctl set ip neighbor del eth0 198.51.100.100 <master MAC>): the old master resolves the VIP to the new MAC right away, ping 0% loss, and the entry does not come back.
  • Deleting the stale entry on the Linux side on the new master (ip neigh del 198.51.100.100 dev eth0): linux-cp removes the VPP entry too, ping 0% loss. So the deletion is synchronised, only nobody deletes the Linux entry when the address becomes local.

So it is not specific to VRRP, it applies to any address that is first a neighbor and later becomes a local address on a VPP bound interface. A simple fix on the VyOS side would be to delete the neighbor entry of every VIP of the group in the keepalived notify_master handler (vyos-vrrp-conntracksync.sh master <group>), for example ip neigh del <vip> dev <interface>. A more general fix would be in VPP or linux-cp, where the adjacency is dropped when the address becomes local. I have not tested a patch for either, and I have not looked at why VPP does not answer the ARP request with the receive entry present.

Workaround until then: ip neigh del <vip> dev <interface> on the new master after a takeover.

*Two additions, one correction.

Correction. I wrote that a fix could use vyos-vrrp-conntracksync.sh in notify_master. That is not right: this helper is only set for sync groups with conntrack-sync, and it exits when conntrackd is not running. For normal groups the state changes are reported by keepalived-fifo.py (notify_fifo_script), which already runs the transition-script of the group. A fix in VyOS would go there, for example by deleting the neighbor entries of the VIPs of the group when a group on a VPP interface becomes MASTER.

Workaround that works today, tested on the same lab pair: a transition-script for the master state.

set high-availability vrrp group VRRP1 transition-script master '/usr/sbin/ip neigh del 192.0.2.100 dev eth1'

After the takeover the log has INSTANCE VRRP1 changed state to MASTER and Running the command: /usr/sbin/ip neigh del ..., the VPP neighbor entry is gone and the old master reaches the VIP again (0% packet loss). One command is needed per VIP.

Cause. The problem is not specific to VyOS. I reproduced it with the unchanged upstream VPP packages (v25.10.0-23 and v27.02-rc0~167, linux-cp only, no VyOS patches). The trace of the ARP request is arp-input -> linux-cp-arp-phy -> arp-reply -> error-drop. In src/vnet/arp/arp.c, arp_dst_fib_check() returns ARP_DST_FIB_ADJ for every entry that has the adjacency source, even if the entry also has the interface source with the local flag, and arp_input() does not answer a request in that case. I have not tested a patch. I reported it to VPP: https://github.com/FDio/vpp/issues/3756

Update: a fix for the underlying VPP problem is uploaded to Gerrit: https://gerrit.fd.io/r/c/vpp/+/47276 (issue: https://github.com/FDio/vpp/issues/3756). It changes one line in arp_dst_fib_check() and adds a unit test. With a build of VPP master plus this change, the reproducer gets its ARP answer after the takeover (0% packet loss) without any neighbor deletion. The transition-script workaround stays useful until the change is merged and VyOS has a VPP build that contains it.

We can always use the patched VPP version https://github.com/vyos/vyos-vpp-patches/tree/rolling/patches/vpp
Needs to check if we can apply it to our version (stable/2510).

I can test and create a PR

Let's not rush and wait for the VPP maintainers' review.

To late the build test and pr.is.allready running.