Summary
nhrpd crashes with SIGSEGV after a GRE tunnel that uses source-interface is deleted and another such tunnel is later created on the same interface. watchfrr restarts nhrpd. No protocols nhrp configuration is needed, because nhrpd tracks every GRE interface.
This is an FRR bug, a use-after-free in nhrpd, and not a bug in VyOS's tunnel or NHRP scripts. Upstream:
- FRR issue: https://github.com/FRRouting/frr/issues/21006 (root cause 1)
- FRR PR with a fix, open and not merged: https://github.com/FRRouting/frr/pull/21007
Affected version
- VyOS rolling with FRR 10.7.1 (stable/10.7 at 9d7b378b80)
- The same code is still present on FRR master
How to reproduce
smoketest/scripts/cli/test_interfaces_tunnel.py triggers it. The test_multiple_gre_tunnel_* cases repeatedly create and delete tun10/tun20 with source-interface and no source-address. One run of the test crashed nhrpd 4 times.
Minimal CLI sequence, based on the smoketest. It hasn't been run on its own yet:
set interfaces dummy dum0 set interfaces tunnel tun10 encapsulation gre set interfaces tunnel tun10 source-interface dum0 set interfaces tunnel tun10 parameters ip key 20 commit delete interfaces tunnel tun10 commit set interfaces tunnel tun10 encapsulation gre set interfaces tunnel tun10 source-interface dum0 set interfaces tunnel tun10 parameters ip key 20 commit
Whether the second commit crashes nhrpd depends on how the freed memory has been reused. Repeating the delete and re-create a few times, or using two tunnels on the same source-interface, hits it reliably.
Check with:
sudo journalctl -b -t watchfrr | grep nhrpd
sudo find /var/tmp/frr -path '*nhrpd*' -name crashlog -exec cat {} \;Actual behavior
watchfrr: nhrpd state -> down : read returned EOF watchfrr: Forked background command: /usr/lib/frr/watchfrr.sh restart nhrpd NHRP: Received signal 11 at 1794770258 (si_addr 0x8, PC 0x55a182c4d064); aborting... NHRP: /usr/lib/frr/nhrpd(nhrp_interface_update_nbma+0x104) [0x55a182c4d064] NHRP: /usr/lib/frr/nhrpd(nhrp_gre_update+0x1a5) [0x55a182c54755] NHRP: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(event_call+0xaa) [0x7fe5e8fa9fba] NHRP: in thread zclient_read scheduled from ../lib/zclient.c:5022 zclient_event()
Some crashes fault at si_addr 0x0 with no backtrace. Any DMVPN running on the box loses its NHRP state on every crash.
Expected behavior
Deleting and re-creating tunnel interfaces doesn't crash nhrpd.
Root cause
When a GRE tunnel has a link device but no local address, nhrpd binds the tunnel to that device. The binding adds a notifier that is embedded in the tunnel's struct nhrp_interface to the device's notifier list. nhrp_if_delete_hook() frees struct nhrp_interface without unlinking that notifier, so:
- a deleted tunnel leaves a freed entry in the underlay's list, and the next tunnel binding to it crashes in notifier_add() (nhrpd/nhrp_interface.c:240)
- a deleted underlay leaves every bound tunnel pointing at freed memory