Page MenuHomeVyOS Platform

nhrpd crashes (SIGSEGV) when GRE tunnels bound to a source-interface are deleted and re-created
In progress, HighPublicBUG

Description

Summary

nhrpd crashes with SIGSEGV after a GRE tunnel that uses source-interface is deleted and another such tunnel is later created on the same interface. watchfrr restarts nhrpd. No protocols nhrp configuration is needed, because nhrpd tracks every GRE interface.

This is an FRR bug, a use-after-free in nhrpd, and not a bug in VyOS's tunnel or NHRP scripts. Upstream:

Affected version

  • VyOS rolling with FRR 10.7.1 (stable/10.7 at 9d7b378b80)
  • The same code is still present on FRR master

How to reproduce

smoketest/scripts/cli/test_interfaces_tunnel.py triggers it. The test_multiple_gre_tunnel_* cases repeatedly create and delete tun10/tun20 with source-interface and no source-address. One run of the test crashed nhrpd 4 times.

Minimal CLI sequence, based on the smoketest. It hasn't been run on its own yet:

set interfaces dummy dum0
set interfaces tunnel tun10 encapsulation gre
set interfaces tunnel tun10 source-interface dum0
set interfaces tunnel tun10 parameters ip key 20
commit
delete interfaces tunnel tun10
commit
set interfaces tunnel tun10 encapsulation gre
set interfaces tunnel tun10 source-interface dum0
set interfaces tunnel tun10 parameters ip key 20
commit

Whether the second commit crashes nhrpd depends on how the freed memory has been reused. Repeating the delete and re-create a few times, or using two tunnels on the same source-interface, hits it reliably.

Check with:

sudo journalctl -b -t watchfrr | grep nhrpd
sudo find /var/tmp/frr -path '*nhrpd*' -name crashlog -exec cat {} \;

Actual behavior

watchfrr: nhrpd state -> down : read returned EOF
watchfrr: Forked background command: /usr/lib/frr/watchfrr.sh restart nhrpd

NHRP: Received signal 11 at 1794770258 (si_addr 0x8, PC 0x55a182c4d064); aborting...
NHRP: /usr/lib/frr/nhrpd(nhrp_interface_update_nbma+0x104) [0x55a182c4d064]
NHRP: /usr/lib/frr/nhrpd(nhrp_gre_update+0x1a5) [0x55a182c54755]
NHRP: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(event_call+0xaa) [0x7fe5e8fa9fba]
NHRP: in thread zclient_read scheduled from ../lib/zclient.c:5022 zclient_event()

Some crashes fault at si_addr 0x0 with no backtrace. Any DMVPN running on the box loses its NHRP state on every crash.

Expected behavior

Deleting and re-creating tunnel interfaces doesn't crash nhrpd.

Root cause

When a GRE tunnel has a link device but no local address, nhrpd binds the tunnel to that device. The binding adds a notifier that is embedded in the tunnel's struct nhrp_interface to the device's notifier list. nhrp_if_delete_hook() frees struct nhrp_interface without unlinking that notifier, so:

  • a deleted tunnel leaves a freed entry in the underlay's list, and the next tunnel binding to it crashes in notifier_add() (nhrpd/nhrp_interface.c:240)
  • a deleted underlay leaves every bound tunnel pointing at freed memory

Details

Version
rolling, scutum
Is it a breaking change?
Perfectly compatible
Issue type
Bug (incorrect behavior)

Event Timeline

c-po changed the task status from Open to In progress.
c-po claimed this task.
c-po triaged this task as High priority.