FRRouting/zebra 7.5.1 does not redistribute routes to other protocols
Closed, ResolvedPublicBUG
Actions

Assigned To

Authored By

	zsdc
	Oct 7 2022, 6:13 PM

Description

Details

In some cases zebra does not redistribute connected routes to other protocols. For example, if there is more than one address added to an interface at the same time.

A configuration example to reproduce the problem:

set high-availability vrrp group vrrp1 interface 'eth0'
set high-availability vrrp group vrrp1 virtual-address 192.168.123.1/24
set high-availability vrrp group vrrp1 virtual-address 192.168.123.2/24
set high-availability vrrp group vrrp1 vrid '100'
set interfaces ethernet eth0 address '192.168.122.5/24'
set protocols bgp 65000 address-family ipv4-unicast redistribute connected

Routing tables will look like this:

vyos@vyos:~$ show ip route 
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

C>* 192.168.122.0/24 is directly connected, eth0, 00:02:12
C * 192.168.123.0/24 is directly connected, eth0, 00:00:45
C>* 192.168.123.0/24 is directly connected, eth0, 00:00:45

vyos@vyos:~$ show ip bgp 
BGP table version is 1, local router ID is 192.168.123.2, vrf id 0
Default local pref 100, local AS 65000
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 192.168.122.0/24 0.0.0.0                  0         32768 ?

Displayed  1 routes and 1 total paths

A faster way to reproduce

To reproduce this it is enough to add to config only:

set protocols bgp 65000 address-family ipv4-unicast redistribute connected

And then try these two commands:

sudo ip a add 192.168.123.1/24 dev eth0 && sudo ip a add 192.168.123.2/24 dev eth0
sudo ip a add 192.168.123.1/24 dev eth0 & sudo ip a add 192.168.123.2/24 dev eth0

The first command will work as expected because addresses are added one by one:

vyos@vyos:~$ sudo ip a add 192.168.123.1/24 dev eth0 && sudo ip a add 192.168.123.2/24 dev eth0
vyos@vyos:~$ show ip route 
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

C * 192.168.123.0/24 is directly connected, eth0, 00:00:02
C>* 192.168.123.0/24 is directly connected, eth0, 00:00:02
vyos@vyos:~$ show ip bgp 
BGP table version is 5, local router ID is 192.168.123.2, vrf id 0
Default local pref 100, local AS 65000
Status codes:  s suppressed, d damped, h history, * valid, > best, = multipath,
               i internal, r RIB-failure, S Stale, R Removed
Nexthop codes: @NNN nexthop's vrf id, < announce-nh-self
Origin codes:  i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 192.168.123.0/24 0.0.0.0                  0         32768 ?

Displayed  1 routes and 1 total paths
vyos@vyos:~$ sudo ip a flush dev eth0

But the second one:

vyos@vyos:~$ sudo ip a add 192.168.123.1/24 dev eth0 & sudo ip a add 192.168.123.2/24 dev eth0
[1] 3312
[1]+  Done                    sudo ip a add 192.168.123.1/24 dev eth0
vyos@vyos:~$ show ip route 
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

C * 192.168.123.0/24 is directly connected, eth0, 00:00:03
C>* 192.168.123.0/24 is directly connected, eth0, 00:00:03
vyos@vyos:~$ show ip bgp 
No BGP prefixes displayed, 0 exist
vyos@vyos:~$ sudo ip a flush dev eth0

FRR debug for the normal command is:

Oct 07 18:02:50 vyos zebra[940]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWADDR(20), len=80, seq=1665165771, pid=3362
Oct 07 18:02:50 vyos zebra[940]: netlink_interface_addr RTM_NEWADDR eth0 flags 0x80:
Oct 07 18:02:50 vyos zebra[940]:   IFA_LOCAL     192.168.123.1/24
Oct 07 18:02:50 vyos zebra[940]:   IFA_ADDRESS   192.168.123.1/24
Oct 07 18:02:50 vyos zebra[940]:   IFA_CACHEINFO pref -1, valid -1
Oct 07 18:02:50 vyos zebra[940]: MESSAGE: ZEBRA_INTERFACE_ADDRESS_ADD 192.168.123.1/24 on eth0(0)
Oct 07 18:02:50 vyos zebra[940]: rib_add_multipath_nhe: (0:0):192.168.123.0/24: Inserting route rn 0x560dcb8dfd00, re 0x560dcb8dfc20 (connected) existing (nil)
Oct 07 18:02:50 vyos zebra[940]: rib_add_multipath_nhe: (0:0):192.168.123.0/24 (MRIB): Inserting route rn 0x560dcb8de610, re 0x560dcb8dd6d0 (connected) existing (nil)
Oct 07 18:02:50 vyos zebra[940]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=60, seq=0, pid=0
Oct 07 18:02:50 vyos zebra[940]: RTM_NEWROUTE ipv4 local proto kernel NS 0
Oct 07 18:02:50 vyos zebra[940]: Route rtm_type: local(2) intentionally ignoring
Oct 07 18:02:50 vyos zebra[940]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=60, seq=0, pid=0
Oct 07 18:02:50 vyos zebra[940]: RTM_NEWROUTE ipv4 unicast proto kernel NS 0
Oct 07 18:02:50 vyos zebra[940]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=60, seq=0, pid=0
Oct 07 18:02:50 vyos zebra[940]: RTM_NEWROUTE ipv4 broadcast proto kernel NS 0
Oct 07 18:02:50 vyos zebra[940]: Route rtm_type: broadcast(3) intentionally ignoring
Oct 07 18:02:50 vyos zebra[940]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=60, seq=0, pid=0
Oct 07 18:02:50 vyos zebra[940]: RTM_NEWROUTE ipv4 broadcast proto kernel NS 0
Oct 07 18:02:50 vyos zebra[940]: Route rtm_type: broadcast(3) intentionally ignoring
Oct 07 18:02:50 vyos zebra[940]: default(0:254):192.168.123.0/24: Adding route rn 0x560dcb8dfd00, re 0x560dcb8dfc20 (connected)
Oct 07 18:02:50 vyos zebra[940]: default(0:254):192.168.123.0/24: Adding route rn 0x560dcb8de610, re 0x560dcb8dd6d0 (connected)
Oct 07 18:02:50 vyos zebra[940]: netlink_nexthop_msg_encode: ID (19): directly connected, eth0(2) vrf default(0)
Oct 07 18:02:50 vyos zebra[940]: netlink_nexthop_msg_encode: RTM_NEWNEXTHOP, id=19
Oct 07 18:02:50 vyos zebra[940]: nl_batch_send: netlink-dp (NS 0), batch size=40, msg cnt=1
Oct 07 18:02:50 vyos zebra[940]: default(0:254):192.168.123.0/24 update_from_ctx(): no fib nhg
Oct 07 18:02:50 vyos zebra[940]: default(0:254):192.168.123.0/24 update_from_ctx(): rib nhg matched, changed 'false'
Oct 07 18:02:50 vyos zebra[940]: (0:254):192.168.123.0/24: Redist update re 0x560dcb8dfc20 (connected), old 0x0 (None)
Oct 07 18:02:50 vyos zebra[940]: redistribute_update: client bgp 192.168.123.0/24(0:254), type=2, distance=0, metric=0
Oct 07 18:02:50 vyos zebra[940]: zsend_redistribute_route: ZEBRA_REDISTRIBUTE_ROUTE_ADD to client bgp: type connected, vrf_id 0, p 192.168.123.0/24
Oct 07 18:02:50 vyos zebra[940]: redistribute_update: client ldp 192.168.123.0/24(0:254), type=2, distance=0, metric=0
Oct 07 18:02:50 vyos zebra[940]: zsend_redistribute_route: ZEBRA_REDISTRIBUTE_ROUTE_ADD to client ldp: type connected, vrf_id 0, p 192.168.123.0/24
Oct 07 18:02:50 vyos zebra[940]: Not Notifying Owner: connected about prefix 192.168.123.0/24(254) 2 vrf: 0
Oct 07 18:02:50 vyos zebra[940]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWADDR(20), len=80, seq=1665165771, pid=3364
Oct 07 18:02:50 vyos zebra[940]: netlink_interface_addr RTM_NEWADDR eth0 flags 0x81:
Oct 07 18:02:50 vyos zebra[940]:   IFA_LOCAL     192.168.123.2/24
Oct 07 18:02:50 vyos zebra[940]:   IFA_ADDRESS   192.168.123.2/24
Oct 07 18:02:50 vyos zebra[940]:   IFA_CACHEINFO pref -1, valid -1
Oct 07 18:02:50 vyos zebra[940]: MESSAGE: ZEBRA_INTERFACE_ADDRESS_ADD 192.168.123.2/24 on eth0(0)
Oct 07 18:02:50 vyos zebra[940]: rib_add_multipath_nhe: (0:254):192.168.123.0/24: Inserting route rn 0x560dcb8dfd00, re 0x560dcb8de6f0 (connected) existing (nil)
Oct 07 18:02:50 vyos zebra[940]: rib_add_multipath_nhe: (0:254):192.168.123.0/24 (MRIB): Inserting route rn 0x560dcb8de610, re 0x560dcb8de100 (connected) existing (nil)
Oct 07 18:02:50 vyos zebra[940]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=60, seq=0, pid=0
Oct 07 18:02:50 vyos zebra[940]: RTM_NEWROUTE ipv4 local proto kernel NS 0
Oct 07 18:02:50 vyos zebra[940]: Route rtm_type: local(2) intentionally ignoring
Oct 07 18:02:50 vyos zebra[940]: default(0:254):192.168.123.0/24: Adding route rn 0x560dcb8de610, re 0x560dcb8dd6d0 (connected)
Oct 07 18:02:50 vyos zebra[940]: default(0:254):192.168.123.0/24 update_from_ctx(): no fib nhg
Oct 07 18:02:50 vyos zebra[940]: default(0:254):192.168.123.0/24 update_from_ctx(): rib nhg matched, changed 'false'
Oct 07 18:02:50 vyos zebra[940]: Not Notifying Owner: connected about prefix 192.168.123.0/24(254) 2 vrf: 0

And in case of problem:

Oct 07 18:03:51 vyos zebra[940]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWADDR(20), len=80, seq=1665165832, pid=3404
Oct 07 18:03:51 vyos zebra[940]: netlink_interface_addr RTM_NEWADDR eth0 flags 0x80:
Oct 07 18:03:51 vyos zebra[940]:   IFA_LOCAL     192.168.123.2/24
Oct 07 18:03:51 vyos zebra[940]:   IFA_ADDRESS   192.168.123.2/24
Oct 07 18:03:51 vyos zebra[940]:   IFA_CACHEINFO pref -1, valid -1
Oct 07 18:03:51 vyos zebra[940]: MESSAGE: ZEBRA_INTERFACE_ADDRESS_ADD 192.168.123.2/24 on eth0(0)
Oct 07 18:03:51 vyos zebra[940]: rib_add_multipath_nhe: (0:0):192.168.123.0/24: Inserting route rn 0x560dcb8dfd00, re 0x560dcb8de180 (connected) existing (nil)
Oct 07 18:03:51 vyos zebra[940]: rib_add_multipath_nhe: (0:0):192.168.123.0/24 (MRIB): Inserting route rn 0x560dcb8de820, re 0x560dcb8de700 (connected) existing (nil)
Oct 07 18:03:51 vyos zebra[940]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=60, seq=0, pid=0
Oct 07 18:03:51 vyos zebra[940]: RTM_NEWROUTE ipv4 local proto kernel NS 0
Oct 07 18:03:51 vyos zebra[940]: Route rtm_type: local(2) intentionally ignoring
Oct 07 18:03:51 vyos zebra[940]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=60, seq=0, pid=0
Oct 07 18:03:51 vyos zebra[940]: RTM_NEWROUTE ipv4 unicast proto kernel NS 0
Oct 07 18:03:51 vyos zebra[940]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=60, seq=0, pid=0
Oct 07 18:03:51 vyos zebra[940]: RTM_NEWROUTE ipv4 broadcast proto kernel NS 0
Oct 07 18:03:51 vyos zebra[940]: Route rtm_type: broadcast(3) intentionally ignoring
Oct 07 18:03:51 vyos zebra[940]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=60, seq=0, pid=0
Oct 07 18:03:51 vyos zebra[940]: RTM_NEWROUTE ipv4 broadcast proto kernel NS 0
Oct 07 18:03:51 vyos zebra[940]: Route rtm_type: broadcast(3) intentionally ignoring
Oct 07 18:03:51 vyos zebra[940]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWADDR(20), len=80, seq=1665165832, pid=3405
Oct 07 18:03:51 vyos zebra[940]: netlink_interface_addr RTM_NEWADDR eth0 flags 0x81:
Oct 07 18:03:51 vyos zebra[940]:   IFA_LOCAL     192.168.123.1/24
Oct 07 18:03:51 vyos zebra[940]:   IFA_ADDRESS   192.168.123.1/24
Oct 07 18:03:51 vyos zebra[940]:   IFA_CACHEINFO pref -1, valid -1
Oct 07 18:03:51 vyos zebra[940]: MESSAGE: ZEBRA_INTERFACE_ADDRESS_ADD 192.168.123.1/24 on eth0(0)
Oct 07 18:03:51 vyos zebra[940]: rib_add_multipath_nhe: (0:254):192.168.123.0/24: Inserting route rn 0x560dcb8dfd00, re 0x560dcb8dda60 (connected) existing (nil)
Oct 07 18:03:51 vyos zebra[940]: rib_add_multipath_nhe: (0:254):192.168.123.0/24 (MRIB): Inserting route rn 0x560dcb8de820, re 0x560dcb8d8140 (connected) existing (nil)
Oct 07 18:03:51 vyos zebra[940]: netlink_parse_info: netlink-listen (NS 0) type RTM_NEWROUTE(24), len=60, seq=0, pid=0
Oct 07 18:03:51 vyos zebra[940]: RTM_NEWROUTE ipv4 local proto kernel NS 0
Oct 07 18:03:51 vyos zebra[940]: Route rtm_type: local(2) intentionally ignoring
Oct 07 18:03:51 vyos zebra[940]: default(0:254):192.168.123.0/24: Adding route rn 0x560dcb8dfd00, re 0x560dcb8de180 (connected)
Oct 07 18:03:51 vyos zebra[940]: default(0:254):192.168.123.0/24: Adding route rn 0x560dcb8de820, re 0x560dcb8de700 (connected)
Oct 07 18:03:51 vyos zebra[940]: netlink_nexthop_msg_encode: ID (22): directly connected, eth0(2) vrf default(0)
Oct 07 18:03:51 vyos zebra[940]: netlink_nexthop_msg_encode: RTM_NEWNEXTHOP, id=22
Oct 07 18:03:51 vyos zebra[940]: nl_batch_send: netlink-dp (NS 0), batch size=40, msg cnt=1
Oct 07 18:03:51 vyos zebra[940]: default(0:254):192.168.123.0/24 update_from_ctx(): no fib nhg
Oct 07 18:03:51 vyos zebra[940]: default(0:254):192.168.123.0/24 update_from_ctx(): rib nhg matched, changed 'false'
Oct 07 18:03:51 vyos zebra[940]: Not Notifying Owner: connected about prefix 192.168.123.0/24(254) 2 vrf: 0

As we can see, the redistribution event is missed in the second case:

Oct 07 18:02:50 vyos zebra[940]: (0:254):192.168.123.0/24: Redist update re 0x560dcb8dfc20 (connected), old 0x0 (None)
Oct 07 18:02:50 vyos zebra[940]: redistribute_update: client bgp 192.168.123.0/24(0:254), type=2, distance=0, metric=0
Oct 07 18:02:50 vyos zebra[940]: zsend_redistribute_route: ZEBRA_REDISTRIBUTE_ROUTE_ADD to client bgp: type connected, vrf_id 0, p 192.168.123.0/24
Oct 07 18:02:50 vyos zebra[940]: redistribute_update: client ldp 192.168.123.0/24(0:254), type=2, distance=0, metric=0
Oct 07 18:02:50 vyos zebra[940]: zsend_redistribute_route: ZEBRA_REDISTRIBUTE_ROUTE_ADD to client ldp: type connected, vrf_id 0, p 192.168.123.0/24

Additional info

The bug cannot be reproduced in FRRouting 8.0+. We should consider FRR update or find and backport the change that fixes this to FRR 7.5.1, if this is possible.

Details

Difficulty level: Hard (possibly days)
Version: 1.3.2
Why the issue appeared?: Issues in third-party code
Is it a breaking change?: Unspecified (possibly destroys the router)
Issue type: Bug (incorrect behavior)

Related Objects

Mentioned In: T5243: Default route is inactive if an interface has multiple ip addresses of the same subnet in 1.3.2 Equuleus
1.3.3
Mentioned Here: T4028: FRR 8.1 routes not being applied to routing table after reboot if an interface has 2 ip addresses

Event Timeline

zsdc created this task.Oct 7 2022, 6:13 PM

zsdc changed the task status from Open to Confirmed.Oct 7 2022, 6:39 PM

zsdc triaged this task as Normal priority.

zsdc changed Version from 1.3.1-S2 to 1.3.2.

zsdc changed Why the issue appeared? from Will be filled on close to Issues in third-party code.

pasik added a subscriber: pasik.Oct 7 2022, 8:20 PM

zsdc updated the task description. (Show Details)Oct 20 2022, 1:49 PM

Viacheslav edited projects, added VyOS 1.3 Equuleus (1.3.3); removed VyOS 1.3 Equuleus.Nov 1 2022, 5:31 PM

Hi @zsdc! This seems to be related to T4028. The relevant commits are:

━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
commit 92980561382fc04380414a6e2f6ca6746c2fe5e9 ┃
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━━
Author: Donald Sharp <[email protected]>
Date:   Mon Apr 19 19:23:45 2021 -0400

    zebra: Allow one connected route per network mask on a interface
    
    Currently FRR reads the kernel for interface state and FRR
    creates a connected route per address on an interface.  If
    you are in a situation where you have multiple addresses
    on an interface just create 1 connected route for them:
    ........
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
commit 7fb9825cf7e762add68f5108df4eddda1247f198 ┃
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━━━━━━━
Author: Volodymyr Huti <[email protected]>
Date:   Mon Apr 11 06:58:14 2022 +0300

    zebra: set ZEBRA_IFC_DOWN on connected routes for inactive interfaces
    
    If you are in a situation where you have multiple addresses on an
    interface, zebra creates one connected route for them.
    The issue is that the rib entry is not created if addresses were
    added before the interface was running.
    .....

From local testing, manually applying Donald`s patch on top of the base_7.6 tag seems to fix the issue -

zebra_connected_c.patch1 KBDownload

[R1]
sudo ip l set dev ens6 up
sudo ip a flush dev ens6
sleep 3
sudo ip a add 192.168.122.3/24 dev ens6 & sudo ip a add 192.168.122.2/24 dev ens6

I can see the prefix being redistributed to another device.

[R2]debian@debian:~$ vtr
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure
S   0.0.0.0/0 [1/0] is directly connected, ens5 inactive, weight 1, 00:01:06
...
B>  192.168.122.0/24 [200/0] via 10.0.0.1 (recursive), weight 1, 00:00:05
  *                            via 10.0.0.1, ens4 onlink, weight 1, 00:00:05

@v.huti Thanks for investigating and testing! How about you create a PR with the patch against 1.3 / equuleus ?

c-po assigned this task to v.huti.Jan 22 2023, 7:28 AM

c-po moved this task from Need Triage to Backlog on the VyOS 1.3 Equuleus (1.3.3) board.Jan 22 2023, 7:39 AM

adestis added a subscriber: adestis.Jan 30 2023, 9:15 AM

PR for 1.3: https://github.com/vyos/vyos-build/pull/300

Viacheslav changed the task status from In progress to Needs testing.Feb 1 2023, 12:04 PM

The required patch is not included in builds and they have the same bug.

Version:          VyOS 1.3-stable-202303270442
Release train:    equuleus

Built by:         [email protected]
Built on:         Mon 27 Mar 2023 04:42 UTC
Build UUID:       1bd11f80-fbaf-4764-95f4-b98cd7f0a00b
Build commit ID:  3ffe9a2689632b

Architecture:     x86_64
Boot via:         installed image
System type:      KVM guest

Hardware vendor:  QEMU
Hardware model:   Standard PC (i440FX + PIIX, 1996)
Hardware S/N:     
Hardware UUID:    7b538279-35bf-420f-aa19-93780e89f330

Copyright:        VyOS maintainers and contributors

This comment has been deleted.

dmbaturin mentioned this in 1.3.3.May 19 2023, 1:04 PM

Viacheslav reopened this task as Needs testing.May 23 2023, 4:09 AM

Viacheslav mentioned this in T5243: Default route is inactive if an interface has multiple ip addresses of the same subnet in 1.3.2 Equuleus.May 28 2023, 6:51 AM

syncer closed this task as Resolved.Jul 11 2023, 12:28 PM

FRRouting/zebra 7.5.1 does not redistribute routes to other protocolsClosed, ResolvedPublicBUGActions

Description

Details

A faster way to reproduce

Additional info

Details

Related Objects

Event Timeline

FRRouting/zebra 7.5.1 does not redistribute routes to other protocols
Closed, ResolvedPublicBUG
Actions