bgpd crashed, when BGP EVPN peers are changed/rebooted.
sometimes, FRR will restart BGPd directly, sometimes not.
I will attach vyos setup in comment.
I tried to backport this patch, and it seems everthing is good right now.
https://github.com/FRRouting/frr/commit/8b087b2a4392a2fdc4645f9c31bb33402f53d3e7
the file I uploaded, is the backport version.
FRR log
Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: Mac Hash Entry : 9 * 16 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: Mac Hash Entry Intf String : 16 * (variably sized) Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP instance : 2 * 12456 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP Name data : 26 * (variably sized) Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP listen socket details : 6 * (variably sized) Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP peer : 13 * 24072 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP peer connection : 13 * 344 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP peer hostname : 21 * (variably sized) Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: Peer group : 4 * 64 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP Peer group hostname : 4 * (variably sized) Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: Peer description : 1 * 33 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP peer af : 6 * 80 Nov 19 15:15:26 vyos-2 BGP[18660]: /lib/x86_64-linux-gnu/libc.so.6(+0x8aeec) [0x7fea48149eec] Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP update group : 1 * 104 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP update subgroup : 1 * 240 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP packet : 1 * 56 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP attribute : 369 * 288 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP aspath : 16 * 40 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP aspath seg : 15 * 24 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP aspath segment data : 15 * (variably sized) Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP aspath str : 16 * (variably sized) Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP table : 337 * 56 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP node : 359 * 128 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP route : 293 * 144 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP ancillary route info : 288 * 72 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP extra info for EVPN : 279 * 32 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP extra info for vrf leaking: 106 * 80 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP connected : 7 * 4 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP static : 1 * 144 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP synchronise : 1 * 48 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP adj in : 290 * 56 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP adj out : 112 * 96 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: extcommunity : 178 * 32 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: extcommunity val : 178 * (variably sized) Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: extcommunity str : 136 * (variably sized) Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: community-list handler : 1 * 120 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP node clear queue : 50 * 8 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP nexthop : 8 * 232 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP peer update interface : 6 * 5 Nov 19 15:15:26 vyos-2 BGP[18660]: /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x12) [0x7fea480fafb2] Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP own address : 7 * 64 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP own tunnel-ip address : 2 * 8 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP redistribution : 2 * 24 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP LABELS : 24 * 24 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP EVPN Information : 20 * 152 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP EVPN MH Information : 1 * 56 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP EVPN Import RT : 20 * 16 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP EVPN VRF Import RT : 2 * 16 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP EVPN Overlay : 2 * 48 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP Martian Addr Intf String : 7 * (variably sized) Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP PBR Context : 2 * 32 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: BGP interface context : 16 * 4 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: showing active allocations in memory group rfapi Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: NVE Configuration : 1 * 3152 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: RFAPI Generic : 1 * 296 Nov 19 15:15:26 vyos-2 frrinit.sh[18660]: core_handler: memstats: RFAPI Import Table : 1 * 208 Nov 19 15:15:26 vyos-2 BGP[18660]: /lib/x86_64-linux-gnu/libc.so.6(abort+0xd3) [0x7fea480e5472] Nov 19 15:15:26 vyos-2 BGP[18660]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(_zlog_assert_failed+0xe9) [0x7fea4851bda9] Nov 19 15:15:26 vyos-2 BGP[18660]: /usr/lib/frr/bgpd(bgp_attr_unintern+0xae) [0x5638c49d72fe] Nov 19 15:15:26 vyos-2 BGP[18660]: /usr/lib/frr/bgpd(bgp_adj_in_remove+0x16) [0x5638c4b10c96] Nov 19 15:15:26 vyos-2 BGP[18660]: /usr/lib/frr/bgpd(+0x170cc4) [0x5638c4a4ccc4] Nov 19 15:15:26 vyos-2 BGP[18660]: /usr/lib/frr/bgpd(bgp_clear_route+0xff) [0x5638c4a51edf] Nov 19 15:15:26 vyos-2 BGP[18660]: /usr/lib/frr/bgpd(bgp_clear_route_all+0x30) [0x5638c4a52020] Nov 19 15:15:26 vyos-2 BGP[18660]: /usr/lib/frr/bgpd(bgp_fsm_change_status+0x258) [0x5638c4a1c168] Nov 19 15:15:26 vyos-2 BGP[18660]: /usr/lib/frr/bgpd(bgp_event_update+0x1ee) [0x5638c4a1dcce] Nov 19 15:15:26 vyos-2 BGP[18660]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(event_call+0x81) [0x7fea484fc8a1] Nov 19 15:15:26 vyos-2 BGP[18660]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(frr_run+0xc0) [0x7fea484a50a0] Nov 19 15:15:26 vyos-2 BGP[18660]: /usr/lib/frr/bgpd(main+0x3d6) [0x5638c49d29e6] Nov 19 15:15:26 vyos-2 BGP[18660]: /lib/x86_64-linux-gnu/libc.so.6(+0x2724a) [0x7fea480e624a] Nov 19 15:15:26 vyos-2 BGP[18660]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7fea480e6305] Nov 19 15:15:26 vyos-2 BGP[18660]: /usr/lib/frr/bgpd(_start+0x21) [0x5638c49d4991] Nov 19 15:15:26 vyos-2 BGP[18660]: in thread bgp_packet_process_error scheduled from ../bgpd/bgp_io.c:255 bgp_process_reads() Nov 19 15:15:26 vyos-2 zebra[18565]: [EC 4043309121] Client 'bgp' (session id 1) encountered an error and is shutting down. Nov 19 15:15:26 vyos-2 zebra[18565]: [EC 4043309121] Client 'vnc' (session id 0) encountered an error and is shutting down. Nov 19 15:15:26 vyos-2 watchfrr[18543]: [HD38Q-0HBRT][EC 268435457] bgpd state -> down : read returned EOF Nov 19 15:15:26 vyos-2 zebra[18565]: [EC 4043309121] Client 'bgp' (session id 0) encountered an error and is shutting down. Nov 19 15:15:26 vyos-2 zebra[18565]: ../zebra/zebra_ptm.c:1285 failed to find process pid registration Nov 19 15:15:26 vyos-2 zebra[18565]: client 111 disconnected 0 bgp routes removed from the rib Nov 19 15:15:26 vyos-2 zebra[18565]: client 111 disconnected 0 bgp nhgs removed from the rib Nov 19 15:15:26 vyos-2 zebra[18565]: client 32 disconnected 0 vnc routes removed from the rib Nov 19 15:15:26 vyos-2 zebra[18565]: client 32 disconnected 0 vnc nhgs removed from the rib Nov 19 15:15:26 vyos-2 zebra[18565]: client 29 disconnected 5 bgp routes removed from the rib Nov 19 15:15:26 vyos-2 zebra[18565]: client 29 disconnected 0 bgp nhgs removed from the rib Nov 19 15:15:31 vyos-2 watchfrr[18543]: [YFT0P-5Q5YX] Forked background command [pid 19606]: /usr/lib/frr/watchfrr.sh restart bgpd Nov 19 15:15:31 vyos-2 frrinit.sh[19606]: Cannot stop bgpd: pid 18660 not running Nov 19 15:15:31 vyos-2 zebra[18565]: client 33 says hello and bids fair to announce only vnc routes vrf=0 Nov 19 15:15:31 vyos-2 zebra[18565]: client 30 says hello and bids fair to announce only bgp routes vrf=0 Nov 19 15:15:31 vyos-2 bgpd[19615]: [EC 33554466] 169.254.0.2 [FSM] Failure handling event BGP_Start in state Idle, prior events (null), (null), fd -1, last reset: No AFI/SAFI activated for peer Nov 19 15:15:31 vyos-2 bgpd[19615]: [EC 33554466] 169.254.0.6 [FSM] Failure handling event BGP_Start in state Idle, prior events (null), (null), fd -1, last reset: No AFI/SAFI activated for peer Nov 19 15:15:31 vyos-2 bgpd[19615]: [EC 33554466] 169.254.254.0 [FSM] Failure handling event BGP_Start in state Idle, prior events (null), (null), fd -1, last reset: No AFI/SAFI activated for peer Nov 19 15:15:31 vyos-2 bgpd[19615]: [EC 33554466] 169.254.254.1 [FSM] Failure handling event BGP_Start in state Idle, prior events (null), (null), fd -1, last reset: No AFI/SAFI activated for peer Nov 19 15:15:31 vyos-2 watchfrr[18543]: [QDG3Y-BY5TN] bgpd state -> up : connect succeeded Nov 19 15:15:32 vyos-2 zebra[18565]: client 112 says hello and bids fair to announce only bgp routes vrf=0 Nov 19 15:15:32 vyos-2 bgpd[19615]: [EC 100663299] Can't get remote address and port: Transport endpoint is not connected Nov 19 15:15:32 vyos-2 bgpd[19615]: [EC 100663299] Can't get remote address and port: Transport endpoint is not connected Nov 19 15:15:32 vyos-2 bgpd[19615]: [EC 100663299] Can't get remote address and port: Transport endpoint is not connected Nov 19 15:15:32 vyos-2 bgpd[19615]: [EC 100663299] Can't get remote address and port: Transport endpoint is not connected Nov 19 15:15:37 vyos-2 bgpd[19615]: [EC 100663299] Can't get remote address and port: Transport endpoint is not connected Nov 19 15:15:38 vyos-2 bgpd[19615]: [EC 100663299] Can't get remote address and port: Transport endpoint is not connected Nov 19 15:15:42 vyos-2 bgpd[19615]: [EC 100663299] Can't get remote address and port: Transport endpoint is not connected