PIMD crashes.
Network Map:
HOST(Consumer)<---->Switch(OnTime with IGMP Snooping)<---->(eth0)VyOS_Left(tun0 gre)<------------>(tun0 gre)VyOS_Right(eth0)<----->Switch(OnTime with IGMP Snooping)<---->Host(Server)
Configurations:
Pimd crashes on VyOS_Left after 2-3 hours after boot.
Configuration:
set interfaces dummy dum0 address '192.168.99.1/32' set interfaces ethernet eth0 address '192.168.10.254/24' set interfaces ethernet eth1 address '192.168.1.1/24' set interfaces ethernet eth1 hw-id '00:e0:4b:87:85:74' set interfaces loopback lo address '10.1.1.1/32' set interfaces tunnel tun0 address '10.10.10.1/30' set interfaces tunnel tun0 enable-multicast set interfaces tunnel tun0 encapsulation 'gre' set interfaces tunnel tun0 remote '192.168.1.2' set interfaces tunnel tun0 source-address '192.168.1.1' set protocols pim interface eth0 igmp set protocols pim interface eth1 igmp set protocols pim interface tun0 igmp set protocols pim rp address 10.1.1.2 group '239.1.1.1/32' set protocols static route 10.1.1.2/32 next-hop 10.10.10.2 set protocols static route 192.168.20.0/24 next-hop 10.10.10.2 set vpn ipsec authentication psk vyos id '192.168.1.1' set vpn ipsec authentication psk vyos id '192.168.1.2' set vpn ipsec authentication psk vyos secret 'MYSECRETKEY' set vpn ipsec esp-group MyESPGroup proposal 1 encryption 'aes128' set vpn ipsec esp-group MyESPGroup proposal 1 hash 'sha1' set vpn ipsec ike-group MyIKEGroup proposal 1 dh-group '2' set vpn ipsec ike-group MyIKEGroup proposal 1 encryption 'aes128' set vpn ipsec ike-group MyIKEGroup proposal 1 hash 'sha1' set vpn ipsec interface 'eth1' set vpn ipsec site-to-site peer right authentication mode 'pre-shared-secret' set vpn ipsec site-to-site peer right authentication remote-id '192.168.1.2' set vpn ipsec site-to-site peer right default-esp-group 'MyESPGroup' set vpn ipsec site-to-site peer right ike-group 'MyIKEGroup' set vpn ipsec site-to-site peer right local-address '192.168.1.1' set vpn ipsec site-to-site peer right remote-address '192.168.1.2' set vpn ipsec site-to-site peer right tunnel 1 local prefix '192.168.1.1/32' set vpn ipsec site-to-site peer right tunnel 1 protocol 'gre' set vpn ipsec site-to-site peer right tunnel 1 remote prefix '192.168.1.2/32'
Logs:
Mar 11 09:20:46 OnTimeCR pimd[2295]: [GW27G-J4HGZ] Scheduling READ event on PIM socket fd=20 Mar 11 09:20:47 OnTimeCR pimd[2295]: [XXYFJ-4M8ZW] Lookedup(224.0.1.129/32): rn 0x55947cd31ce0 found:224.0.0.0/4 Mar 11 09:20:47 OnTimeCR pimd[2295]: [REHPD-MYS7W] pim_upstream_sg_running: Handling unscanned inherited_olist for (192.168.10.10,224.0.1.129)[default] Mar 11 09:20:55 OnTimeCR pimd[2295]: [VP7A0-9MMYH] pim_upstream_send_join: RPF'(*,239.255.255.250)=0.0.0.0(Joined) for Interface tun0 Mar 11 09:20:55 OnTimeCR pimd[2295]: [M4NX7-916PT] pim_upstream_send_join: can't send join upstream: RPF'(*,239.255.255.250)=0.0.0.0 Mar 11 09:20:55 OnTimeCR pimd[2295]: [S0AES-PYS8Z] pim_joinprune_send: from 0.0.0.0 on tun0 Mar 11 09:20:55 OnTimeCR pimd[2295]: [Z4QK6-SWCXC] join_timer_start: starting 60 sec timer for upstream (S,G)=(*,239.255.255.250) Mar 11 09:21:01 OnTimeCR PIM[2295]: Received signal 11 at 1741684861 (si_addr 0x0, PC 0x7f3720d07346); aborting... Mar 11 09:21:01 OnTimeCR PIM[2295]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_backtrace_sigsafe+0x6f) [0x7f3720cbf3af] Mar 11 09:21:01 OnTimeCR kernel: pimreg: left allmulticast mode Mar 11 09:21:01 OnTimeCR kernel: igb 0000:03:00.0 eth0: left allmulticast mode Mar 11 09:21:01 OnTimeCR kernel: igb 0000:05:00.0 eth1: left allmulticast mode Mar 11 09:21:01 OnTimeCR kernel: tun0: left allmulticast mode Mar 11 09:21:01 OnTimeCR PIM[2295]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(zlog_signal+0xf5) [0x7f3720cbf5b5] Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: showing active allocations in memory group libfrr Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Buffer : 2 * 24 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Host config : 6 * (variably sized) Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Command Tokens : 3692 * 72 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Command Token Text : 2623 * (variably sized) Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Command Token Help : 2623 * (variably sized) Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Command Argument Name : 755 * (variably sized) Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Scripting : 14 * (variably sized) Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Graph : 19 * 8 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Graph Node : 4396 * 32 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Hash : 88 * (variably sized) Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Hash Bucket : 787 * 32 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Hash Index : 44 * (variably sized) Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Interface : 10 * 280 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Connected : 10 * 48 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Link List : 168 * 40 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Link Node : 97 * 24 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Nexthop : 3 * 152 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Northbound Node : 226 * 1192 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Northbound Configuration : 2 * 24 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Northbound Configuration Entry: 5 * 1032 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Prefix : 12 * 56 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Privilege information : 3 * (variably sized) Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Stream : 5 * (variably sized) Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Stream FIFO : 1 * 64 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Route table : 2 * 56 Mar 11 09:21:01 OnTimeCR PIM[2295]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(+0xf6941) [0x7f3720cf4941] Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Route node : 2 * 120 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Thread : 36 * 160 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Thread master : 4 * (variably sized) Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Thread Poll Info : 2 * 8192 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Thread stats : 29 * 112 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Typed-hash bucket : 3 * (variably sized) Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Typed-heap array : 1 * 576 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Vector : 8831 * 24 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Vector index : 8831 * (variably sized) Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: VRF : 1 * 216 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: VRF bit-map : 3 * 8 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: VTY server : 2 * 32 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Timer Wheel : 2 * (variably sized) Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Timer Wheel Slot List : 1 * 800 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: YANG module : 8 * 48 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Zclient : 2 * 3144 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: Redistribution instance IDs : 3 * 2 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: log thread-local buffer : 1 * 24608 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: showing active allocations in memory group logging subsystem Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: syslog target : 1 * 56 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: showing active allocations in memory group pimd Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: PIM SSM (S,G) channel OIL : 3 * 600 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: PIM interface : 4 * 376 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: PIM interface IGMP socket : 3 * 152 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: PIM interface IGMP group : 2 * 72 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: PIM interface IGMP source : 2 * 56 Mar 11 09:21:01 OnTimeCR PIM[2295]: /lib/x86_64-linux-gnu/libc.so.6(+0x3c050) [0x7f3720a3f050] Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: PIM interface neighbor : 2 * 80 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: PIM interface (S,G) state : 2 * 216 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: PIM upstream (S,G) state : 3 * 256 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: PIM RP info : 2 * 128 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: PIM JP AGG Group : 1 * 16 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: PIM global state : 1 * 856 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: PIM nexthop cache state : 3 * 120 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: PIM SSM configuration : 1 * 8 Mar 11 09:21:01 OnTimeCR vyos-configd[2295]: core_handler: memstats: PIM Router information : 1 * 176 Mar 11 09:21:01 OnTimeCR PIM[2295]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(typed_rb_remove+0x146) [0x7f3720d07346] Mar 11 09:21:01 OnTimeCR PIM[2295]: /usr/lib/frr/pimd(pim_channel_oil_del+0xa6) [0x559471851c26] Mar 11 09:21:01 OnTimeCR PIM[2295]: /usr/lib/frr/pimd(tib_sg_gm_prune+0x67) [0x559471859997] Mar 11 09:21:01 OnTimeCR PIM[2295]: /usr/lib/frr/pimd(igmp_source_forward_stop+0x5e) [0x55947187741e] Mar 11 09:21:01 OnTimeCR PIM[2295]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(event_call+0x81) [0x7f3720d06d61] Mar 11 09:21:01 OnTimeCR PIM[2295]: /usr/lib/x86_64-linux-gnu/frr/libfrr.so.0(frr_run+0xc0) [0x7f3720cb7120] Mar 11 09:21:01 OnTimeCR PIM[2295]: /usr/lib/frr/pimd(main+0xeb) [0x55947182e10b] Mar 11 09:21:01 OnTimeCR PIM[2295]: /lib/x86_64-linux-gnu/libc.so.6(+0x2724a) [0x7f3720a2a24a] Mar 11 09:21:01 OnTimeCR PIM[2295]: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0x85) [0x7f3720a2a305] Mar 11 09:21:01 OnTimeCR PIM[2295]: /usr/lib/frr/pimd(_start+0x21) [0x55947182e171] Mar 11 09:21:01 OnTimeCR PIM[2295]: in thread igmp_source_timer scheduled from ../pimd/pim_igmpv3.c:212 igmp_source_timer_on() Mar 11 09:21:01 OnTimeCR pimd[2295]: [QWG82-RYZEX] ifmembership_set: (S,G)=(*,239.1.1.10) membership now is NOINFO on interface eth0 Mar 11 09:21:01 OnTimeCR pimd[2295]: [RKCXG-R42M8] pim_upstream_switch: PIM_UPSTREAM_(*,239.1.1.10): (S,G) old: Joined new: NotJoined Mar 11 09:21:01 OnTimeCR pimd[2295]: [SAWDH-JC8EZ] pim_ifchannel_update_assert_tracking_desired: AssertTrackingDesired(*,239.1.1.10,eth0) changed from 1 to 0 Mar 11 09:21:01 OnTimeCR pimd[2295]: [PNHRP-A9BNM] pim_forward_stop: (S,G)=(*,239.1.1.10) oif=eth0 installed: 1 Mar 11 09:21:01 OnTimeCR pimd[2295]: [S0AES-PYS8Z] pim_joinprune_send: from 0.0.0.0 on tun0 Mar 11 09:21:01 OnTimeCR pimd[2295]: [PW4DZ-JXY98] ../pimd/pim_ifchannel.c pim_ifchannel_local_membership_del: Prune(S,G)=(*,239.1.1.10)(eth0) from (192.168.20.55,239.1.1.10) Mar 11 09:21:01 OnTimeCR pimd[2295]: [RKCXG-R42M8] pim_upstream_switch: PIM_UPSTREAM_(192.168.20.55,239.1.1.10): (S,G) old: Joined new: NotJoined Mar 11 09:21:01 OnTimeCR pimd[2295]: [XXYFJ-4M8ZW] Lookedup(239.1.1.10/32): rn 0x55947cef8fc0 found:239.1.1.0/24 Mar 11 09:21:01 OnTimeCR pimd[2295]: [S0AES-PYS8Z] pim_joinprune_send: from 10.10.10.2 on tun0 Mar 11 09:21:01 OnTimeCR pimd[2295]: [K89KQ-TD67B] pim_joinprune_send: interface tun0 num_joins 0 num_prunes 1 Mar 11 09:21:01 OnTimeCR pimd[2295]: [QHEGA-0QHKJ] pim_msg_send: to 224.0.0.13 on tun0: msg_size=34 checksum=f3fb Mar 11 09:21:01 OnTimeCR pimd[2295]: [JPMDE-1Y6GP] up (192.168.20.55,239.1.1.10) remove from nbr tun0/10.10.10.2 jp-agg-list Mar 11 09:21:01 OnTimeCR pimd[2295]: [QF563-RV25M] pim_ifchannel_delete: ifchannel entry (*,239.1.1.10)(eth0) del start Mar 11 09:21:01 OnTimeCR pimd[2295]: [MZWPQ-D4ZGM] pim_upstream_del(pim_ifchannel_delete): Delete (*,239.1.1.10)[default] ref count: 1, flags: 0 c_oil ref count 1 (Pre decrement) Mar 11 09:21:01 OnTimeCR pimd[2295]: [MNJ55-MN6HE] pim_upstream free vrf:default (*,239.1.1.10) flags 0x0 Mar 11 09:21:01 OnTimeCR pimd[2295]: [XMTWS-NFYRH] pim_upstream_del: Deregister upstream (*,239.1.1.10) addr 10.10.10.1 with Zebra NHT Mar 11 09:21:01 OnTimeCR pimd[2295]: [WZYPQ-X56ZA] pim_ifchannel_delete: ifchannel entry (*,239.1.1.10)(eth0) is deleted Mar 11 09:21:01 OnTimeCR zebra[1650]: [N5M5Y-J5BPG][EC 4043309121] Client 'pim' (session id 0) encountered an error and is shutting down. Mar 11 09:21:01 OnTimeCR zebra[1650]: [N5M5Y-J5BPG][EC 4043309121] Client 'system' (session id 0) encountered an error and is shutting down. Mar 11 09:21:01 OnTimeCR zebra[1650]: [YDZ55-W3VM6] release_daemon_table_chunks: Released 0 table chunks Mar 11 09:21:01 OnTimeCR zebra[1650]: [JPSA8-5KYEA] client 104 disconnected 0 pim routes removed from the rib Mar 11 09:21:01 OnTimeCR zebra[1650]: [S929C-NZR3N] client 104 disconnected 0 pim nhgs removed from the rib Mar 11 09:21:01 OnTimeCR zebra[1650]: [YDZ55-W3VM6] release_daemon_table_chunks: Released 0 table chunks Mar 11 09:21:01 OnTimeCR zebra[1650]: [JPSA8-5KYEA] client 107 disconnected 0 system routes removed from the rib Mar 11 09:21:01 OnTimeCR zebra[1650]: [S929C-NZR3N] client 107 disconnected 0 system nhgs removed from the rib Mar 11 09:21:01 OnTimeCR bgpd[1658]: [VCGF0-X62M1][EC 100663301] INTERFACE_STATE: Cannot find IF pimreg in VRF 0
This issue does not occur in the rolling release.