Page MenuHomeVyOS Platform

FRR some process does not answer in timeout and watchfrr was killed by watchdog
Open, NormalPublicBUG

Description

Logs:

Dec 19 04:17:58 vyos-test-01 kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
Dec 19 04:17:58 vyos-test-01 kernel: net_ratelimit: 22 callbacks suppressed
Dec 19 04:17:58 vyos-test-01 kernel: IPv4: martian source 192.168.1.1 from 192.168.1.5, on dev eth4
Dec 19 04:17:58 vyos-test-01 kernel: rcu:         3-....: (8 ticks this GP) idle=ca3c/1/0x4000000000000000 softirq=117679788/117679790 fqs=4193
Dec 19 04:17:58 vyos-test-01 kernel: rcu:         (detected by 0, t=162366 jiffies, g=78510985, q=47444 ncpus=4)
Dec 19 04:17:58 vyos-test-01 kernel: Sending NMI from CPU 0 to CPUs 3:
Dec 19 04:17:58 vyos-test-01 kernel: NMI backtrace for cpu 3
Dec 19 04:17:58 vyos-test-01 kernel: CPU: 3 PID: 742782 Comm: kworker/3:2 Not tainted 6.6.32-amd64-vyos #1
Dec 19 04:17:58 vyos-test-01 kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
Dec 19 04:17:58 vyos-test-01 kernel: Workqueue: events_freezable_power_ disk_events_workfn
Dec 19 04:17:58 vyos-test-01 kernel: watchdog: BUG: soft lockup - CPU#2 stuck for 132s! [swapper/2:0]
Dec 19 04:17:58 vyos-test-01 kernel: RIP: 0010:trigger_load_balance+0x0/0x370
Dec 19 04:17:58 vyos-test-01 kernel: Modules linked in:
Dec 19 04:17:58 vyos-test-01 kernel: Code: 8b 04 fd 40 7c e3 8d 8b bc 06 70 0a 00 00 be 02 00 00 00 e9 42 f9 ff ff 66 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 <48> 8b 87 08 0a 00 00 48 85 c0 74 7a 41 57 49 89 ff 41 56 41 55 41
Dec 19 04:17:58 vyos-test-01 kernel: RSP: 0018:ffffb48480158ef0 EFLAGS: 00010096
Dec 19 04:17:58 vyos-test-01 kernel:  macvlan
Dec 19 04:17:58 vyos-test-01 kernel: 
Dec 19 04:17:58 vyos-test-01 kernel: RAX: ffff973377daea40 RBX: ffff973377d9dd00 RCX: ffff97324029ae00
Dec 19 04:17:58 vyos-test-01 kernel: RDX: 0000000000000000 RSI: 0000000000000006 RDI: ffff973377daea40
Dec 19 04:17:58 vyos-test-01 kernel:  vxlan
Dec 19 04:17:58 vyos-test-01 kernel: RBP: 0000000000000000 R08: ffff973377daeac0 R09: ffff97326f8e4590
Dec 19 04:17:58 vyos-test-01 kernel: R10: ffff973377daeb00 R11: 0000000000000000 R12: ffffb48483337a28
Dec 19 04:17:58 vyos-test-01 kernel: R13: ffff973377da0260 R14: ffff973377da0750 R15: ffff973377da0240
Dec 19 04:17:58 vyos-test-01 kernel:  ip6_udp_tunnel udp_tunnel
Dec 19 04:17:58 vyos-test-01 kernel: FS:  0000000000000000(0000) GS:ffff973377d80000(0000) knlGS:0000000000000000
Dec 19 04:17:58 vyos-test-01 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 19 04:17:58 vyos-test-01 kernel: CR2: 00007efc7c000020 CR3: 0000000121b46000 CR4: 0000000000750ee0
Dec 19 04:17:58 vyos-test-01 kernel:  sha3_generic
Dec 19 04:17:58 vyos-test-01 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec 19 04:17:58 vyos-test-01 kernel:  jitterentropy_rng
Dec 19 04:17:58 vyos-test-01 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec 19 04:17:58 vyos-test-01 kernel: PKRU: 55555554
Dec 19 04:17:58 vyos-test-01 kernel: Call Trace:
Dec 19 04:17:58 vyos-test-01 kernel:  drbg ansi_cprng echainiv geniv esp4 nft_reject_ipv4 nf_reject_ipv4
Dec 19 04:17:58 vyos-test-01 kernel:  <NMI>
Dec 19 04:17:58 vyos-test-01 kernel:  nft_reject nfnetlink_log nft_log xfrm_user xfrm_algo twofish_generic twofish_avx_x86_64 twofish_x86_64_3way twofish_x86_64 twofish_common serpent_avx2 serpent_avx_x86_64 serpent_sse2_x86_64 serpent_generic blowfish_generic blowfish_x86_64 blowfish_common
Dec 19 04:17:58 vyos-test-01 kernel:  ? nmi_cpu_backtrace+0x95/0x100
Dec 19 04:17:58 vyos-test-01 kernel:  cast5_avx_x86_64 cast5_generic cast_common ecb
Dec 19 04:17:58 vyos-test-01 kernel:  ? nmi_cpu_backtrace_handler+0x8/0x10
Dec 19 04:17:58 vyos-test-01 kernel:  des_generic libdes algif_skcipher camellia_generic camellia_aesni_avx2
Dec 19 04:17:58 vyos-test-01 kernel:  ? nmi_handle+0x55/0x150
Dec 19 04:17:58 vyos-test-01 kernel:  ? __pfx_trigger_load_balance+0x10/0x10
Dec 19 04:17:58 vyos-test-01 kernel:  camellia_aesni_avx_x86_64 camellia_x86_64
Dec 19 04:17:58 vyos-test-01 kernel:  ? default_do_nmi+0x6b/0x2c0
Dec 19 04:17:58 vyos-test-01 kernel:  xcbc
Dec 19 04:17:58 vyos-test-01 kernel:  ? exc_nmi+0x10d/0x140
Dec 19 04:17:58 vyos-test-01 kernel:  md4 algif_hash af_alg
Dec 19 04:17:58 vyos-test-01 kernel:  ? end_repeat_nmi+0x16/0x67
Dec 19 04:17:58 vyos-test-01 kernel:  xfrm_interface xfrm6_tunnel tunnel4 tunnel6 nft_masq
Dec 19 04:17:58 vyos-test-01 kernel:  ? __pfx_trigger_load_balance+0x10/0x10
Dec 19 04:17:58 vyos-test-01 kernel:  ? __pfx_trigger_load_balance+0x10/0x10
Dec 19 04:17:58 vyos-test-01 kernel:  nft_nat
Dec 19 04:17:58 vyos-test-01 kernel:  ? __pfx_trigger_load_balance+0x10/0x10
Dec 19 04:17:58 vyos-test-01 kernel:  </NMI>
Dec 19 04:17:58 vyos-test-01 kernel:  nf_nat_tftp
Dec 19 04:17:58 vyos-test-01 kernel:  <IRQ>
Dec 19 04:17:58 vyos-test-01 kernel:  nf_conntrack_tftp nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp
Dec 19 04:17:58 vyos-test-01 kernel:  update_process_times+0x81/0x90
Dec 19 04:17:58 vyos-test-01 kernel:  nf_nat_h323 nf_conntrack_h323 nf_nat_ftp nf_conntrack_ftp af_packet
Dec 19 04:17:58 vyos-test-01 kernel:  tick_sched_timer+0x7a/0xb0
Dec 19 04:17:58 vyos-test-01 kernel:  nft_ct nft_chain_nat
Dec 19 04:17:58 vyos-test-01 kernel:  ? __pfx_tick_sched_timer+0x10/0x10
Dec 19 04:17:58 vyos-test-01 kernel:  nf_nat
Dec 19 04:17:58 vyos-test-01 kernel:  __hrtimer_run_queues+0x10a/0x2a0
Dec 19 04:17:58 vyos-test-01 kernel:  nf_tables
Dec 19 04:17:58 vyos-test-01 kernel:  hrtimer_interrupt+0xf9/0x230
Dec 19 04:17:58 vyos-test-01 kernel:  nfnetlink_cthelper nf_conntrack
Dec 19 04:17:58 vyos-test-01 kernel:  __sysvec_apic_timer_interrupt+0x66/0x170
Dec 19 04:17:58 vyos-test-01 kernel:  nf_defrag_ipv6 nf_defrag_ipv4 nfnetlink
Dec 19 04:17:58 vyos-test-01 kernel:  sysvec_apic_timer_interrupt+0x85/0xb0
Dec 19 04:17:58 vyos-test-01 kernel:  tcp_diag inet_diag binfmt_misc intel_rapl_common
Dec 19 04:17:58 vyos-test-01 kernel:  </IRQ>
Dec 19 04:17:58 vyos-test-01 kernel:  <TASK>
Dec 19 04:17:58 vyos-test-01 kernel:  crct10dif_pclmul crc32_pclmul
Dec 19 04:17:58 vyos-test-01 kernel:  asm_sysvec_apic_timer_interrupt+0x16/0x20
Dec 19 04:17:58 vyos-test-01 kernel:  ghash_clmulni_intel sha512_ssse3 sha512_generic sha256_ssse3 sha1_ssse3
Dec 19 04:17:58 vyos-test-01 kernel: RIP: 0010:_raw_spin_unlock_irqrestore+0x14/0x40
Dec 19 04:17:58 vyos-test-01 kernel:  aesni_intel
Dec 19 04:17:58 vyos-test-01 kernel: Code: 00 00 31 c0 eb f1 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 e8 fb 07 00 00 90 f7 c6 00 02 00 00 74 06 fb 0f 1f 44 00 00 <bf> 01 00 00 00 e8 f2 4c 68 ff 65 8b 05 c3 6f 9e 72 85 c0 74 05 c3
Dec 19 04:17:58 vyos-test-01 kernel: RSP: 0018:ffffb48483337ad0 EFLAGS: 00000206
Dec 19 04:17:58 vyos-test-01 kernel:  crypto_simd
Dec 19 04:17:58 vyos-test-01 kernel: RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000000
Dec 19 04:17:58 vyos-test-01 kernel: RDX: 0000000000000000 RSI: 0000000000000293 RDI: ffff9732491d8d80
Dec 19 04:17:58 vyos-test-01 kernel: RBP: 0000000000000293 R08: 0000000000000001 R09: ffffffff8e043c60
Dec 19 04:17:58 vyos-test-01 kernel:  cryptd
Dec 19 04:17:58 vyos-test-01 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff973248b90000
Dec 19 04:17:58 vyos-test-01 kernel: R13: ffff973248bfb800 R14: ffff9732492c2000 R15: 0000000000000000
Dec 19 04:17:58 vyos-test-01 kernel:  rapl virtio_balloon sg pcspkr evdev button tcp_bbr sch_fq_codel mpls_iptunnel mpls_router ip_tunnel br_netfilter bridge stp llc efi_pstore fuse configfs ip_tables x_tables autofs4 usb_storage ohci_hcd squashfs lz4_decompress loop overlay ext4 crc16 mbcache jbd2 isofs efivarfs nls_ascii hid_generic usbhid hid sd_mod virtio_net net_failover virtio_scsi failover
Dec 19 04:17:58 vyos-test-01 kernel:  ata_scsi_queuecmd+0x4a/0x70 [libata]
Dec 19 04:17:58 vyos-test-01 kernel:  sr_mod cdrom ata_piix crc32c_intel virtio_pci virtio_pci_legacy_dev libata virtio_pci_modern_dev virtio i2c_piix4 virtio_ring scsi_mod uhci_hcd scsi_common ehci_hcd
Dec 19 04:17:58 vyos-test-01 kernel: CPU: 2 PID: 0 Comm: swapper/2 Not tainted 6.6.32-amd64-vyos #1
Dec 19 04:17:58 vyos-test-01 kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
Dec 19 04:17:58 vyos-test-01 kernel: RIP: 0010:vprintk_emit+0x228/0x2b0
Dec 19 04:17:58 vyos-test-01 kernel: Code: ae 01 84 c0 74 0d f3 90 0f b6 05 c6 f6 ae 01 84 c0 75 f3 e8 da 09 00 00 f7 c5 00 02 00 00 0f 84 10 ff ff ff fb 0f 1f 44 00 00 <e9> 05 ff ff ff 48 c7 c7 20 bc 7f 8e e8 e7 b3 93 00 e8 b2 09 00 00
Dec 19 04:17:58 vyos-test-01 kernel: RSP: 0018:ffffb48480124a80 EFLAGS: 00000206
Dec 19 04:17:58 vyos-test-01 kernel: RAX: 0000000000000000 RBX: 000000000000003e RCX: 0000000000000000
Dec 19 04:17:58 vyos-test-01 kernel: RDX: 0000000000000103 RSI: 0000000000000002 RDI: 00000000ffffffff
Dec 19 04:17:58 vyos-test-01 kernel: RBP: 0000000000000246 R08: ffffffff8e04a880 R09: 000000008e810493
Dec 19 04:17:58 vyos-test-01 kernel: R10: ffffffffffffffff R11: ffffffff8e0b3ad0 R12: ffff9732403be600
Dec 19 04:17:58 vyos-test-01 kernel: R13: ffffffff8de22450 R14: ffffb48480124ac8 R15: 0000000000000000
Dec 19 04:17:58 vyos-test-01 kernel: FS:  0000000000000000(0000) GS:ffff973377d00000(0000) knlGS:0000000000000000
Dec 19 04:17:58 vyos-test-01 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 19 04:17:58 vyos-test-01 kernel: CR2: 00007efc877ba480 CR3: 000000016318e000 CR4: 0000000000750ee0
Dec 19 04:17:58 vyos-test-01 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec 19 04:17:58 vyos-test-01 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec 19 04:17:58 vyos-test-01 kernel: PKRU: 55555554
Dec 19 04:17:58 vyos-test-01 kernel: Call Trace:
Dec 19 04:17:58 vyos-test-01 kernel:  <IRQ>
Dec 19 04:17:58 vyos-test-01 kernel:  ? watchdog_timer_fn+0x223/0x290
Dec 19 04:17:58 vyos-test-01 kernel:  ? __pfx_watchdog_timer_fn+0x10/0x10
Dec 19 04:17:58 vyos-test-01 kernel:  ? __hrtimer_run_queues+0x10a/0x2a0
Dec 19 04:17:58 vyos-test-01 kernel:  ? hrtimer_interrupt+0xf9/0x230
Dec 19 04:17:58 vyos-test-01 kernel:  scsi_queue_rq+0x362/0xb20 [scsi_mod]
Dec 19 04:17:58 vyos-test-01 kernel:  ? __sysvec_apic_timer_interrupt+0x66/0x170
Dec 19 04:17:58 vyos-test-01 kernel:  ? sysvec_apic_timer_interrupt+0x39/0xb0
Dec 19 04:17:58 vyos-test-01 kernel:  ? asm_sysvec_apic_timer_interrupt+0x16/0x20
Dec 19 04:17:58 vyos-test-01 kernel:  ? vprintk_emit+0x228/0x2b0
Dec 19 04:17:58 vyos-test-01 kernel:  _printk+0x53/0x70
Dec 19 04:17:58 vyos-test-01 kernel:  ? ___ratelimit+0x9f/0x110
Dec 19 04:17:58 vyos-test-01 kernel:  blk_mq_dispatch_rq_list+0x2d8/0x850
Dec 19 04:17:58 vyos-test-01 kernel:  ip_handle_martian_source+0x68/0xc0
Dec 19 04:17:58 vyos-test-01 kernel:  ? _raw_spin_lock_irq+0x20/0x40
Dec 19 04:17:58 vyos-test-01 kernel:  ip_route_input_slow+0xa95/0xb80
Dec 19 04:17:58 vyos-test-01 kernel:  ? update_group_capacity+0x20/0x1f0
Dec 19 04:17:58 vyos-test-01 kernel:  ip_route_input_noref+0x8e/0xa0
Dec 19 04:17:58 vyos-test-01 kernel:  ? get_page_from_freelist+0x143f/0x17b0
Dec 19 04:17:58 vyos-test-01 kernel:  arp_process+0x456/0x850
Dec 19 04:17:58 vyos-test-01 kernel:  __blk_mq_sched_dispatch_requests+0xab/0x600
Dec 19 04:17:58 vyos-test-01 kernel:  ? __pfx_autoremove_wake_function+0x10/0x10
Dec 19 04:17:58 vyos-test-01 kernel:  blk_mq_sched_dispatch_requests+0x2e/0x60
Dec 19 04:17:58 vyos-test-01 kernel:  ? page_to_skb+0x364/0x4f0 [virtio_net]
Dec 19 04:17:58 vyos-test-01 kernel:  blk_mq_run_hw_queue+0x159/0x190
Dec 19 04:17:58 vyos-test-01 kernel:  __netif_receive_skb_list_core+0x266/0x2c0
Dec 19 04:17:58 vyos-test-01 kernel:  blk_execute_rq+0x10d/0x220
Dec 19 04:17:58 vyos-test-01 kernel:  netif_receive_skb_list_internal+0x1ac/0x2e0
Dec 19 04:17:58 vyos-test-01 kernel:  scsi_execute_cmd+0xfc/0x2d0 [scsi_mod]
Dec 19 04:17:58 vyos-test-01 kernel:  napi_complete_done+0x69/0x1a0
Dec 19 04:17:58 vyos-test-01 kernel:  virtnet_poll+0x3e6/0x550 [virtio_net]
Dec 19 04:17:58 vyos-test-01 kernel:  __napi_poll+0x23/0x1b0
Dec 19 04:17:58 vyos-test-01 kernel:  net_rx_action+0x147/0x2c0
Dec 19 04:17:58 vyos-test-01 kernel:  sr_check_events+0xc0/0x2b0 [sr_mod]
Dec 19 04:17:58 vyos-test-01 kernel:  ? __napi_schedule+0xaf/0xc0
Dec 19 04:17:58 vyos-test-01 kernel:  __do_softirq+0xe8/0x2ef
Dec 19 04:17:58 vyos-test-01 kernel:  ? _raw_spin_unlock+0x10/0x30
Dec 19 04:17:58 vyos-test-01 kernel:  __irq_exit_rcu+0x71/0xc0
Dec 19 04:17:58 vyos-test-01 kernel:  common_interrupt+0xa5/0xc0
Dec 19 04:17:58 vyos-test-01 kernel:  cdrom_check_events+0x12/0x30 [cdrom]
Dec 19 04:17:58 vyos-test-01 kernel:  </IRQ>
Dec 19 04:17:58 vyos-test-01 kernel:  <TASK>
Dec 19 04:17:58 vyos-test-01 kernel:  asm_common_interrupt+0x22/0x40
Dec 19 04:17:58 vyos-test-01 kernel: RIP: 0010:pv_native_safe_halt+0xb/0x10
Dec 19 04:17:58 vyos-test-01 kernel: Code: 0b 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 90 0f 00 2d 79 67 3d 00 fb f4 <c3> cc cc cc cc 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 8b
Dec 19 04:17:58 vyos-test-01 kernel: RSP: 0018:ffffb484800abed0 EFLAGS: 00000246
Dec 19 04:17:58 vyos-test-01 kernel:  disk_check_events+0x32/0x100
Dec 19 04:17:58 vyos-test-01 kernel: 
Dec 19 04:17:58 vyos-test-01 kernel: RAX: 0000000000000002 RBX: 0000000000000002 RCX: 0000000000000000
Dec 19 04:17:58 vyos-test-01 kernel: RDX: 4000000000000000 RSI: ffffffff8dd90c03 RDI: 00000000466afc54
Dec 19 04:17:58 vyos-test-01 kernel: RBP: ffff97324027dc00 R08: 0000000000000001 R09: 0000000000000000
Dec 19 04:17:58 vyos-test-01 kernel: R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000000
Dec 19 04:17:58 vyos-test-01 kernel: R13: 0000000000000000 R14: ffff97324027dc00 R15: 0000000000000000
Dec 19 04:17:58 vyos-test-01 kernel:  default_idle+0x5/0x20
Dec 19 04:17:58 vyos-test-01 kernel:  default_idle_call+0x26/0xd0
Dec 19 04:17:58 vyos-test-01 kernel:  do_idle+0x1f1/0x230
Dec 19 04:17:58 vyos-test-01 kernel:  process_one_work+0x16c/0x340
Dec 19 04:17:58 vyos-test-01 kernel:  cpu_startup_entry+0x21/0x30
Dec 19 04:17:58 vyos-test-01 kernel:  start_secondary+0x116/0x140
Dec 19 04:17:58 vyos-test-01 kernel:  worker_thread+0x272/0x390
Dec 19 04:17:58 vyos-test-01 kernel:  secondary_startup_64_no_verify+0x178/0x17b
Dec 19 04:17:58 vyos-test-01 kernel:  ? preempt_count_add+0x65/0xa0
Dec 19 04:17:58 vyos-test-01 kernel:  ? __pfx_worker_thread+0x10/0x10
Dec 19 04:17:58 vyos-test-01 kernel:  </TASK>
Dec 19 04:17:58 vyos-test-01 kernel:  kthread+0xeb/0x120
Dec 19 04:17:58 vyos-test-01 kernel:  ? __pfx_kthread+0x10/0x10
Dec 19 04:17:58 vyos-test-01 kernel:  ret_from_fork+0x28/0x40
Dec 19 04:17:58 vyos-test-01 kernel:  ? __pfx_kthread+0x10/0x10
Dec 19 04:17:58 vyos-test-01 kernel:  ret_from_fork_asm+0x1b/0x30
Dec 19 04:17:58 vyos-test-01 kernel:  </TASK>
Dec 19 04:17:58 vyos-test-01 kernel: INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 1.019 msecs
Dec 19 04:17:58 vyos-test-01 kernel: rcu: rcu_preempt kthread timer wakeup didn't happen for 141368 jiffies! g78510985 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x200
Dec 19 04:17:58 vyos-test-01 kernel: clocksource: Long readout interval, skipping watchdog check: cs_nsec: 161763937289 wd_nsec: 161763860307
Dec 19 04:17:58 vyos-test-01 kernel: ll header: 00000000: ff ff ff ff ff ff bc 24 11 c8 16 9f 08 06
Dec 19 04:17:58 vyos-test-01 kernel: rcu:         Possible timer handling issue on cpu=0 timer-softirq=6553895
Dec 19 04:17:58 vyos-test-01 kernel: rcu: rcu_preempt kthread starved for 141370 jiffies! g78510985 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x200 ->cpu=0
Dec 19 04:17:58 vyos-test-01 kernel: rcu:         Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
Dec 19 04:17:58 vyos-test-01 kernel: rcu: RCU grace-period kthread stack dump:
Dec 19 04:17:58 vyos-test-01 kernel: task:rcu_preempt     state:R
Dec 19 04:17:58 vyos-test-01 kernel: IPv4: martian source 192.0.2.1 from 192.0.2.2, on dev eth3
Dec 19 04:17:58 vyos-test-01 kernel: ll header: 00000000: ff ff ff ff ff ff 72 27 cd 7f 0a e5 08 06
Dec 19 04:17:58 vyos-test-01 kernel: IPv4: martian source 192.0.2.1 from 192.0.2.2, on dev eth3
Dec 19 04:17:58 vyos-test-01 kernel:  stack:0     pid:16    ppid:2      flags:0x00004000
Dec 19 04:17:58 vyos-test-01 kernel: ll header: 00000000: ff ff ff ff ff ff 72 27 cd 7f 0a e5 08 06
Dec 19 04:17:58 vyos-test-01 kernel: Call Trace:
Dec 19 04:17:58 vyos-test-01 kernel:  <TASK>
Dec 19 04:17:58 vyos-test-01 kernel:  __schedule+0x379/0x9a0
Dec 19 04:17:58 vyos-test-01 kernel:  ? __pfx_rcu_gp_kthread+0x10/0x10
Dec 19 04:17:58 vyos-test-01 kernel:  schedule+0x58/0xd0
Dec 19 04:17:58 vyos-test-01 kernel:  schedule_timeout+0x82/0x150
Dec 19 04:17:58 vyos-test-01 kernel:  ? __pfx_process_timeout+0x10/0x10
Dec 19 04:17:58 vyos-test-01 kernel:  rcu_gp_fqs_loop+0x12f/0x520
Dec 19 04:17:58 vyos-test-01 kernel:  rcu_gp_kthread+0xca/0x160
Dec 19 04:17:58 vyos-test-01 kernel:  kthread+0xeb/0x120
Dec 19 04:17:58 vyos-test-01 kernel:  ? __pfx_kthread+0x10/0x10
Dec 19 04:17:58 vyos-test-01 kernel:  ret_from_fork+0x28/0x40
Dec 19 04:17:58 vyos-test-01 kernel:  ? __pfx_kthread+0x10/0x10
Dec 19 04:17:58 vyos-test-01 kernel:  ret_from_fork_asm+0x1b/0x30
Dec 19 04:17:58 vyos-test-01 kernel:  </TASK>
Dec 19 04:17:58 vyos-test-01 kernel: rcu: Stack dump where RCU GP kthread last ran:
Dec 19 04:17:58 vyos-test-01 kernel: CPU: 0 PID: 0 Comm: swapper/0 Tainted: G             L     6.6.32-amd64-vyos #1
Dec 19 04:17:58 vyos-test-01 kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
Dec 19 04:17:58 vyos-test-01 kernel: RIP: 0010:pv_native_safe_halt+0xb/0x10
Dec 19 04:17:58 vyos-test-01 systemd[1]: systemd-logind.service: Watchdog timeout (limit 3min)!
Dec 19 04:17:58 vyos-test-01 systemd[1]: systemd-logind.service: Killing process 964 (systemd-logind) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Watchdog timeout (limit 1min)!
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Killing process 1460 (watchfrr) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Killing process 1479 (zebra) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Killing process 1484 (mgmtd) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Killing process 1486 (bgpd) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Killing process 1493 (ripd) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Killing process 1496 (ripngd) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Killing process 1504 (ospfd) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Killing process 1507 (ospf6d) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Killing process 1513 (isisd) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Killing process 1516 (babeld) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Killing process 1519 (pim6d) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Killing process 1525 (ldpd) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Killing process 1526 (ldpd) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Killing process 1528 (ldpd) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Killing process 1532 (staticd) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Killing process 1536 (bfdd) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Main process exited, code=killed, status=6/ABRT
Dec 19 04:17:58 vyos-test-01 kernel: watchdog: BUG: soft lockup - CPU#1 stuck for 117s! [ksoftirqd/1:23]

Details

Version
1.4.0
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Event Timeline

There are around 300 interfaces:

  • around 200 vxlan interface
  • around 30 bridge interfaces
  • around 30 Pseudo-Ethernet/MACvlan interfaces
  • around 16 Ethernet interfaces
  • around 30 Bridge interfaces

Configured VRF, static route, dhcp etc
When we run sudo pkill watchfrr configs losts

vyos@vyos-test-01:~$ sudo pkill watchfrr
vyos@vyos-test-01:~$ sudo vtysh -c "show run"
Building configuration...

Current configuration:
!
frr version 9.1
frr defaults traditional
hostname vyos-test-01
log syslog
log facility local7
service integrated-vtysh-config
!
rpki
exit
!
end
Viacheslav triaged this task as Normal priority.Dec 19 2024, 1:53 PM