Page MenuHomeVyOS Platform
Create Task
Maniphest T6963

FRR config is lost upon daemon restart
Closed, ResolvedPublicBUG
Actions

Description

Logs:

Dec 19 04:17:58 vyos-test-01 kernel: rcu: INFO: rcu_preempt detected stalls on CPUs/tasks:
Dec 19 04:17:58 vyos-test-01 kernel: net_ratelimit: 22 callbacks suppressed
Dec 19 04:17:58 vyos-test-01 kernel: IPv4: martian source 192.168.1.1 from 192.168.1.5, on dev eth4
Dec 19 04:17:58 vyos-test-01 kernel: rcu:         3-....: (8 ticks this GP) idle=ca3c/1/0x4000000000000000 softirq=117679788/117679790 fqs=4193
Dec 19 04:17:58 vyos-test-01 kernel: rcu:         (detected by 0, t=162366 jiffies, g=78510985, q=47444 ncpus=4)
Dec 19 04:17:58 vyos-test-01 kernel: Sending NMI from CPU 0 to CPUs 3:
Dec 19 04:17:58 vyos-test-01 kernel: NMI backtrace for cpu 3
Dec 19 04:17:58 vyos-test-01 kernel: CPU: 3 PID: 742782 Comm: kworker/3:2 Not tainted 6.6.32-amd64-vyos #1
Dec 19 04:17:58 vyos-test-01 kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
Dec 19 04:17:58 vyos-test-01 kernel: Workqueue: events_freezable_power_ disk_events_workfn
Dec 19 04:17:58 vyos-test-01 kernel: watchdog: BUG: soft lockup - CPU#2 stuck for 132s! [swapper/2:0]
Dec 19 04:17:58 vyos-test-01 kernel: RIP: 0010:trigger_load_balance+0x0/0x370
Dec 19 04:17:58 vyos-test-01 kernel: Modules linked in:
Dec 19 04:17:58 vyos-test-01 kernel: Code: 8b 04 fd 40 7c e3 8d 8b bc 06 70 0a 00 00 be 02 00 00 00 e9 42 f9 ff ff 66 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 <48> 8b 87 08 0a 00 00 48 85 c0 74 7a 41 57 49 89 ff 41 56 41 55 41
Dec 19 04:17:58 vyos-test-01 kernel: RSP: 0018:ffffb48480158ef0 EFLAGS: 00010096
Dec 19 04:17:58 vyos-test-01 kernel:  macvlan
Dec 19 04:17:58 vyos-test-01 kernel: 
Dec 19 04:17:58 vyos-test-01 kernel: RAX: ffff973377daea40 RBX: ffff973377d9dd00 RCX: ffff97324029ae00
Dec 19 04:17:58 vyos-test-01 kernel: RDX: 0000000000000000 RSI: 0000000000000006 RDI: ffff973377daea40
Dec 19 04:17:58 vyos-test-01 kernel:  vxlan
Dec 19 04:17:58 vyos-test-01 kernel: RBP: 0000000000000000 R08: ffff973377daeac0 R09: ffff97326f8e4590
Dec 19 04:17:58 vyos-test-01 kernel: R10: ffff973377daeb00 R11: 0000000000000000 R12: ffffb48483337a28
Dec 19 04:17:58 vyos-test-01 kernel: R13: ffff973377da0260 R14: ffff973377da0750 R15: ffff973377da0240
Dec 19 04:17:58 vyos-test-01 kernel:  ip6_udp_tunnel udp_tunnel
Dec 19 04:17:58 vyos-test-01 kernel: FS:  0000000000000000(0000) GS:ffff973377d80000(0000) knlGS:0000000000000000
Dec 19 04:17:58 vyos-test-01 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 19 04:17:58 vyos-test-01 kernel: CR2: 00007efc7c000020 CR3: 0000000121b46000 CR4: 0000000000750ee0
Dec 19 04:17:58 vyos-test-01 kernel:  sha3_generic
Dec 19 04:17:58 vyos-test-01 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec 19 04:17:58 vyos-test-01 kernel:  jitterentropy_rng
Dec 19 04:17:58 vyos-test-01 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec 19 04:17:58 vyos-test-01 kernel: PKRU: 55555554
Dec 19 04:17:58 vyos-test-01 kernel: Call Trace:
Dec 19 04:17:58 vyos-test-01 kernel:  drbg ansi_cprng echainiv geniv esp4 nft_reject_ipv4 nf_reject_ipv4
Dec 19 04:17:58 vyos-test-01 kernel:  <NMI>
Dec 19 04:17:58 vyos-test-01 kernel:  nft_reject nfnetlink_log nft_log xfrm_user xfrm_algo twofish_generic twofish_avx_x86_64 twofish_x86_64_3way twofish_x86_64 twofish_common serpent_avx2 serpent_avx_x86_64 serpent_sse2_x86_64 serpent_generic blowfish_generic blowfish_x86_64 blowfish_common
Dec 19 04:17:58 vyos-test-01 kernel:  ? nmi_cpu_backtrace+0x95/0x100
Dec 19 04:17:58 vyos-test-01 kernel:  cast5_avx_x86_64 cast5_generic cast_common ecb
Dec 19 04:17:58 vyos-test-01 kernel:  ? nmi_cpu_backtrace_handler+0x8/0x10
Dec 19 04:17:58 vyos-test-01 kernel:  des_generic libdes algif_skcipher camellia_generic camellia_aesni_avx2
Dec 19 04:17:58 vyos-test-01 kernel:  ? nmi_handle+0x55/0x150
Dec 19 04:17:58 vyos-test-01 kernel:  ? __pfx_trigger_load_balance+0x10/0x10
Dec 19 04:17:58 vyos-test-01 kernel:  camellia_aesni_avx_x86_64 camellia_x86_64
Dec 19 04:17:58 vyos-test-01 kernel:  ? default_do_nmi+0x6b/0x2c0
Dec 19 04:17:58 vyos-test-01 kernel:  xcbc
Dec 19 04:17:58 vyos-test-01 kernel:  ? exc_nmi+0x10d/0x140
Dec 19 04:17:58 vyos-test-01 kernel:  md4 algif_hash af_alg
Dec 19 04:17:58 vyos-test-01 kernel:  ? end_repeat_nmi+0x16/0x67
Dec 19 04:17:58 vyos-test-01 kernel:  xfrm_interface xfrm6_tunnel tunnel4 tunnel6 nft_masq
Dec 19 04:17:58 vyos-test-01 kernel:  ? __pfx_trigger_load_balance+0x10/0x10
Dec 19 04:17:58 vyos-test-01 kernel:  ? __pfx_trigger_load_balance+0x10/0x10
Dec 19 04:17:58 vyos-test-01 kernel:  nft_nat
Dec 19 04:17:58 vyos-test-01 kernel:  ? __pfx_trigger_load_balance+0x10/0x10
Dec 19 04:17:58 vyos-test-01 kernel:  </NMI>
Dec 19 04:17:58 vyos-test-01 kernel:  nf_nat_tftp
Dec 19 04:17:58 vyos-test-01 kernel:  <IRQ>
Dec 19 04:17:58 vyos-test-01 kernel:  nf_conntrack_tftp nf_nat_sip nf_conntrack_sip nf_nat_pptp nf_conntrack_pptp
Dec 19 04:17:58 vyos-test-01 kernel:  update_process_times+0x81/0x90
Dec 19 04:17:58 vyos-test-01 kernel:  nf_nat_h323 nf_conntrack_h323 nf_nat_ftp nf_conntrack_ftp af_packet
Dec 19 04:17:58 vyos-test-01 kernel:  tick_sched_timer+0x7a/0xb0
Dec 19 04:17:58 vyos-test-01 kernel:  nft_ct nft_chain_nat
Dec 19 04:17:58 vyos-test-01 kernel:  ? __pfx_tick_sched_timer+0x10/0x10
Dec 19 04:17:58 vyos-test-01 kernel:  nf_nat
Dec 19 04:17:58 vyos-test-01 kernel:  __hrtimer_run_queues+0x10a/0x2a0
Dec 19 04:17:58 vyos-test-01 kernel:  nf_tables
Dec 19 04:17:58 vyos-test-01 kernel:  hrtimer_interrupt+0xf9/0x230
Dec 19 04:17:58 vyos-test-01 kernel:  nfnetlink_cthelper nf_conntrack
Dec 19 04:17:58 vyos-test-01 kernel:  __sysvec_apic_timer_interrupt+0x66/0x170
Dec 19 04:17:58 vyos-test-01 kernel:  nf_defrag_ipv6 nf_defrag_ipv4 nfnetlink
Dec 19 04:17:58 vyos-test-01 kernel:  sysvec_apic_timer_interrupt+0x85/0xb0
Dec 19 04:17:58 vyos-test-01 kernel:  tcp_diag inet_diag binfmt_misc intel_rapl_common
Dec 19 04:17:58 vyos-test-01 kernel:  </IRQ>
Dec 19 04:17:58 vyos-test-01 kernel:  <TASK>
Dec 19 04:17:58 vyos-test-01 kernel:  crct10dif_pclmul crc32_pclmul
Dec 19 04:17:58 vyos-test-01 kernel:  asm_sysvec_apic_timer_interrupt+0x16/0x20
Dec 19 04:17:58 vyos-test-01 kernel:  ghash_clmulni_intel sha512_ssse3 sha512_generic sha256_ssse3 sha1_ssse3
Dec 19 04:17:58 vyos-test-01 kernel: RIP: 0010:_raw_spin_unlock_irqrestore+0x14/0x40
Dec 19 04:17:58 vyos-test-01 kernel:  aesni_intel
Dec 19 04:17:58 vyos-test-01 kernel: Code: 00 00 31 c0 eb f1 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 e8 fb 07 00 00 90 f7 c6 00 02 00 00 74 06 fb 0f 1f 44 00 00 <bf> 01 00 00 00 e8 f2 4c 68 ff 65 8b 05 c3 6f 9e 72 85 c0 74 05 c3
Dec 19 04:17:58 vyos-test-01 kernel: RSP: 0018:ffffb48483337ad0 EFLAGS: 00000206
Dec 19 04:17:58 vyos-test-01 kernel:  crypto_simd
Dec 19 04:17:58 vyos-test-01 kernel: RAX: 0000000000000001 RBX: 0000000000000000 RCX: 0000000000000000
Dec 19 04:17:58 vyos-test-01 kernel: RDX: 0000000000000000 RSI: 0000000000000293 RDI: ffff9732491d8d80
Dec 19 04:17:58 vyos-test-01 kernel: RBP: 0000000000000293 R08: 0000000000000001 R09: ffffffff8e043c60
Dec 19 04:17:58 vyos-test-01 kernel:  cryptd
Dec 19 04:17:58 vyos-test-01 kernel: R10: 0000000000000000 R11: 0000000000000000 R12: ffff973248b90000
Dec 19 04:17:58 vyos-test-01 kernel: R13: ffff973248bfb800 R14: ffff9732492c2000 R15: 0000000000000000
Dec 19 04:17:58 vyos-test-01 kernel:  rapl virtio_balloon sg pcspkr evdev button tcp_bbr sch_fq_codel mpls_iptunnel mpls_router ip_tunnel br_netfilter bridge stp llc efi_pstore fuse configfs ip_tables x_tables autofs4 usb_storage ohci_hcd squashfs lz4_decompress loop overlay ext4 crc16 mbcache jbd2 isofs efivarfs nls_ascii hid_generic usbhid hid sd_mod virtio_net net_failover virtio_scsi failover
Dec 19 04:17:58 vyos-test-01 kernel:  ata_scsi_queuecmd+0x4a/0x70 [libata]
Dec 19 04:17:58 vyos-test-01 kernel:  sr_mod cdrom ata_piix crc32c_intel virtio_pci virtio_pci_legacy_dev libata virtio_pci_modern_dev virtio i2c_piix4 virtio_ring scsi_mod uhci_hcd scsi_common ehci_hcd
Dec 19 04:17:58 vyos-test-01 kernel: CPU: 2 PID: 0 Comm: swapper/2 Not tainted 6.6.32-amd64-vyos #1
Dec 19 04:17:58 vyos-test-01 kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
Dec 19 04:17:58 vyos-test-01 kernel: RIP: 0010:vprintk_emit+0x228/0x2b0
Dec 19 04:17:58 vyos-test-01 kernel: Code: ae 01 84 c0 74 0d f3 90 0f b6 05 c6 f6 ae 01 84 c0 75 f3 e8 da 09 00 00 f7 c5 00 02 00 00 0f 84 10 ff ff ff fb 0f 1f 44 00 00 <e9> 05 ff ff ff 48 c7 c7 20 bc 7f 8e e8 e7 b3 93 00 e8 b2 09 00 00
Dec 19 04:17:58 vyos-test-01 kernel: RSP: 0018:ffffb48480124a80 EFLAGS: 00000206
Dec 19 04:17:58 vyos-test-01 kernel: RAX: 0000000000000000 RBX: 000000000000003e RCX: 0000000000000000
Dec 19 04:17:58 vyos-test-01 kernel: RDX: 0000000000000103 RSI: 0000000000000002 RDI: 00000000ffffffff
Dec 19 04:17:58 vyos-test-01 kernel: RBP: 0000000000000246 R08: ffffffff8e04a880 R09: 000000008e810493
Dec 19 04:17:58 vyos-test-01 kernel: R10: ffffffffffffffff R11: ffffffff8e0b3ad0 R12: ffff9732403be600
Dec 19 04:17:58 vyos-test-01 kernel: R13: ffffffff8de22450 R14: ffffb48480124ac8 R15: 0000000000000000
Dec 19 04:17:58 vyos-test-01 kernel: FS:  0000000000000000(0000) GS:ffff973377d00000(0000) knlGS:0000000000000000
Dec 19 04:17:58 vyos-test-01 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Dec 19 04:17:58 vyos-test-01 kernel: CR2: 00007efc877ba480 CR3: 000000016318e000 CR4: 0000000000750ee0
Dec 19 04:17:58 vyos-test-01 kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
Dec 19 04:17:58 vyos-test-01 kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Dec 19 04:17:58 vyos-test-01 kernel: PKRU: 55555554
Dec 19 04:17:58 vyos-test-01 kernel: Call Trace:
Dec 19 04:17:58 vyos-test-01 kernel:  <IRQ>
Dec 19 04:17:58 vyos-test-01 kernel:  ? watchdog_timer_fn+0x223/0x290
Dec 19 04:17:58 vyos-test-01 kernel:  ? __pfx_watchdog_timer_fn+0x10/0x10
Dec 19 04:17:58 vyos-test-01 kernel:  ? __hrtimer_run_queues+0x10a/0x2a0
Dec 19 04:17:58 vyos-test-01 kernel:  ? hrtimer_interrupt+0xf9/0x230
Dec 19 04:17:58 vyos-test-01 kernel:  scsi_queue_rq+0x362/0xb20 [scsi_mod]
Dec 19 04:17:58 vyos-test-01 kernel:  ? __sysvec_apic_timer_interrupt+0x66/0x170
Dec 19 04:17:58 vyos-test-01 kernel:  ? sysvec_apic_timer_interrupt+0x39/0xb0
Dec 19 04:17:58 vyos-test-01 kernel:  ? asm_sysvec_apic_timer_interrupt+0x16/0x20
Dec 19 04:17:58 vyos-test-01 kernel:  ? vprintk_emit+0x228/0x2b0
Dec 19 04:17:58 vyos-test-01 kernel:  _printk+0x53/0x70
Dec 19 04:17:58 vyos-test-01 kernel:  ? ___ratelimit+0x9f/0x110
Dec 19 04:17:58 vyos-test-01 kernel:  blk_mq_dispatch_rq_list+0x2d8/0x850
Dec 19 04:17:58 vyos-test-01 kernel:  ip_handle_martian_source+0x68/0xc0
Dec 19 04:17:58 vyos-test-01 kernel:  ? _raw_spin_lock_irq+0x20/0x40
Dec 19 04:17:58 vyos-test-01 kernel:  ip_route_input_slow+0xa95/0xb80
Dec 19 04:17:58 vyos-test-01 kernel:  ? update_group_capacity+0x20/0x1f0
Dec 19 04:17:58 vyos-test-01 kernel:  ip_route_input_noref+0x8e/0xa0
Dec 19 04:17:58 vyos-test-01 kernel:  ? get_page_from_freelist+0x143f/0x17b0
Dec 19 04:17:58 vyos-test-01 kernel:  arp_process+0x456/0x850
Dec 19 04:17:58 vyos-test-01 kernel:  __blk_mq_sched_dispatch_requests+0xab/0x600
Dec 19 04:17:58 vyos-test-01 kernel:  ? __pfx_autoremove_wake_function+0x10/0x10
Dec 19 04:17:58 vyos-test-01 kernel:  blk_mq_sched_dispatch_requests+0x2e/0x60
Dec 19 04:17:58 vyos-test-01 kernel:  ? page_to_skb+0x364/0x4f0 [virtio_net]
Dec 19 04:17:58 vyos-test-01 kernel:  blk_mq_run_hw_queue+0x159/0x190
Dec 19 04:17:58 vyos-test-01 kernel:  __netif_receive_skb_list_core+0x266/0x2c0
Dec 19 04:17:58 vyos-test-01 kernel:  blk_execute_rq+0x10d/0x220
Dec 19 04:17:58 vyos-test-01 kernel:  netif_receive_skb_list_internal+0x1ac/0x2e0
Dec 19 04:17:58 vyos-test-01 kernel:  scsi_execute_cmd+0xfc/0x2d0 [scsi_mod]
Dec 19 04:17:58 vyos-test-01 kernel:  napi_complete_done+0x69/0x1a0
Dec 19 04:17:58 vyos-test-01 kernel:  virtnet_poll+0x3e6/0x550 [virtio_net]
Dec 19 04:17:58 vyos-test-01 kernel:  __napi_poll+0x23/0x1b0
Dec 19 04:17:58 vyos-test-01 kernel:  net_rx_action+0x147/0x2c0
Dec 19 04:17:58 vyos-test-01 kernel:  sr_check_events+0xc0/0x2b0 [sr_mod]
Dec 19 04:17:58 vyos-test-01 kernel:  ? __napi_schedule+0xaf/0xc0
Dec 19 04:17:58 vyos-test-01 kernel:  __do_softirq+0xe8/0x2ef
Dec 19 04:17:58 vyos-test-01 kernel:  ? _raw_spin_unlock+0x10/0x30
Dec 19 04:17:58 vyos-test-01 kernel:  __irq_exit_rcu+0x71/0xc0
Dec 19 04:17:58 vyos-test-01 kernel:  common_interrupt+0xa5/0xc0
Dec 19 04:17:58 vyos-test-01 kernel:  cdrom_check_events+0x12/0x30 [cdrom]
Dec 19 04:17:58 vyos-test-01 kernel:  </IRQ>
Dec 19 04:17:58 vyos-test-01 kernel:  <TASK>
Dec 19 04:17:58 vyos-test-01 kernel:  asm_common_interrupt+0x22/0x40
Dec 19 04:17:58 vyos-test-01 kernel: RIP: 0010:pv_native_safe_halt+0xb/0x10
Dec 19 04:17:58 vyos-test-01 kernel: Code: 0b 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 66 90 0f 00 2d 79 67 3d 00 fb f4 <c3> cc cc cc cc 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 8b
Dec 19 04:17:58 vyos-test-01 kernel: RSP: 0018:ffffb484800abed0 EFLAGS: 00000246
Dec 19 04:17:58 vyos-test-01 kernel:  disk_check_events+0x32/0x100
Dec 19 04:17:58 vyos-test-01 kernel: 
Dec 19 04:17:58 vyos-test-01 kernel: RAX: 0000000000000002 RBX: 0000000000000002 RCX: 0000000000000000
Dec 19 04:17:58 vyos-test-01 kernel: RDX: 4000000000000000 RSI: ffffffff8dd90c03 RDI: 00000000466afc54
Dec 19 04:17:58 vyos-test-01 kernel: RBP: ffff97324027dc00 R08: 0000000000000001 R09: 0000000000000000
Dec 19 04:17:58 vyos-test-01 kernel: R10: 0000000000000001 R11: 0000000000000000 R12: 0000000000000000
Dec 19 04:17:58 vyos-test-01 kernel: R13: 0000000000000000 R14: ffff97324027dc00 R15: 0000000000000000
Dec 19 04:17:58 vyos-test-01 kernel:  default_idle+0x5/0x20
Dec 19 04:17:58 vyos-test-01 kernel:  default_idle_call+0x26/0xd0
Dec 19 04:17:58 vyos-test-01 kernel:  do_idle+0x1f1/0x230
Dec 19 04:17:58 vyos-test-01 kernel:  process_one_work+0x16c/0x340
Dec 19 04:17:58 vyos-test-01 kernel:  cpu_startup_entry+0x21/0x30
Dec 19 04:17:58 vyos-test-01 kernel:  start_secondary+0x116/0x140
Dec 19 04:17:58 vyos-test-01 kernel:  worker_thread+0x272/0x390
Dec 19 04:17:58 vyos-test-01 kernel:  secondary_startup_64_no_verify+0x178/0x17b
Dec 19 04:17:58 vyos-test-01 kernel:  ? preempt_count_add+0x65/0xa0
Dec 19 04:17:58 vyos-test-01 kernel:  ? __pfx_worker_thread+0x10/0x10
Dec 19 04:17:58 vyos-test-01 kernel:  </TASK>
Dec 19 04:17:58 vyos-test-01 kernel:  kthread+0xeb/0x120
Dec 19 04:17:58 vyos-test-01 kernel:  ? __pfx_kthread+0x10/0x10
Dec 19 04:17:58 vyos-test-01 kernel:  ret_from_fork+0x28/0x40
Dec 19 04:17:58 vyos-test-01 kernel:  ? __pfx_kthread+0x10/0x10
Dec 19 04:17:58 vyos-test-01 kernel:  ret_from_fork_asm+0x1b/0x30
Dec 19 04:17:58 vyos-test-01 kernel:  </TASK>
Dec 19 04:17:58 vyos-test-01 kernel: INFO: NMI handler (nmi_cpu_backtrace_handler) took too long to run: 1.019 msecs
Dec 19 04:17:58 vyos-test-01 kernel: rcu: rcu_preempt kthread timer wakeup didn't happen for 141368 jiffies! g78510985 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x200
Dec 19 04:17:58 vyos-test-01 kernel: clocksource: Long readout interval, skipping watchdog check: cs_nsec: 161763937289 wd_nsec: 161763860307
Dec 19 04:17:58 vyos-test-01 kernel: ll header: 00000000: ff ff ff ff ff ff bc 24 11 c8 16 9f 08 06
Dec 19 04:17:58 vyos-test-01 kernel: rcu:         Possible timer handling issue on cpu=0 timer-softirq=6553895
Dec 19 04:17:58 vyos-test-01 kernel: rcu: rcu_preempt kthread starved for 141370 jiffies! g78510985 f0x2 RCU_GP_WAIT_FQS(5) ->state=0x200 ->cpu=0
Dec 19 04:17:58 vyos-test-01 kernel: rcu:         Unless rcu_preempt kthread gets sufficient CPU time, OOM is now expected behavior.
Dec 19 04:17:58 vyos-test-01 kernel: rcu: RCU grace-period kthread stack dump:
Dec 19 04:17:58 vyos-test-01 kernel: task:rcu_preempt     state:R
Dec 19 04:17:58 vyos-test-01 kernel: IPv4: martian source 192.0.2.1 from 192.0.2.2, on dev eth3
Dec 19 04:17:58 vyos-test-01 kernel: ll header: 00000000: ff ff ff ff ff ff 72 27 cd 7f 0a e5 08 06
Dec 19 04:17:58 vyos-test-01 kernel: IPv4: martian source 192.0.2.1 from 192.0.2.2, on dev eth3
Dec 19 04:17:58 vyos-test-01 kernel:  stack:0     pid:16    ppid:2      flags:0x00004000
Dec 19 04:17:58 vyos-test-01 kernel: ll header: 00000000: ff ff ff ff ff ff 72 27 cd 7f 0a e5 08 06
Dec 19 04:17:58 vyos-test-01 kernel: Call Trace:
Dec 19 04:17:58 vyos-test-01 kernel:  <TASK>
Dec 19 04:17:58 vyos-test-01 kernel:  __schedule+0x379/0x9a0
Dec 19 04:17:58 vyos-test-01 kernel:  ? __pfx_rcu_gp_kthread+0x10/0x10
Dec 19 04:17:58 vyos-test-01 kernel:  schedule+0x58/0xd0
Dec 19 04:17:58 vyos-test-01 kernel:  schedule_timeout+0x82/0x150
Dec 19 04:17:58 vyos-test-01 kernel:  ? __pfx_process_timeout+0x10/0x10
Dec 19 04:17:58 vyos-test-01 kernel:  rcu_gp_fqs_loop+0x12f/0x520
Dec 19 04:17:58 vyos-test-01 kernel:  rcu_gp_kthread+0xca/0x160
Dec 19 04:17:58 vyos-test-01 kernel:  kthread+0xeb/0x120
Dec 19 04:17:58 vyos-test-01 kernel:  ? __pfx_kthread+0x10/0x10
Dec 19 04:17:58 vyos-test-01 kernel:  ret_from_fork+0x28/0x40
Dec 19 04:17:58 vyos-test-01 kernel:  ? __pfx_kthread+0x10/0x10
Dec 19 04:17:58 vyos-test-01 kernel:  ret_from_fork_asm+0x1b/0x30
Dec 19 04:17:58 vyos-test-01 kernel:  </TASK>
Dec 19 04:17:58 vyos-test-01 kernel: rcu: Stack dump where RCU GP kthread last ran:
Dec 19 04:17:58 vyos-test-01 kernel: CPU: 0 PID: 0 Comm: swapper/0 Tainted: G             L     6.6.32-amd64-vyos #1
Dec 19 04:17:58 vyos-test-01 kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
Dec 19 04:17:58 vyos-test-01 kernel: RIP: 0010:pv_native_safe_halt+0xb/0x10
Dec 19 04:17:58 vyos-test-01 systemd[1]: systemd-logind.service: Watchdog timeout (limit 3min)!
Dec 19 04:17:58 vyos-test-01 systemd[1]: systemd-logind.service: Killing process 964 (systemd-logind) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Watchdog timeout (limit 1min)!
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Killing process 1460 (watchfrr) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Killing process 1479 (zebra) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Killing process 1484 (mgmtd) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Killing process 1486 (bgpd) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Killing process 1493 (ripd) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Killing process 1496 (ripngd) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Killing process 1504 (ospfd) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Killing process 1507 (ospf6d) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Killing process 1513 (isisd) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Killing process 1516 (babeld) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Killing process 1519 (pim6d) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Killing process 1525 (ldpd) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Killing process 1526 (ldpd) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Killing process 1528 (ldpd) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Killing process 1532 (staticd) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Killing process 1536 (bfdd) with signal SIGABRT.
Dec 19 04:17:58 vyos-test-01 systemd[1]: frr.service: Main process exited, code=killed, status=6/ABRT
Dec 19 04:17:58 vyos-test-01 kernel: watchdog: BUG: soft lockup - CPU#1 stuck for 117s! [ksoftirqd/1:23]

Details

Version
1.4.0
Is it a breaking change?
Perfectly compatible
Issue type
Bug (incorrect behavior)

Related Objects

Mentioned In
1.4.3

Event Timeline

a.hajiyev created this task.Dec 19 2024, 1:06 PM
a.hajiyev added a comment.EditedDec 19 2024, 1:24 PM

There are around 300 interfaces:

  • around 200 vxlan interface
  • around 30 bridge interfaces
  • around 30 Pseudo-Ethernet/MACvlan interfaces
  • around 16 Ethernet interfaces
  • around 30 Bridge interfaces

Configured VRF, static route, dhcp etc
When we run sudo pkill watchfrr configs losts

vyos@vyos-test-01:~$ sudo pkill watchfrr
vyos@vyos-test-01:~$ sudo vtysh -c "show run"
Building configuration...

Current configuration:
!
frr version 9.1
frr defaults traditional
hostname vyos-test-01
log syslog
log facility local7
service integrated-vtysh-config
!
rpki
exit
!
end
Viacheslav triaged this task as Normal priority.Dec 19 2024, 1:53 PM
pasik subscribed.Dec 19 2024, 5:09 PM
vnguyen2015 subscribed.Dec 20 2024, 2:16 AM
dmbaturin edited projects, added VyOS Rolling, VyOS 1.5 Circinus, VyOS 1.4 Sagitta (1.4.3); removed VyOS 1.4 Sagitta.May 6 2025, 2:52 PM
a.hajiyev added a comment.May 16 2025, 12:03 PM

Tested in VyOS 1.4.2 problem persists

vyos@R-01:~$
vyos@R-01:~$ sudo pkill watchfrr
vyos@R-01:~$ sudo vtysh -c "show run"
Building configuration...

Current configuration:
!
frr version 9.1.3
frr defaults traditional
hostname R-01
log syslog
log facility local7
service integrated-vtysh-config
!
rpki
exit
!
end
vyos@R-01:~$
vyos@R-01:~$
vyos@R-01:~$ show version | grep -i version
Version:          VyOS 1.4.2
vyos@R-01:~$
vyos@R-01:~$
Viacheslav subscribed.May 16 2025, 12:53 PM

@a.hajiyev Check the latest rolling as a fix was added resently https://github.com/vyos/vyos-1x/pull/4500

a.hajiyev added a comment.EditedMay 19 2025, 4:57 AM

Tested in VyOS 2025.05.13-0019-rolling, issue resolved.

vyos@vyos# run sh version | grep -i version
Version:          VyOS 2025.05.13-0019-rolling
[edit]

vyos@vyos# sudo vtysh -c "show version"
FRRouting 10.2.2 (vyos) on Linux(6.6.89-vyos).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
Viacheslav closed this task as Resolved.May 19 2025, 9:36 AM
Viacheslav claimed this task.
Viacheslav moved this task from Need Triage to Completed on the VyOS Rolling board.
Viacheslav moved this task from Open to Finished on the VyOS 1.5 Circinus board.
Viacheslav moved this task from Backlog to Finished on the VyOS 1.4 Sagitta (1.4.3) board.
dmbaturin renamed this task from FRR some process does not answer in timeout and watchfrr was killed by watchdog to FRR config is lost upon daemon restart.May 20 2025, 2:08 PM
dmbaturin changed Is it a breaking change? from Unspecified (possibly destroys the router) to Perfectly compatible.
dmbaturin mentioned this in 1.4.3.Jul 17 2025, 8:18 AM