Page MenuHomeVyOS Platform

Conntrack-sync external cache commit error
Resolved (N/A)PublicBUG

Description

Report from the forum

To reproduce:

set interfaces ethernet eth0 address '10.0.5.2/30'

set high-availability vrrp group FOO address 192.0.2.1/32
set high-availability vrrp group FOO interface 'eth0'
set high-availability vrrp group FOO no-preempt
set high-availability vrrp group FOO priority '200'
set high-availability vrrp group FOO vrid '10'
set high-availability vrrp sync-group SGRP member 'FOO'

set service conntrack-sync accept-protocol 'tcp'
set service conntrack-sync accept-protocol 'udp'
set service conntrack-sync accept-protocol 'icmp'
set service conntrack-sync accept-protocol 'sctp'
set service conntrack-sync accept-protocol 'dccp'
set service conntrack-sync failover-mechanism vrrp sync-group 'SGRP'
set service conntrack-sync interface eth0 peer '10.0.5.1'

Change master -> backup restart vrrp on the old master
Logs from the new master:

May 03 11:23:09 vyos2 Keepalived_vrrp[10708]: (FOO) Entering MASTER STATE
May 03 11:23:09 vyos2 Keepalived_vrrp[10708]: VRRP_Group(SGRP) Syncing instances to MASTER state
May 03 11:23:09 vyos2 conntrack-tools[10726]: vyatta-vrrp-conntracksync invoked at Wed May  3 11:23:09 AM UTC 2023
May 03 11:23:09 vyos2 keepalived-fifo.py[10709]: Received message: INSTANCE "FOO" MASTER 200
May 03 11:23:09 vyos2 keepalived-fifo.py[10709]: INSTANCE FOO changed state to MASTER
May 03 11:23:09 vyos2 keepalived-fifo.py[10709]: Received message: GROUP "SGRP" MASTER 0
May 03 11:23:09 vyos2 keepalived-fifo.py[10709]: GROUP SGRP changed state to MASTER
May 03 11:23:09 vyos2 conntrack-tools[10730]: vyos2 transitioning to MASTER state for VRRP sync-group [SGRP]
May 03 11:23:09 vyos2 conntrackd[10236]: [Wed May  3 11:23:09 2023] (pid=10236) [notice] committing all external caches
May 03 11:23:09 vyos2 conntrackd[10236]: [Wed May  3 11:23:09 2023] (pid=10236) [ERROR] commit-create: Device or resource busy
May 03 11:23:09 vyos2 conntrackd[10236]: [Wed May  3 11:23:09 2023] (pid=10236) [ERROR] commit-create: Device or resource busy
May 03 11:23:09 vyos2 conntrackd[10236]: [Wed May  3 11:23:09 2023] (pid=10236) [ERROR] commit-create: Device or resource busy
May 03 11:23:09 vyos2 conntrack-tools[10236]: committing all external caches
May 03 11:23:09 vyos2 conntrackd[10236]: [Wed May  3 11:23:09 2023] (pid=10236) [ERROR] commit-create: Device or resource busy
May 03 11:23:09 vyos2 conntrackd[10236]: [Wed May  3 11:23:09 2023] (pid=10236) [ERROR] commit-create: Device or resource busy
May 03 11:23:09 vyos2 conntrackd[10236]: [Wed May  3 11:23:09 2023] (pid=10236) [notice] Committed 0 new entries
May 03 11:23:09 vyos2 conntrackd[10236]: [Wed May  3 11:23:09 2023] (pid=10236) [notice] 5 entries can't be committed
May 03 11:23:09 vyos2 conntrackd[10236]: [Wed May  3 11:23:09 2023] (pid=10236) [notice] commit has taken 0.001960 seconds
May 03 11:23:09 vyos2 conntrack-tools[10236]: commit-create: Device or resource busy
May 03 11:23:09 vyos2 conntrack-tools[10236]: udp      17 26 src=10.x.x.48 dst=10.x.x.254 sport=68 dport=67 mark=0
May 03 11:23:09 vyos2 conntrack-tools[10236]: commit-create: Device or resource busy
May 03 11:23:09 vyos2 conntrack-tools[10236]: udp      17 59 src=10.x.x.60 dst=255.255.255.255 sport=5678 dport=5678 [UNREPLIED] mark=0
May 03 11:23:09 vyos2 conntrack-tools[10236]: commit-create: Device or resource busy
May 03 11:23:09 vyos2 conntrack-tools[10236]: tcp      6 429113 ESTABLISHED src=10.x.x.1 dst=10.x.x.46 sport=62058 dport=22 [ASSURED] mark=0
May 03 11:23:09 vyos2 conntrack-tools[10236]: commit-create: Device or resource busy
May 03 11:23:09 vyos2 conntrack-tools[10236]: udp      17 59 src=10.0.5.2 dst=10.0.5.1 sport=45554 dport=3780 [UNREPLIED] mark=0
May 03 11:23:09 vyos2 conntrack-tools[10236]: commit-create: Device or resource busy
May 03 11:23:09 vyos2 conntrack-tools[10236]: udp      17 59 src=10.0.5.1 dst=10.0.5.2 sport=59002 dport=3780 [UNREPLIED] mark=0
May 03 11:23:09 vyos2 conntrack-tools[10236]: Committed 0 new entries
May 03 11:23:09 vyos2 conntrack-tools[10236]: 5 entries can't be committed
May 03 11:23:09 vyos2 conntrack-tools[10236]: commit has taken 0.001960 seconds
May 03 11:23:09 vyos2 conntrackd[10236]: [Wed May  3 11:23:09 2023] (pid=10236) [ERROR] ignoring flush command, commit still in progress
May 03 11:23:09 vyos2 conntrack-tools[10236]: ignoring flush command, commit still in progress
May 03 11:23:09 vyos2 conntrackd[10236]: [Wed May  3 11:23:09 2023] (pid=10236) [notice] resync with master conntrack table
May 03 11:23:09 vyos2 conntrack-tools[10236]: resync with master conntrack table
May 03 11:23:09 vyos2 conntrackd[10236]: [Wed May  3 11:23:09 2023] (pid=10236) [notice] sending bulk update
May 03 11:23:09 vyos2 conntrack-tools[10236]: sending bulk update
May 03 11:23:10 vyos2 conntrackd[10236]: [Wed May  3 11:23:10 2023] (pid=10236) [notice] resync requested by other node
May 03 11:23:10 vyos2 conntrackd[10236]: [Wed May  3 11:23:10 2023] (pid=10236) [notice] sending bulk update
May 03 11:23:10 vyos2 conntrack-tools[10236]: resync requested by other node
May 03 11:23:10 vyos2 conntrack-tools[10236]: sending bulk update
^C

Details

Difficulty level
Normal (likely a few hours)
Version
VyOS 1.4-rolling-202305030317
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Event Timeline

Not working. The same errors

vyos@vyos:~$ show ver
Version:          VyOS 1.4-rolling-202305091821
Release train:    current

Built by:         [email protected]
Built on:         Tue 09 May 2023 18:21 UTC
Build UUID:       0985bf69-617b-43d3-9f9c-71048567afb6
Build commit ID:  a234c0cb6cfa2d

Architecture:     x86_64
Boot via:         installed image
System type:      KVM guest

Hardware vendor:  QEMU
Hardware model:   Standard PC (i440FX + PIIX, 1996)
Hardware S/N:
Hardware UUID:    afe5b3f1-d108-4536-bfd4-7e608bfd09e7

Copyright:        VyOS maintainers and contributors

I tested with 1.3.2 and it works.

Version 1.4-rolling-202305150317 does not have the problem.

Viacheslav changed the task status from Open to Needs testing.May 15 2023, 3:03 PM