VRRP conntrack-sync dropping packets passing through the router
Open, Requires assessmentPublicBUG
Actions

Assigned To

None

Authored By

	Daya
	Sep 8 2019, 10:49 AM

Description

When "service conntrack-sync accept-protocol 'tcp,udp,icmp'" is enabled the router will start dropping packets and sessions that are passing through the router.

Version:
Version: VyOS 1.2.2
Built by: Sentrium S.L.
Built on: Mon 15 Jul 2019 04:10 UTC
Build UUID: b8264020-1697-4e7c-9457-2119b2c94535
Build Commit ID: 1d5a0fdcc288d0

////

Full config section as per below:

//
set service conntrack-sync accept-protocol 'tcp,udp,icmp'\
set service conntrack-sync event-listen-queue-size '8'
set service conntrack-sync failover-mechanism vrrp sync-group 'sgroup1'
set service conntrack-sync interface eth0
set service conntrack-sync mcast-group '225.0.0.50'
set service conntrack-sync sync-queue-size '8'
//

While the traffic through the router is interrupted below can be seen:

Log messages

Sep 06 02:08:41 wdc-rtr01v kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Sep 06 02:08:41 wdc-rtr01v kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Sep 06 02:08:41 wdc-rtr01v kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Sep 06 02:08:41 wdc-rtr01v kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Sep 06 02:08:41 wdc-rtr01v kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Sep 06 02:08:41 wdc-rtr01v kernel: nf_conntrack: nf_conntrack: table full, dropping packet

Connection track status

connections created: 3828923 failed: 17643161

At this present to resolve the issue and restore the traffic passing through the router is to remove the conntrack-sync accept protocol statement is the work around.

Details

Difficulty level: Normal (likely a few hours)
Version: 1.2.2
Why the issue appeared?: Will be filled on close
Is it a breaking change?: Unspecified (possibly destroys the router)

Event Timeline

Daya renamed this task from VRRP conntrack-sync dropping packet to VRRP conntrack-sync dropping packets passing through the router.Sep 8 2019, 10:49 AM

Daya created this task.

Hello @Daya , you can set custom kernel params for nf_conntrack

set system sysctl custom net.netfilter.nf_conntrack_max value 786432
set system sysctl custom net.nf_conntrack_max value 786432

Thanks for that, What I am suspecting is once the maximum value is reached the router is starting to drop packets, rather clearing the stale connections.

pasik added a subscriber: pasik.Sep 9 2019, 4:31 PM

This seems to be an issue in 1.4 as well, I have the exact same symptoms, and removing the accept-protocol fixes the issue.

@trae32566 Extentd conntrack table and reduce timeouts:
for example

set system sysctl parameter net.netfilter.nf_conntrack_generic_timeout value 60
set system sysctl parameter net.netfilter.nf_conntrack_icmp_timeout value 10
set system sysctl parameter net.netfilter.nf_conntrack_icmpv6_timeout value 10
set system sysctl parameter net.netfilter.nf_conntrack_tcp_timeout_close_wait value 20
set system sysctl parameter net.netfilter.nf_conntrack_tcp_timeout_established value 1800
set system sysctl parameter net.netfilter.nf_conntrack_tcp_timeout_fin_wait value 30
set system sysctl parameter net.netfilter.nf_conntrack_tcp_timeout_syn_recv value 30
set system sysctl parameter net.netfilter.nf_conntrack_tcp_timeout_syn_sent value 60
set system sysctl parameter net.netfilter.nf_conntrack_tcp_timeout_time_wait value 120
set system sysctl parameter net.netfilter.nf_conntrack_udp_timeout_stream value 60

In T1641#125443, @Viacheslav wrote:

@trae32566 Extentd conntrack table and reduce timeouts:
for example

I can give this a try, but the exact same settings on other routers using 1.3S1 work fine with accept-protocol specified, and the hardware for these (2x 8 core VM on a 5950X) is more than enough for the traffic I'm pushing. To be honest I actually don't even need accept-protocol though, I just always assumed it was mandatory.