Page MenuHomeVyOS Platform

VRRP conntrack-sync dropping packets passing through the router
Not ApplicablePublicBUG

Description

When "service conntrack-sync accept-protocol 'tcp,udp,icmp'" is enabled the router will start dropping packets and sessions that are passing through the router.

Version:
Version: VyOS 1.2.2
Built by: Sentrium S.L.
Built on: Mon 15 Jul 2019 04:10 UTC
Build UUID: b8264020-1697-4e7c-9457-2119b2c94535
Build Commit ID: 1d5a0fdcc288d0

////

Full config section as per below:

//
set service conntrack-sync accept-protocol 'tcp,udp,icmp'\
set service conntrack-sync event-listen-queue-size '8'
set service conntrack-sync failover-mechanism vrrp sync-group 'sgroup1'
set service conntrack-sync interface eth0
set service conntrack-sync mcast-group '225.0.0.50'
set service conntrack-sync sync-queue-size '8'
//

While the traffic through the router is interrupted below can be seen:

Log messages

Sep 06 02:08:41 wdc-rtr01v kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Sep 06 02:08:41 wdc-rtr01v kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Sep 06 02:08:41 wdc-rtr01v kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Sep 06 02:08:41 wdc-rtr01v kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Sep 06 02:08:41 wdc-rtr01v kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Sep 06 02:08:41 wdc-rtr01v kernel: nf_conntrack: nf_conntrack: table full, dropping packet

Connection track status

connections created: 3828923 failed: 17643161

At this present to resolve the issue and restore the traffic passing through the router is to remove the conntrack-sync accept protocol statement is the work around.

Details

Version
1.2.2
Is it a breaking change?
Perfectly compatible
Issue type
Bug (incorrect behavior)

Event Timeline

Daya renamed this task from VRRP conntrack-sync dropping packet to VRRP conntrack-sync dropping packets passing through the router.
Unknown Object (User) subscribed.Sep 8 2019, 4:37 PM

Hello @Daya , you can set custom kernel params for nf_conntrack

set system sysctl custom net.netfilter.nf_conntrack_max value 786432
set system sysctl custom net.nf_conntrack_max value 786432

Thanks for that, What I am suspecting is once the maximum value is reached the router is starting to drop packets, rather clearing the stale connections.

This seems to be an issue in 1.4 as well, I have the exact same symptoms, and removing the accept-protocol fixes the issue.

@trae32566 Extentd conntrack table and reduce timeouts:
for example

set system sysctl parameter net.netfilter.nf_conntrack_generic_timeout value 60
set system sysctl parameter net.netfilter.nf_conntrack_icmp_timeout value 10
set system sysctl parameter net.netfilter.nf_conntrack_icmpv6_timeout value 10
set system sysctl parameter net.netfilter.nf_conntrack_tcp_timeout_close_wait value 20
set system sysctl parameter net.netfilter.nf_conntrack_tcp_timeout_established value 1800
set system sysctl parameter net.netfilter.nf_conntrack_tcp_timeout_fin_wait value 30
set system sysctl parameter net.netfilter.nf_conntrack_tcp_timeout_syn_recv value 30
set system sysctl parameter net.netfilter.nf_conntrack_tcp_timeout_syn_sent value 60
set system sysctl parameter net.netfilter.nf_conntrack_tcp_timeout_time_wait value 120
set system sysctl parameter net.netfilter.nf_conntrack_udp_timeout_stream value 60

@trae32566 Extentd conntrack table and reduce timeouts:
for example

I can give this a try, but the exact same settings on other routers using 1.3S1 work fine with accept-protocol specified, and the hardware for these (2x 8 core VM on a 5950X) is more than enough for the traffic I'm pushing. To be honest I actually don't even need accept-protocol though, I just always assumed it was mandatory.

dmbaturin set Issue type to Unspecified (please specify).
Viacheslav changed the task status from Open to Needs reporter action.Apr 7 2024, 5:20 PM

@Daya @trae32566 Any updates?

I think this was resolved at some point, but I ended up removing it (the accept-protocol stuff) from my config since it didn't appear necessary and was causing issues, so I'm not certain.

dmbaturin removed a project: Restricted Project.Oct 14 2024, 8:47 AM
dmbaturin changed Is it a breaking change? from Unspecified (possibly destroys the router) to Perfectly compatible.
dmbaturin changed Issue type from Unspecified (please specify) to Bug (incorrect behavior).
dmbaturin subscribed.

If anyone can find reproducing conditions for this task, feel free to reopen.