VRRP conntrack-sync dropping packets passing through the router
Not ApplicablePublicBUG
Actions

Assigned To

None

Authored By

	Daya
	Sep 8 2019, 10:49 AM

Description

When "service conntrack-sync accept-protocol 'tcp,udp,icmp'" is enabled the router will start dropping packets and sessions that are passing through the router.

Version:
Version: VyOS 1.2.2
Built by: Sentrium S.L.
Built on: Mon 15 Jul 2019 04:10 UTC
Build UUID: b8264020-1697-4e7c-9457-2119b2c94535
Build Commit ID: 1d5a0fdcc288d0

////

Full config section as per below:

//
set service conntrack-sync accept-protocol 'tcp,udp,icmp'\
set service conntrack-sync event-listen-queue-size '8'
set service conntrack-sync failover-mechanism vrrp sync-group 'sgroup1'
set service conntrack-sync interface eth0
set service conntrack-sync mcast-group '225.0.0.50'
set service conntrack-sync sync-queue-size '8'
//

While the traffic through the router is interrupted below can be seen:

Log messages

Sep 06 02:08:41 wdc-rtr01v kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Sep 06 02:08:41 wdc-rtr01v kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Sep 06 02:08:41 wdc-rtr01v kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Sep 06 02:08:41 wdc-rtr01v kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Sep 06 02:08:41 wdc-rtr01v kernel: nf_conntrack: nf_conntrack: table full, dropping packet
Sep 06 02:08:41 wdc-rtr01v kernel: nf_conntrack: nf_conntrack: table full, dropping packet

Connection track status

connections created: 3828923 failed: 17643161

At this present to resolve the issue and restore the traffic passing through the router is to remove the conntrack-sync accept protocol statement is the work around.

Details

Version: 1.2.2
Is it a breaking change?: Perfectly compatible
Issue type: Bug (incorrect behavior)

Event Timeline

Daya created this task.Sep 8 2019, 10:49 AM

Daya renamed this task from VRRP conntrack-sync dropping packet to VRRP conntrack-sync dropping packets passing through the router.

Hello @Daya , you can set custom kernel params for nf_conntrack

set system sysctl custom net.netfilter.nf_conntrack_max value 786432
set system sysctl custom net.nf_conntrack_max value 786432

Thanks for that, What I am suspecting is once the maximum value is reached the router is starting to drop packets, rather clearing the stale connections.

pasik subscribed.Sep 9 2019, 4:31 PM

This seems to be an issue in 1.4 as well, I have the exact same symptoms, and removing the accept-protocol fixes the issue.

@trae32566 Extentd conntrack table and reduce timeouts:
for example

set system sysctl parameter net.netfilter.nf_conntrack_generic_timeout value 60
set system sysctl parameter net.netfilter.nf_conntrack_icmp_timeout value 10
set system sysctl parameter net.netfilter.nf_conntrack_icmpv6_timeout value 10
set system sysctl parameter net.netfilter.nf_conntrack_tcp_timeout_close_wait value 20
set system sysctl parameter net.netfilter.nf_conntrack_tcp_timeout_established value 1800
set system sysctl parameter net.netfilter.nf_conntrack_tcp_timeout_fin_wait value 30
set system sysctl parameter net.netfilter.nf_conntrack_tcp_timeout_syn_recv value 30
set system sysctl parameter net.netfilter.nf_conntrack_tcp_timeout_syn_sent value 60
set system sysctl parameter net.netfilter.nf_conntrack_tcp_timeout_time_wait value 120
set system sysctl parameter net.netfilter.nf_conntrack_udp_timeout_stream value 60

In T1641#125443, @Viacheslav wrote:

@trae32566 Extentd conntrack table and reduce timeouts:
for example

I can give this a try, but the exact same settings on other routers using 1.3S1 work fine with accept-protocol specified, and the hardware for these (2x 8 core VM on a 5950X) is more than enough for the traffic I'm pushing. To be honest I actually don't even need accept-protocol though, I just always assumed it was mandatory.

rea13 subscribed.Sep 19 2023, 9:05 PM

dmbaturin triaged this task as High priority.Jan 9 2024, 3:48 PM

dmbaturin edited projects, added VyOS 1.4 Sagitta, VyOS 1.5 Circinus; removed VyOS 1.2 Crux (VyOS 1.2.2).

dmbaturin set Issue type to Unspecified (please specify).

@Daya @trae32566 Is it not the issue anymore?

@Daya @trae32566 Any updates?

syncer edited projects, added VyOS 1.4 Sagitta (1.4.0-GA); removed VyOS 1.4 Sagitta.May 7 2024, 8:02 AM

I think this was resolved at some point, but I ended up removing it (the accept-protocol stuff) from my config since it didn't appear necessary and was causing issues, so I'm not certain.

dmbaturin edited projects, added VyOS 1.4 Sagitta (1.4.1); removed VyOS 1.4 Sagitta (1.4.0-GA), VyOS 1.5 Circinus.Jul 2 2024, 7:29 PM

dmbaturin added a project: Restricted Project.Sep 16 2024, 4:15 PM

dmbaturin removed a project: Restricted Project.Oct 14 2024, 8:47 AM

dmbaturin changed Is it a breaking change? from Unspecified (possibly destroys the router) to Perfectly compatible.

dmbaturin changed Issue type from Unspecified (please specify) to Bug (incorrect behavior).

vyosbot added a project: Bugs.Oct 14 2024, 11:32 AM

dmbaturin edited projects, added VyOS 1.4 Sagitta (1.4.2); removed Bugs, VyOS 1.4 Sagitta (1.4.1).Dec 3 2024, 4:31 PM

dmbaturin edited projects, added VyOS 1.4 Sagitta (1.4.3), VyOS Rolling, VyOS 1.5 Circinus; removed VyOS 1.4 Sagitta (1.4.2).Jan 23 2025, 4:09 PM

If anyone can find reproducing conditions for this task, feel free to reopen.

dmbaturin removed projects: VyOS 1.5 Circinus, VyOS Rolling, VyOS 1.4 Sagitta (1.4.3).Jan 28 2025, 4:11 PM