The VyOS default "set system conntrack modules nfs" and following non-defaults, combined with heavy NFSv3 traffic cause zombie entries, that should time out in 10 seconds by default, but get 5d long timeout instead in the conntrack table, which makes it fill up over time and start dropping new connections:
set service conntrack-sync accept-protocol 'tcp' set service conntrack-sync accept-protocol 'udp' set service conntrack-sync accept-protocol 'icmp' set service conntrack-sync failover-mechanism vrrp sync-group 'vrrp-sync-group' set service conntrack-sync interface gnv0
431999 ipv4 2 tcp 6 431999 CLOSE src=192.168.1.123 dst=192.168.3.222 sport=42780 dport=111 src=192.168.3.222 dst=192.168.1.123 sport=111 dport=42780 [ASSURED] mark=0 helper=rpc use=1 431999 ipv4 2 tcp 6 431999 CLOSE src=192.168.1.123 dst=192.168.3.222 sport=42672 dport=111 src=192.168.3.222 dst=192.168.1.123 sport=111 dport=42672 [ASSURED] mark=0 helper=rpc use=1 431999 ipv4 2 tcp 6 431999 CLOSE src=192.168.1.123 dst=192.168.3.222 sport=42580 dport=111 src=192.168.3.222 dst=192.168.1.123 sport=111 dport=42580 [ASSURED] mark=0 helper=rpc use=1 431999 ipv4 2 tcp 6 431999 CLOSE src=192.168.1.123 dst=192.168.3.222 sport=42512 dport=111 src=192.168.3.222 dst=192.168.1.123 sport=111 dport=42512 [ASSURED] mark=0 helper=rpc use=1 431999 ipv4 2 tcp 6 431999 CLOSE src=192.168.1.123 dst=192.168.3.222 sport=42416 dport=111 src=192.168.3.222 dst=192.168.1.123 sport=111 dport=42416 [ASSURED] mark=0 helper=rpc use=1 431999 ipv4 2 tcp 6 431999 CLOSE src=192.168.1.123 dst=192.168.3.222 sport=42312 dport=111 src=192.168.3.222 dst=192.168.1.123 sport=111 dport=42312 [ASSURED] mark=0 helper=rpc use=1 431999 ipv4 2 tcp 6 431999 CLOSE src=192.168.1.123 dst=192.168.3.222 sport=42098 dport=111 src=192.168.3.222 dst=192.168.1.123 sport=111 dport=42098 [ASSURED] mark=0 helper=rpc use=1 431999 ipv4 2 tcp 6 431999 CLOSE src=192.168.1.123 dst=192.168.3.222 sport=42040 dport=111 src=192.168.3.222 dst=192.168.1.123 sport=111 dport=42040 [ASSURED] mark=0 helper=rpc use=1 431999 ipv4 2 tcp 6 431999 CLOSE src=192.168.1.123 dst=192.168.3.222 sport=41788 dport=111 src=192.168.3.222 dst=192.168.1.123 sport=111 dport=41788 [ASSURED] mark=0 helper=rpc use=1 431999 ipv4 2 tcp 6 431999 CLOSE src=192.168.1.123 dst=192.168.3.222 sport=41784 dport=111 src=192.168.3.222 dst=192.168.1.123 sport=111 dport=41784 [ASSURED] mark=0 helper=rpc use=1 431999 ipv4 2 tcp 6 431999 CLOSE src=192.168.1.123 dst=192.168.3.222 sport=41782 dport=111 src=192.168.3.222 dst=192.168.1.123 sport=111 dport=41782 [ASSURED] mark=0 helper=rpc use=1 431999 ipv4 2 tcp 6 431999 CLOSE src=192.168.1.123 dst=192.168.3.222 sport=41378 dport=111 src=192.168.3.222 dst=192.168.1.123 sport=111 dport=41378 [ASSURED] mark=0 helper=rpc use=1 431999 ipv4 2 tcp 6 431999 CLOSE src=192.168.1.123 dst=192.168.3.222 sport=40980 dport=111 src=192.168.3.222 dst=192.168.1.123 sport=111 dport=40980 [ASSURED] mark=0 helper=rpc use=1 431999 ipv4 2 tcp 6 431999 CLOSE src=192.168.1.123 dst=192.168.3.222 sport=40454 dport=111 src=192.168.3.222 dst=192.168.1.123 sport=111 dport=40454 [ASSURED] mark=0 helper=rpc use=1 431998 ipv4 2 tcp 6 431998 CLOSE src=192.168.1.123 dst=192.168.3.222 sport=34010 dport=111 src=192.168.3.222 dst=192.168.1.123 sport=111 dport=34010 [ASSURED] mark=0 helper=rpc use=1 431998 ipv4 2 tcp 6 431998 CLOSE src=192.168.1.123 dst=192.168.3.222 sport=34008 dport=111 src=192.168.3.222 dst=192.168.1.123 sport=111 dport=34008 [ASSURED] mark=0 helper=rpc use=1 431998 ipv4 2 tcp 6 431998 CLOSE src=192.168.1.123 dst=192.168.3.222 sport=33320 dport=111 src=192.168.3.222 dst=192.168.1.123 sport=111 dport=33320 [ASSURED] mark=0 helper=rpc use=1 431998 ipv4 2 tcp 6 431998 CLOSE src=192.168.1.123 dst=192.168.3.222 sport=33318 dport=111 src=192.168.3.222 dst=192.168.1.123 sport=111 dport=33318 [ASSURED] mark=0 helper=rpc use=1
I used the following python script to reproduce this issue in my test environment:
#!/usr/bin/env python3 import argparse, socket, struct, threading, time def worker(dst, port, duration, rst, timeout): end = time.time() + duration while time.time() < end: try: s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) s.settimeout(timeout) s.connect((dst, port)) if rst: # SO_LINGER on, timeout=0 => close sends RST instead of FIN s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack('ii', 1, 0)) s.close() except Exception: try: s.close() except Exception: pass def main(): ap = argparse.ArgumentParser() ap.add_argument("--dst", required=True) ap.add_argument("--port", type=int, default=111) ap.add_argument("--threads", type=int, default=400) ap.add_argument("--seconds", type=int, default=30) ap.add_argument("--timeout", type=float, default=1.0) ap.add_argument("--rst", action="store_true") args = ap.parse_args() ts = [] for _ in range(args.threads): t = threading.Thread(target=worker, args=(args.dst, args.port, args.seconds, args.rst, args.timeout), daemon=True) t.start() ts.append(t) for t in ts: t.join() if __name__ == "__main__": main()
As soon as I disable either the conntrack-sync (by delete system service conntrack-sync) OR delete system conntrack modules nfs, the problem disappears. Even the old stuck zombie sessions get deleted automatically, when we trigger some new traffic.
The issue was there with all versions I tried:
VyOS 1.5-rolling-202412160007
VyOS 1.5-rolling-202602022358
VyOS 1.4.4 LTS
In summary, using conntrack-sync AND conntrack module nfs (which is enabled by default in VyOS) AND NFSv3, which relies heavily on small connections churn on port 111, we run into this bug. Disabling either conntrack-sync or the nfs module fixes it.