Page MenuHomeVyOS Platform

conntrack-tools flooding logs
Needs testing, HighPublic

Description

Hi,

I was running 1.2.5 and have upgraded to 1.2.6-epa1.
I have not changed any configuration, but now my logs are filled with the following:

Aug 16 08:33:50 ferrari conntrack-tools[6386]: tcp      6 src=37.48.65.66 dst=202.137.243.17 sport=46330 dport=49371 src=192.168.0.5 dst=37.48.65.66 sport=49371 dport=46330 [ASSURED]
Aug 16 08:33:50 ferrari conntrack-tools[6386]: tcp      6 src=192.168.0.104 dst=216.58.199.46 sport=59625 dport=443 src=216.58.199.46 dst=202.137.243.17 sport=443 dport=59625 [ASSURED]
Aug 16 08:33:51 ferrari conntrack-tools[6386]: tcp      6 src=192.168.0.252 dst=192.168.10.2 sport=59104 dport=161 src=192.168.10.2 dst=192.168.0.252 sport=161 dport=59104 [ASSURED]
Aug 16 08:33:51 ferrari conntrack-tools[6386]: tcp      6 src=192.168.0.252 dst=192.168.10.2 sport=59122 dport=161 src=192.168.10.2 dst=192.168.0.252 sport=161 dport=59122 [ASSURED]
Aug 16 08:33:51 ferrari conntrack-tools[6386]: tcp      6 src=184.171.210.143 dst=202.137.243.17 sport=33325 dport=49371 src=192.168.0.5 dst=184.171.210.143 sport=49371 dport=33325 [ASSURED]
Aug 16 08:33:52 ferrari conntrack-tools[6386]: tcp      6 src=192.168.0.106 dst=192.168.10.2 sport=43479 dport=993 src=192.168.10.2 dst=192.168.0.106 sport=993 dport=43479 [ASSURED]
Aug 16 08:33:52 ferrari conntrack-tools[6386]: tcp      6 src=65.60.150.40 dst=202.137.243.17 sport=42888 dport=49371 src=192.168.0.5 dst=65.60.150.40 sport=49371 dport=42888 [ASSURED]
Aug 16 08:33:52 ferrari conntrack-tools[6386]: tcp      6 src=192.168.0.5 dst=37.48.65.66 sport=49086 dport=80 src=37.48.65.66 dst=202.137.243.17 sport=80 dport=49086 [ASSURED]
Aug 16 08:33:54 ferrari conntrack-tools[6386]: tcp      6 src=192.168.0.7 dst=34.206.114.4 sport=44826 dport=443 src=34.206.114.4 dst=202.137.243.17 sport=443 dport=44826 [ASSURED]
Aug 16 08:33:54 ferrari conntrack-tools[6386]: tcp      6 src=192.168.0.5 dst=37.48.65.66 sport=49084 dport=80 src=37.48.65.66 dst=202.137.243.17 sport=80 dport=49084 [ASSURED]
Aug 16 08:33:55 ferrari conntrack-tools[6386]: tcp      6 src=37.48.65.66 dst=202.137.243.17 sport=46328 dport=49371 src=192.168.0.5 dst=37.48.65.66 sport=49371 dport=46328 [ASSURED]
Aug 16 08:33:57 ferrari conntrack-tools[6386]: tcp      6 src=192.168.0.106 dst=192.168.10.2 sport=43478 dport=993 src=192.168.10.2 dst=192.168.0.106 sport=993 dport=43478 [ASSURED]
Aug 16 08:33:57 ferrari conntrack-tools[6386]: tcp      6 src=192.168.0.120 dst=216.58.199.46 sport=12526 dport=443 src=216.58.199.46 dst=202.137.243.17 sport=443 dport=12526 [ASSURED]
Aug 16 08:33:57 ferrari conntrack-tools[6386]: tcp      6 src=192.168.0.5 dst=37.48.65.66 sport=49088 dport=80 src=37.48.65.66 dst=202.137.243.17 sport=80 dport=49088 [ASSURED]

I have two routers in my small network, they use VRRP and conntrack-sync between them so that on failover state can be maintained.
Config of my conntrack sync on my primary router (the one from which the logs above are collected)

[edit service conntrack-sync]
tim@ferrari# show
 accept-protocol tcp,udp,icmp
 event-listen-queue-size 8
 expect-sync all
 failover-mechanism {
     vrrp {
         sync-group failover-group
     }
 }
 interface eth1 {
 }
 mcast-group 225.0.0.50
 sync-queue-size 8
[edit service conntrack-sync]

There are so many conntrack logs that "show log" is essentially useless unless I do " show log | no-match conntrack-tools"

Thanks!

Details

Difficulty level
Unknown (require assessment)
Version
1.2.6-epa1
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Behavior change
Issue type
Unspecified (please specify)

Event Timeline

tjh created this object in space S1 VyOS Public.

So I fixed this on my setup by kill -9 conntrackd
and then sudo /etc/init.d/conntrackd start

It seems that maybe on first boot (maybe from an upgrade, which this box has been) that there's some logging going on?

It seems to have fixed it.
Prior to doing the above, I'd also turned off protocol debug in syslog, which I realised was on, but even restarting syslog didn't stop the flood of logs.

(This is what was turned on when the box booted)

global {
    facility all {
        level info
    }
    facility protocols {
        level debug
    }
}

So I don't know if it's something where the config didn't honour the disable of protocol debug, or if conntrackd didn't see the updated disable of protocol debug until it was killed, or something else that's setup on a first time bootup since a 1.2.5-1.2.6 upgrade, but one of those things has resolved the problem.

syncer changed the task status from Open to Needs testing.Sep 8 2020, 10:42 PM

So I just hit this bug again upgrading from 1.2.6-epa1 to 1.2.6.

This time I checked further and I found this when checking what processes were running:

root      7161  0.2  0.4  19784  2268 ?        Ss   16:40   0:01 /usr/sbin/conntrackd -C /etc/conntrackd/conntrackd.conf -d
root      8478  0.8  1.0  26648  5276 ?        S<s  16:40   0:03 /usr/sbin/conntrackd -C /etc/conntrackd/conntrackd.conf

On bootup, I had two conntrackd's running, one of which had "-d" appended to it.

I thought at first -d meant "debug" but it actually means "Run in daemon mode"

Anyway, the same fix as previous was applied - I killed both conntrackd's and then restarted conntrackd.

Now I'm not being spammed with logging.

Something isn't right when restart VyOS.

Additionally, it only happens after a system image upgrade - it doesn't seem to happen if you reboot again after that.

This is still a problem, I have just upgraded from 1.2.6-S1 to 1.2.7-epa1 and had two conntrackd's running on the primary router.

dmbaturin set Issue type to Unspecified (please specify).

I presume it's no longer an issue, but I'd like to confirm.

I stopped using conntrack-sync before I moved to 1.3 (which I am currently running) so I can't confirm either way.
I expect it's no longer an issue though and this task can be closed.