There are no other reports about such issues
we need valid reproduction procedure for this, otherwise, not much that we can do

@syncer Re-opening this. Had the first exact same incident on a different router with an IPv6 BGP session on a RJ45 connection, so that would rule out any issues with the Intel X710 card in relation to this issue.

any L2/L3 issue affecting TCP between the BGP speaker will cause this message. Looking forward to a TCP dump of the traffic when it occurs.

@Merjin is trying this:

sudo sysctl -w net.ipv6.route.max_size=131072

https://serverfault.com/questions/902161/linux-host-randomly-stops-answering-ipv6-neighbor-solicitation-requests
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=861115

The IPv6 DFZ is now between 80k and 90k routes

Some statistics from ipv6 bgp summary

RIB entries 167048, using 29 MiB of memory
Peers 59, using 1206 KiB of memory
Peer groups 12, using 768 bytes of memory

This router currently receives 2 full tables of 86758 routes
6 partials tables of 22000 to 45000 routes (some are full tables but with less preferred paths) and some 50 peers with <1000 routes,

It would really help testing this setting if someone else has about the same setup in number of peers and routes.

We've got full IPv4 and IPv6 routing tables on our VyOS boxes, and we *definitely* needed to increase net.ipv6.route.max_size (we picked 256k to give us some headroom).

Marek Isalski Today at 6:31 PM

I'm pretty sure that the contents of /proc/sys/net/ipv6/route/max_size is not a cache, but the actual number of slots that the routing table can contain. And I believe this because without changing it on our core routers, we were blackholing IPv6 traffic (we were learning the routes into FRR, but they were not being installed in the Linux kernel…).

Worse: if routes flapped, sometimes our loopback routes would be de-installed, something was learned from eBGP, and now the loopback would not end up reprogrammed into the Linux kernel's route table. This meant we had internal routing problems during our testing of VyOS back in April 2019. It then dawned on me: max_size needed to be increased (I had seen this before on a Quagga-based deployment, but hadn't expected VyOS to have a low number of IPv6 routes).

We deliberately set our routing table sizes to this in our production deployment:
sysctl:

custom:
    net.ipv4.route.max_size: 2097152
    net.ipv6.route.max_size: 262144
    net.ipv4.conf.all.log_martians: 0

Marek Isalski 1 hour ago
The reason why I don't set them both to 2 Billion is because, especially with IPv6, the Linux kernel inserts a temporary "cache" entry into the IPv6 route table radix tree. This uses up RAM. When there is no RAM left, the kernel might OOPS (this was the subject of the CVE-2018-19299 bug in MikroTik's RouterOS which had set the IPv6 max_size so big that it made the routers vulnerable to remote unauthenticated DoS by making them transit IPv6 packets to many destination addresses). (edited)

Marek Isalski 1 hour ago
So instead I set them to be big enough that I don't have to worry for the next 2-3 years, based on the rate of growth of the DFZ currently. If somebody needs more routes (e.g. they actually have millions of /32s on their internal network), they obvious can sysctl it themselves. But to me, it seemed a sensible default to start with a router which is "production-ready" for the DFZ, while not potentially being able to DoS itself through RAM exhaustion.

Marek Isalski 1 hour ago
The "cache" entries are cleaned up when the IPv6 table is full (and also periodically)… but there is still the limit of not being able to install more than max_size routes in the kernel — whether they've come from FRR, kernel routes for interfaces, or manually via ip route add

https://vincent.bernat.ch/en/blog/2017-ipv6-route-lookup-linux

Just to confirm, increasing the route,max_size fixed this issue completely. I think it can be closed. But maybe we should set these settings by default before closing this.

related to T1699