@Viacheslav it happened yesterday again but the stack trace was different. This time it was complaining that BGPD did not respond and the frr watch process tried to restart it, which of course did not help the situation.
I will continue to monitor but i think we can close this issue and wait for more details when it happens again.

Just to confirm, increasing the route,max_size fixed this issue completely. I think it can be closed. But maybe we should set these settings by default before closing this.

Some statistics from ipv6 bgp summary

@thomas-mangin the sessions are still stable, for 7 days now. The only thing changed was that max_size limit. Also no packetloss on the IPv6 connections has been observed during this time.

@syncer Re-opening this. Had the first exact same incident on a different router with an IPv6 BGP session on a RJ45 connection, so that would rule out any issues with the Intel X710 card in relation to this issue.

We ave no RPKI filtering active yet, so https://github.com/FRRouting/frr/issues/5458 seems not related.

This issue exists in firmware 6.8 and 7.1.
Starting with 7.1 i can see that the disable command 'sudo ethtool --set-priv-flags <interface name> disable-fw-lldp on' also works.

Found that in drivers 2.3.6 and newer this should also work:

Upgraded the firmware of the X710 adapters from 6.0 to 6.8, waiting for Dell to get 7.0 and 7.2 ready. But for now the sessions are 18 hours stable so little optimistic that it was a firmware issue and not BGPd causing issues

Trying to find more information with debugging settings of zebra process.
Apr 12 23:52:18 router zebra[1472]: 0:2404:5780:3::/48: Route install failed
Apr 12 23:52:18 router zebra[1472]: 0:2404:5780:3::/48 Stale dplane result for old_re 0x555f6166b300
Apr 12 23:52:18 router zebra[1472]: 0:2404:5780:3::/48 Processing dplane ctx 0x555f765ff7a0, op ROUTE_UPDATE result FAILURE

My main question is why is this message displayed and do we need to worry.

I have had the maximum-paths setting for years since Vyos 1.1.x and I have a lot of routes ipv4 and ipv6 installed in the routing table with 2 or 3 routes even if they are not the same. I am not specifically using ecmp I just have multiple routes for fast failover.

I have the following:
set protocols bgp as maximum-paths ebgp '3'
set protocols bgp as maximum-paths ibgp '3'

After receiving
zebra[1507]: 0:2804:fa0:8000::/33: Route install failed

A router reboot last week reminded me to never to write mem in vtysh (but after looking it was automatic bij me :( )
The router booted with the configuration in FRR already loaded, and then Vyos tried to populate FRR based on the Vyos configuration and everything was broken :-)
It didn't help that the configuration i saved in FRR was a couple of months old.

@hagbard

set service snmp community dummycomm authorization 'ro'
set service snmp community dummycomm client '8.8.8.8'
set service snmp community dummycomm client '8.8.4.4'
set service snmp contact '[email protected]'
set service snmp location 'Datacenter, City, Country'

@hagbard i tried testing by installing the package.
The service is running but not working correctly.
The following is shown:
Jan 07 10:25:54 server snmpd[9979]: /etc/snmp/snmpd.conf: line 10: Warning: Unknown token: smuxpeer.
Jan 07 10:25:54 server snmpd[9979]: /etc/snmp/snmpd.conf: line 11: Warning: Unknown token: smuxpeer.
Jan 07 10:25:54 server snmpd[9979]: /etc/snmp/snmpd.conf: line 12: Warning: Unknown token: smuxsocket.
Jan 07 10:25:54 server snmpd[9979]: notificationEvent OID: linkUp
Jan 07 10:25:54 server snmpd[9979]: /etc/snmp/snmpd.conf: line 21: Error: unknown notification OID
Jan 07 10:25:54 server snmpd[9979]: notificationEvent OID: linkDown
Jan 07 10:25:54 server snmpd[9979]: /etc/snmp/snmpd.conf: line 22: Error: unknown notification OID
Jan 07 10:25:54 server snmpd[9979]: /etc/snmp/snmpd.conf: line 23: Warning: Unknown token: monitor.
Jan 07 10:25:54 server snmpd[9979]: /etc/snmp/snmpd.conf: line 24: Warning: Unknown token: monitor.
Jan 07 10:25:54 server snmpd[9979]: net-snmp: 2 error(s) in config file(s)

I have built the current branch this evening, version is 1.3.0-rolling because i named it that way.

Tested by installing the new package
https://packages.debian.org/bullseye/amd64/iperf

Most of the times when i try a second time it just works.

@jestabro i am not able to build the vyos-1x package because of dependencies on other packages.
Can we include them in rolling so i can test tomorrow?

@jestabro i have encountered the first situation in my networks where i really need RFC-Complaint VRRP
(Some devices do not learn the MAC-address on the VRRP gateway, it works for some time and then stops).

Ok so that would mean the BGP4 info is exposed to the SNMP server and someone has to check it on a client.

@cpo the SNMP server has to support the MIB to export the OID. And afterwards the client has to have an up2data MIB to map it again. If the server does not export it the client can update the MIB but the OID won’t be there.

I was reading old documentation. It does work once I set it with the configuration instead of editing the file directly.

@dmbaturin did you find time to check the pull requests? Checked with latest crux build and the issue still exists.
I can test this if needed.

In 1.2.3 build this error does not appear and it seems to work correctly

You have to add a sync-group.
set high-availability vrrp sync-group intgroup member int1
set service conntrack-sync failover-mechanism vrrp sync-group intgroup

I have prepared the change
https://github.com/mevertse/vyatta-cfg-quagga/commit/d3685f504e0c22a8c88899aebc1e8705637259c4

This router is receiving BGP from several internal BGP routers each with full table peers or couple of peerings.

This one is running on Hyper-V 2016 and is not pushing any traffic. It is my test router and experimenting with RPKI.
The routers doing traffic are on hardware and not running 1.2.x yet.

I am running 1.2.1 compiled on 17-04-2019, uptime is 6 days without issue.
RIB entries 1366663, using 209 MiB of memory
Peers 16, using 330 KiB of memory
Peer groups 4, using 256 bytes of memory

Relevant config:

With 1.2.0-H4 this issue seems to be fixed on my router.

Do you have an iso to test? I tried latest rolling and also my own iso built from current and i continue to see this issue.
It makes transitioning to 1.2.0 impossible at this moment. Still at 1.1.8 on the routers.

Well that is interesting, the package versions are still the same but now it works.

I tested the current release and the issue still exists.
After adding staticd=yes on a reboot everything seemed to work.
More specific routes work even when a larger blackhole route is present.
However adding a new blackhole route while more specific routes exist (and work fine) stops them from working. The new blackhole routes get loaded and supress the more specific routes.
Removed a blackhole route allow other routes to work even when they are not part of the blackhole route.

The only packages i manually build are vyatta-cfg-firewall and vyatta-nat, the rest of the packages directories are empty so the packages get downloaded.

@kroy by using chroot and trying to install the vyatta-cfg-quagga package i found out what is causing my build iso error:

@Maltahl just re-tested this, with the staticd=yes added, and a reboot done.
When i add two static routes i would expect the /24 route to work because it is more specific. But is does not and show ip route shows only the /23 blackhole.

@Maltahl for me it was not fixed with that addition, and i read above that others had this as well.

@kroy everything is at current, except 'frr' because then i get 7.1dev and i would like 6.0.2 to test if this solved it.
I used debian/master branch from FRR.

I am experiencing the same issues with a router i tested with 1.2.0 current.
Can we create a test release going back to FRR 6.0.2?

@zsdc i meant test with 1.2.0 :-)

We are seeing this issue mostly on BGP routers with Internet Exchange connections because at a reboot we are hitting max-prefix limits with a lot of peers.
At this moment it is not possible to upgrade to latest 1.2.0, still running 1.1.8.

@hagbard Did you merge the second PR also? For vyos/vyatta-nat?

@hagbard created an iso image and loaded it in a VM. I can add the configuration and at commit the right ip6tables rules are created.

@hagbard the changes are created with the patch files mentioned earlier.
I am in the process of creating packages and an iso with it.

@hagbard PRs created, first time so hope its done right.

@jmlccdmd
I added a second router and configured conntrack-sync.
Failover and preempt failback works correct.
Both routers show statistics for the firewall rules

But if you run only on the first router, including the VRRP setup it does not work?

@jmlccdmd
I have recreated your setup with Vyos 1.2.0-rc10 and it seems to be working correctly

After some searching i found that with the following commands it works:

@hagbard
Test setup created

I can probably create a test setup this week and test the normal implementation in NPTv6 te see what is not working in my production setups.
As i do not have a build server at this moment creating a PR would be a bit difficult, the patch files are included in the comments above.

Then we do not have the same setup :-)
set interfaces ethernet eth0 address 'x.x.x.225/24'
set interfaces ethernet eth0 address 'x:x:x:x::225/64'
set interfaces ethernet eth1 address '10.0.201.1/24'
set interfaces ethernet eth1 address 'fd00:10:0:201::1/64'
set nat nptv6 rule 10 outbound-interface 'eth0'
set nat nptv6 rule 10 source prefix 'fd00:10:0:201::/64'
set nat nptv6 rule 10 translation prefix 'x:x:x:x::/64'
set protocols static route 0.0.0.0/0 next-hop x.x.x.1
set protocols static route6 ::/0 next-hop x:x:x:x::1

@hagbard What do you want me to try. I downloaded and loaded that rolling image and i do not see the proposed patches in it.
I rebooted the router and it did not work.

According to FRR this is normal behavior.
https://github.com/FRRouting/frr/pull/3044

Do you mean the 31 and 32 also couldn’t ping eachother?

https://www.shrubbery.net/mibs/BGP4-MIB.txt
We can start with this like @rherold suggests since FRR supports BGP4 MIB.

Seems duplicate with https://phabricator.vyos.net/T366
If we use the Cisco BGP MIBv2 we solve both issues.

I see in the config that you do not have an interface IP on the VRRP members.
This works in 1.1.8 most of the time. But can you test if 1.2.0 works with those added. The hello source address is not needed then and the chances are the kernel wil load the connected route this way.

Solved by disabling engine ID when the version is 9, not sure if this is enough but on my router it works.

Until version 1.7.0 it was possible to (mistakenly) configure the
NetFlow v9 SourceID field/IPFIX Observation Domain ID with the old
NetFlow v5 jargon, ie. '1:1'. This is now threated as invalid and
a positive 32-bit number, ie. '100000', is expected. If exporting
NetFlow v5, nothing changed: the Engine ID/Engine Type input, ie.
'1:1', is still valid and expected.

This behavior was already present in the old Quagga implementation in Vyos 1.1.7.
As a workaround we always shutdown the peers when doing a planned reboot.

Hi, yeah i was planning on adding it later. Thanks.

@dmbaturin I believe you forgot to create the Sync-Group. The following configuration is working, and it is really nice to see how this got created during migration from Vyos 1.1.8, and to finally have IPv6 in the VRRP configuration.

One extra patch needed

SInce this is a 1-to-1 mapping it also opens up all incoming ports to the hosts behind the router.

It was at 2GB because it is a test router running in the production network.
I increased memory and will check to see if this resolves it.

@syncer
It is a virtual machine on Hyper-V 2016 with 2 cores, 2GB memory.
I compared the output from top over te past week and the only changes are in the uacctd processes.

With the following patch the statements are migrated. Needs some work to also migrate the metric and route-map settings. Will try to do that later.

The idea for 'helper ftp' is a bit harder to implement, because it seems it requires iptables -t raw and currently we only have filter and mangle.

Since during parsing we cannot detect if it is FTP traffic or not, because you can choose whatever port you want, i think the only solution would be to add something like

Probable cause is:
https://unix.stackexchange.com/questions/385693/iptables-nf-conntrack-ftp-not-working-under-debian

Upgrade to 1.2.0-rc10 and BGP is still working fine. It starts at boot and loads all BGP peers and several full tables.

With some small changes i changed the scripts to create the proposed NETMAP rules when editing the NPTv6 rules in the configuration. For me this is working and surviving reboot.
I can add the diff files if anyone wants them.

Re-tested with 1.2.0-rc9 with the same result.
I found that i can enable debug in /opt/vyatta/sbin/vyos-update-nptv6.pl to see what is added and deleted.

This seems to be easy.
sudo chmod 776 /opt/vyatta/etc/config/archive/config.boot