Using native nft tooling does not seem to help, I will keep investigating - seems to be very little information online on the topic, mostly people having the same issue as me.

@Viacheslav

I have conducted additional testing. I can replicate the issue on a fresh VM. It seems the issue arises if the DHCP server is slow to respond upon bootup.

Same results here. If I remove the routes and re-add them they work perfectly. The issue only comes up at reboot.

Issue is back in 1.4.0-rc3

Confirmed working properly in 1.4.0-rc1

I checked 6 routers, 2 showed the issue. No idea what the impact is. All of them running 1.3.2.

I could have sworn it took quite a while for the wlb.out to be populated when testing earlier on. However as I´ve built out the config a bit more I am now only able to reproduce it a few seconds after reboot - so your theory holds. Let's close out the bug report for now and I´ll reopen if something changes down the line.

Further testing reveals that re-connecting the interfaces after reboot solves the issue:

https://github.com/FRRouting/frr/pull/12364
riw777 merged commit 91b6db4 into FRRouting:master Feb 14, 2023

In T4466#124827, @florin wrote:

https://drive.google.com/file/d/1-5G8UPZfw0UJalLJKPVkzoA6AKC5k7Lm/view?usp=sharing

Wow, well done! You don't happen to have an ISO you could share?

Did you then end up with a fully working nic, bridging included?

Confirmed working in 1.3.0 LTS.

Can also confirm that 1.3.0-epa3 is broken, so something must have changed between epa3 and 202112180443.

Tried 1.3-beta-202112180443, seems to be working as it should now.

Tested against 1.3.0-EPA3, same problem.

Thanks!

sudo journalctl -b

lspci -knn

@c-po I expected this to go into 1.2.7 but it did not. How would I tag this properly so it hopefully makes it into 1.2.8?

https://github.com/vyos/vyatta-cfg-quagga/pull/60

https://github.com/vyos/vyatta-cfg-quagga/pull/59

This has come up multiple times before, see https://phabricator.vyos.net/T1698 for the solution.

Reworking my config so that it uses eth0->eth3 instead of eth1->eth4 makes everything work as expected. So something has clearly changed regarding the interface naming/creation logic.

VyOS added those during the 2nd boot after upgrade. I would assume bug related to the fact that my config doesn't include eth0.

In T2523#65310, @c-po wrote:

Why is there no eth0 on VyOS 1.2.5?

Bump

@syncer Any chance of getting this merged into 1.2.2?

Any chance of getting this merged into 1.2.2?

In T1148#33158, @dmbaturin wrote:

@danhusan IPv6 should not be affected. Workaround for IPv4:

Small config, just 4 interfaces with IPv4 and IPv6 + some BGP config. I am running the VyOS instance in ESXi with some fairly modern hardware.
Unfortunately I cannot just reboot this device at will. If you provide your email I can send over the config.

In T1160#31671, @c-po wrote:

But what should be the desired behavior?

• When no network or client is speciefied we allow allo but as soon as one network/client is specified we limit it down?

No need to disable VNC :-) The FRR package should be fixed already so just need to make sure it is in EPA3.

That seems odd, my global peerings all reset at the time of the test. @mariusno and @rherold did you experience something similar?

Wow, this explains why all my sessions dropped yesterday.

RC11 is getting old, please retry with latest rolling: https://downloads.vyos.io/rolling/current/amd64/vyos-1.2.0-rolling%2B201901040337-amd64.iso

IPv6 seems to have the same issue. Peer shutdown in configuration, reboot, results below:

@c-po https://github.com/vyos/vyos-build/pull/35 is also needed. Without it upgrades (install image ...) will fail.

Added new pull requests, built and tested, working fine.

In T1120#29573, @c-po wrote:

instead of differentiating between raid and non raid installations - why not always wait 5 seconds for the discs to settle? As this is only done once on startup this is IMHO better then a special case.

https://github.com/vyos/vyatta-cfg-system/pull/97

And FYI max-ipv6-routes=10 and max-ipv4-routes=10 doesn't seem to help either.

In T1131#29483, @hagbard wrote:

Hi @danhusan, did you ever try another poll value, like 3 secs or 5 or anything like that? If set to 0, the host system won't show you any updated meta data, like if you change the ip address etc.
(https://github.com/vmware/open-vm-tools/blob/master/open-vm-tools/services/plugins/guestInfo/guestInfoServer.c#L1662)
I'm therefore not entirely sure if that should be treated as a special case scenario (we could publish a kb if you run into that condition), or if it is a general issue since you 2 were the only ones experience that issue as far as I know.
I'm also not sure it only is triggered by your situation (full bgp table) or if it can happen on other occasions as well, if you came across more issues regarding that value, please let me know.

PR https://github.com/vyos/vyatta-cfg-system/pull/96
Thanks @kroy

It is vmware-tools collecting all the routes from VyOS and reporting them to the hypervisor so you can see them in the ESXi GUI. Not really optimal for full tables.

Do you have a full bgp table on this box?

IIRC I worked around this by editing /etc/vmware-tools/tools.conf and adding:

And FYI, just to make sure that something wasn't triggered by hanging out in GRUB.
rootdelay=0 fails
rootdelay=2 works, so just a small pause is needed on my system at least.

@kroy you are clearly on a roll today, rootdelay=10 also did the trick.

Spot on, booting with acpi=off makes it come up properly and configurations are actually saved.
Still getting some errors but not sure if they matter or not:

Some data I wasn't able to find in any log files:

https://forum.vyos.io/t/vyos-1-2-0rc10-raid-1-fresh-install-unable-to-save-config/3059
These users might have the same issue. I noticed one of the symptoms of having the system is in this state is that the config is lost between reboots.

I am not able to reproduce it in vmware workstation.

I can let it sit forever, the status wont change as the array is missing one disk. Notice how after (auto read only) raid1 is only says sdb1[0] (sda1 is missing).
Also we can see that this wont fix itself from looking at the df -h output where the system clearly has booted straight from sda1 instead of from the raid array (/dev/md127)

After reboot, running from local disks:

During install (booted from ISO):