This feature now is in the Cloud-init for 1.3 and must be backported after testing.

@kroy how about testing this in 1.3? It must work now.

Tested with:

set service dns dynamic interface eth0.203 service custom host-name 'test.vyos.net'
set service dns dynamic interface eth0.203 service custom login 'vyos'
set service dns dynamic interface eth0.203 service custom password 'vyos'
set service dns dynamic interface eth0.203 service custom protocol 'dyndns2'
set service dns dynamic interface eth0.203 service custom server 'vyos.io'

This also happens on service deletion

@Viacheslav it happened yesterday again but the stack trace was different. This time it was complaining that BGPD did not respond and the frr watch process tried to restart it, which of course did not help the situation.
I will continue to monitor but i think we can close this issue and wait for more details when it happens again.

Even with customers routes redistributed by OSPF instead of iBGP, it has just crashed again:

I tried unit-cache earlier but it seems to have issues too - I've seen duplicate routes if the same client (all have static IP assigned by RADIUS based on username) connects to a different PPPoE server and the old route is not removed, as if the cached (not removed) PPPoE interfaces were not seen as removed in FRR. But I haven't investigated this in more detail as it's a production setup, can't experiment too much on live customers.
I'm considering if I could go back to redistributing PPPoE customers /32 routes in OSPF instead of iBGP - it has been that way for a few years (using MikroTik, before moving to VyOS), but I've recently changed it following "BGP Best Current Practices" http://www.bgp4all.com.au/pfs/_media/workshops/05-bgp-bcp.pdf which recommends using OSPF only for infrastructure, not customers - seems logical to me as BGP was designed for much larger routing tables (all of the Internet), but perhaps OSPF is still good enough for just a few hundreds of customers.

Hello @marekm, I think [ppp]unit-cache=n might help in this case, but the main issue in FRR. Do you want a package for the test with these improvements?

unit-cache=n
By default is disabled: unit-cache=0

I've just had two different routers (one bare metal and one VM) crash roughly at the same time, triggered by many PPPoE sessions disconnecting at the same time due to a short power failure (routers itself had power all the time, but power was interrupted for about a minute to a switch on the network between the routers and PPPoE clients). Stack traces are very similar (absolute addresses differ, but the same functions and offsets in them). And again, each time watchfrr restarted bgpd but it was not working until reboot. No problems so far with two other BGP routers running a similar configu but without any dynamic interfaces (only OSPF and BGP, no PPPoE servers).

It looks like that the build process messed it up, it did create the version file at the beginning of the build, not at the end. After the file usr/share/vyos/version.json was create, pkg installations took place a few minutes alter, that's why everything in the image is newer than the version file, therefore the command output is absolutely correct. I'll check if I can find out what went wrong during the build, since it appears that only 1.2.6 is affected.

In T2820#74102, @Viacheslav wrote:

@marekm
Can you check your BGP configuration if "router-id" is declared?
Also, what is with interface names?

ppp-lot29
ppp-jmg22
ppp-rol81
ppp-rod8

Do you use scripts with renaming? How to reproduce it?

/usr/libexec/vyos/op_mode/version.py:
Built on: Thu 13 Aug 2020 11:57 UTC

Happens also when just using the booted image without install. Investigating.

It crashed again after 5 days in 1.2.6-epa1, in the same function, also when a dynamic PPPoE interface was deleted.
It happens less frequently after the former customers who repeatedly failed authentication have been physically disconnected.
Again, BGP no longer works after watchfrr has restarted the bgpd process. All works again after reboot.

Is probably fixed in https://phabricator.vyos.net/T1291 according to cpo on slack

Maybe related - https://github.com/FRRouting/frr/issues/6439

This happens only when in config before migration exists nodes system 'ntp' without other params.

Fixed int the latest rolling, VyOS 1.3-rolling-202008110118

I am giving up with HFSC. I have been studying it for a long time, I have tested it in many different ways, without VyOS too. The only thing I have found is that this is is not a problem of VyOS.

related to T1699

@Merijn Have such problems been repeated?

The issue did not reproduce neither in 1.2.5 nor in 1.3 version.
Try in the new release and re-open the ticket if any new information appeared.

@olofl I can't confirm this bug int the 1.2.5 LTS version.

I can't confirm this bug.

vyos@vyos# set vpn ipsec ike-group IKEv2_DEFAULT mobike disable 
[edit]
vyos@vyos# commit
[edit]
vyos@vyos# run show version 
Version:          VyOS 1.2.5
Built by:         Sentrium S.L.
Built on:         Sun 12 Apr 2020 15:18 UTC
Build UUID:       1695c660-d785-4b16-a54b-66d6a02ea24f
Build Commit ID:  48cc9fc46569e6

I thought the problem could be related to VyOS, but now I have found the same behavior when using directly tc hfsc.

show interfaces vrrp

I have tried the above scenario in the VyOS 1.3-rolling-202007200117 which is the latest version and the issue did not reproduce. So I would request you to try in the latest version and share your feedback.

https://github.com/vyos/vyatta-cfg-vpn/pull/34

Can confirm that the image now contains wireguard.ko.

Understood, will re-check and close the report once confirmed.

@linuxgemini we do not support DKMS.

Did a rebuild and found the silent error:

While I was checking my old build, I have seen this on /live/filesystem.packages:

This command doing not what you are expecting. It shows virtual VRRP interfaces running in RFC3768 compatibility mode. Add the rfc3768-compatibility option to a VRRP group and a new virtual interface should be listed in the output.
If you want to change this behavior, please describe how exactly.

Possible by backporting https://github.com/vyos/vyos-1x/pull/452 and https://github.com/vyos/vyatta-cfg-system/pull/125 though I think some code using Config would need to be modified - add .exists calls before each .return_value(s) - 1.3's vyos.config doesn't require them, 1.2's I think does.

Ok, nice. And how about 1.2? Is it possible to fix this with the 1.2.6 release?

This is already fixed in 1.3

The user-data dir actually is preserved on upgrade, it's just the check that is faulty. Need to look into it.

It looks like this bug is in the kernel.
1.2.5 - 4.19.106-amd64

No, seems to be fixed! I was pretty sure it was upstream, must be resolved now.

Could anyone test if it's still reproducible?

When googling on the error given, T109 shows up where I had posted about this in 2018. I'm not sure it's related to this. Im not sure any configuration has been lost on reboot.

https://phabricator.vyos.net/T1564

Also in 1.2.5

vyos@vyos:~$ show protocols bfd peer 10.203.42.1 
% Unknown command: show bfd     peer 10.203.42.1 local-address 10.203.42.254 vrf default
vyos@vyos:~$ 
vyos@vyos:~$ show protocols bfd peer 10.203.42.1 counters 
% Unknown command: show bfd     peer 10.203.42.1 local-address 10.203.42.254 vrf default counters
vyos@vyos:~$ 
vyos@vyos:~$ 
vyos@vyos:~$ show version 
Version:          VyOS 1.2.5