Page MenuHomeVyOS Platform
Create Task
Maniphest T3172

Builds sometime after 2020-12-17 have broken routing after reboot
Closed, ResolvedPublicBUG
Actions

Description

Confirmed on the 2020-12-31 build too.

Take this config:

 route 0.0.0.0/0 {
     next-hop x.x.x.x {
     }
 }

table 146 {
     interface-route 0.0.0.0/0 {
         next-hop-interface tun2 {
         }
     }
 }
 table 200 {
     interface-route 0.0.0.0/0 {
         next-hop-interface wg4 {
         }
     }
 }

Results in

S>* 0.0.0.0/0 [1/0] is directly connected, tun2, weight 1, 00:03:46
  *                 is directly connected, wg4, weight 1, 00:03:46
  *                 via x.x.x.x, eth0.7, weight 1, 00:03:46 (normal default gateway)

That's bad, it should look like:

S>* 0.0.0.0/0 [1/0] via x.x.x.x, eth0.7, 02:35:00

After a reboot, this is what vtysh looks like (missing the table options)

ip route 0.0.0.0/0 x.x.x.x
ip route 0.0.0.0/0 wg4
ip route 0.0.0.0/0 tun2

If I delete the two table X configs and readd them, the appropriate config happens in vtysh

ip route 0.0.0.0/0 tun2 table 146
ip route 0.0.0.0/0 wg4 table 200
ip route 0.0.0.0/0 x.x.x.x

So something happens during a reboot that prevents the table from getting added to FRR, but it works fine if configing from the CLI (until a reboot)

Details

Difficulty level
Unknown (require assessment)
Version
1.3
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Related Objects
Search...

Event Timeline

kroy renamed this task from Builds sometime after 2020-12-17 have broken routing to Builds sometime after 2020-12-17 have broken routing after reboot.Dec 31 2020, 7:43 PM
kroy created this task.
jestabro added a subscriber: jestabro.Dec 31 2020, 8:17 PM
pasik added a subscriber: pasik.Jan 1 2021, 5:15 PM
c-po added a subscriber: c-po.EditedJan 2 2021, 12:34 PM

It feels like a bug which we received from upgrading to FRR 7.5 series.

https://github.com/FRRouting/frr/issues/7806

Running the VyOS code from vyatta-cfg-quagga repo with set -x yields the following commands send to FRR:

vyos@vyos# vtysh -c 'configure terminal' -c 'ip route 0.0.0.0/0 eth1 table 146'
vyos@vyos# vtysh -c 'configure terminal' -c 'ip route 0.0.0.0/0 vtun10 table 200'

vyos@vyos# vtysh -c "show run" | grep 0.0.0.0/0
ip route 0.0.0.0/0 vtun10 table 200
ip route 0.0.0.0/0 eth1 table 200
ip route 0.0.0.0/0 172.18.201.254 10

Running the exact same commands on FRR 7.3 series shows:

vyos@vyos# vtysh -c "show run" | grep 0.0.0.0/0
ip route 0.0.0.0/0 vtun10 table 200
ip route 0.0.0.0/0 eth1 table 146
ip route 0.0.0.0/0 172.18.201.254 10
kroy added a comment.Jan 2 2021, 10:17 PM

Just to add to the knowledge surrounding this issue...

The reason it worked when I deleted/readded the commands after boot, because it is order dependent. So really it was fixing itself by accident, and that's not really a temporary bandaid for anyone until the source FRR is fixed.

c-po added a parent task: T3182: Main blocker Task for FRR 7.4/7.5 series update.Jan 3 2021, 4:51 PM
hensur added a subscriber: hensur.Jan 9 2021, 12:56 PM
c-po closed this task as Resolved.Jan 16 2021, 8:05 AM

Reverted to 7.3 let's wait for the fix to pop up in 7.5 FRR branch and we can try again.

c-po added a comment.Feb 14 2021, 7:48 AM

Please re-test with latest current rolling release. Reverted back for FRR 7.5 and that bug should have been fixed in there.

Cheeze_It added a subscriber: Cheeze_It.EditedFeb 17 2021, 2:10 AM

Here is some more testing. Currently on FRR 7.5:

vyos@vyos:~$ show version frr
FRRouting 7.5-20210212-00-gbc460d132 (vyos).

VyOS version is:

vyos@vyos:~$ show version

Version:          VyOS 1.4-rolling-202102141111
Release Train:    sagitta

Here's my config:

vyos@vyos:~$ show configuration commands | match static
set protocols static route 10.255.255.255/32 interface eth0
set protocols static table 100 route 10.255.255.255/32 interface eth0
set protocols static table 101 route 10.255.255.255/32 interface eth0
set protocols static table 102 route 10.255.255.255/32 interface eth0
set protocols static table 103 route 10.255.255.255/32 interface eth0
set protocols static table 104 route 10.255.255.255/32 interface eth0

Here's how the routing table looks:

vyos@vyos:~$ show ip route static
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

S>* 10.255.255.255/32 [1/0] is directly connected, eth0, weight 1, 00:02:03

Here's how the other routing tables look:

vyos@vyos:~$ show ip route table 100
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF default table 100:
S>* 10.255.255.255/32 [1/0] is directly connected, eth0, weight 1, 00:02:30
vyos@vyos:~$ show ip route table 101
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF default table 101:
S>* 10.255.255.255/32 [1/0] is directly connected, eth0, weight 1, 00:02:31
vyos@vyos:~$ show ip route table 102
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF default table 102:
S>* 10.255.255.255/32 [1/0] is directly connected, eth0, weight 1, 00:02:32
vyos@vyos:~$ show ip route table 103
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF default table 103:
S>* 10.255.255.255/32 [1/0] is directly connected, eth0, weight 1, 00:02:33
vyos@vyos:~$ show ip route table 104
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF default table 104:
S>* 10.255.255.255/32 [1/0] is directly connected, eth0, weight 1, 00:02:34

Reboot and login:

login as: vyos
Pre-authentication banner message from server:
| Welcome to VyOS
End of banner message from server
[email protected]'s password:
Linux vyos 5.10.14-amd64-vyos #1 SMP Thu Feb 11 19:53:47 UTC 2021 x86_64

The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Wed Feb 17 01:56:55 2021 from 10.0.0.2


vyos@vyos:~$
vyos@vyos:~$
vyos@vyos:~$
vyos@vyos:~$
vyos@vyos:~$ show run

  Invalid command: show [run]

vyos@vyos:~$ show ip route static
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

S>* 10.255.255.255/32 [1/0] is directly connected, eth0, weight 1, 00:00:16

Checking the tables now:

vyos@vyos:~$ show ip route table 100
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF default table 100:
S>* 10.255.255.255/32 [1/0] is directly connected, eth0, weight 1, 00:01:50
vyos@vyos:~$ show ip route table 101
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF default table 101:
S>* 10.255.255.255/32 [1/0] is directly connected, eth0, weight 1, 00:01:52
vyos@vyos:~$ show ip route table 102
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF default table 102:
S>* 10.255.255.255/32 [1/0] is directly connected, eth0, weight 1, 00:01:53
vyos@vyos:~$ show ip route table 103
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF default table 103:
S>* 10.255.255.255/32 [1/0] is directly connected, eth0, weight 1, 00:01:54
vyos@vyos:~$ show ip route table 104
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF default table 104:
S>* 10.255.255.255/32 [1/0] is directly connected, eth0, weight 1, 00:01:54

It *seems* to work now with FRR on the tables with static routes:

vyos@vyos:~$ vtysh -c "show run" | grep 10.255.255.255/32
ip route 10.255.255.255/32 eth0 table 104
ip route 10.255.255.255/32 eth0 table 103
ip route 10.255.255.255/32 eth0 table 102
ip route 10.255.255.255/32 eth0 table 101
ip route 10.255.255.255/32 eth0 table 100
ip route 10.255.255.255/32 eth0
kroy added a comment.Feb 17 2021, 3:21 AM

Sorry. Yep. Can confirm, problem fixed.

c-po added a comment.Feb 17 2021, 10:26 AM

Perfect, thanks

erkin set Issue type to Bug (incorrect behavior).Aug 29 2021, 11:39 AM
syncer moved this task from Need Triage to Finished on the VyOS 1.3 Equuleus board.Nov 6 2021, 11:43 AM
syncer edited projects, added VyOS 1.3 Equuleus (1.3.0); removed VyOS 1.3 Equuleus.Aug 29 2022, 12:16 PM
syncer moved this task from Need Triage to Finished on the VyOS 1.3 Equuleus (1.3.0) board.