Page MenuHomeVyOS Platform

Policy Based Routing with DHCP Interface Issue
Closed, ResolvedPublicBUG

Description

In my setup I have two ISPs. One is static and the other is dynamic. I was using VyOS_1.4-rolling-202101171022 for a while and things worked fine until one day this no longer worked. I installed VyOS_1.4-rolling-202106180936 and had the same issue. Then I tried VyOS_1.3-beta-202106180642 and it worked again. I upgraded to vyos-1.3.0-rc6-amd64 and it continued working. I compiled vyos-1.3.0-amd64.iso and installed it and it lo longer works. I rolled back to vyos-1.3.0-rc6-amd64 and it worked as expected. I never saw epa releases posted, so I'm not sure at what point after 1.3.0 rc6 this feature broke.

The issue when it doesn't work is that the table is not updated with the route for the dynamic connection. You can see what it should look like from the 'show ip route table 111' command below. When it's not working the table is blank. I have the relevant config below. I have removed IP specifics from the outputs.

falkor@nenyas-edge-01:~$ show ip route table 111
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF default table 111:
S>* 0.0.0.0/0 [1/0] via x.x.x.1, bond1.111, weight 1, 00:28:40
falkor@nenyas-edge-01:~$


falkor@nenyas-edge-01:~$ show conf commands | match 'network-group (FIOS|WOW)_01_INET_NETS network|policy route|table'
set firewall group network-group FIOS_01_INET_NETS network '192.168.x.0/24'
set firewall group network-group FIOS_01_INET_NETS network '10.x.x.0/24'
set firewall group network-group FIOS_01_INET_NETS network '10.x.x.0/24'
set firewall group network-group FIOS_01_INET_NETS network '10.x.x.0/24'
set firewall group network-group FIOS_01_INET_NETS network '10.x.x.0/24'
set firewall group network-group FIOS_01_INET_NETS network '192.168.x.0/24'
set firewall group network-group WOW_01_INET_NETS network 'x.x.x.x/29'
set firewall group network-group WOW_01_INET_NETS network 'x.x.x.x/29'
set firewall group network-group WOW_01_INET_NETS network 'x.x.x.x/27'
set firewall group network-group WOW_01_INET_NETS network 'x.x.x.x/27'
set firewall group network-group WOW_01_INET_NETS network 'x.x.x.x/28'
set interfaces bonding bond1 vif 211 policy route 'FIOS_01_INET'
set interfaces bonding bond1 vif 221 policy route 'WOW_01_INET'
set policy route FIOS_01_INET rule 1000 description 'Route traffic to ISP Modem (192.168.x.x) to ISP Interface'
set policy route FIOS_01_INET rule 1000 destination address '192.168.x.0/24'
set policy route FIOS_01_INET rule 1000 disable
set policy route FIOS_01_INET rule 1000 set table 'main'
set policy route FIOS_01_INET rule 1001 description 'Route traffic from the specified subnets through FIOS_01_INET'
set policy route FIOS_01_INET rule 1001 set table '111'
set policy route FIOS_01_INET rule 1001 source group network-group 'FIOS_01_INET_NETS'
set policy route WOW_01_INET rule 1001 description 'Route traffic from the specified subnets through WOW_01_INET'
set policy route WOW_01_INET rule 1001 set table '121'
set policy route WOW_01_INET rule 1001 source group network-group 'WOW_01_INET_NETS'
set protocols static table 111 route 0.0.0.0/0 dhcp-interface 'bond1.111'
set protocols static table 121 route 0.0.0.0/0 next-hop x.x.x.x
falkor@nenyas-edge-01:~$

Details

Difficulty level
Unknown (require assessment)
Version
VyOS 1.3.0
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Perfectly compatible
Issue type
Bug (incorrect behavior)

Event Timeline

@Rhongomiant Am I understanding correctly that you don't see the default route in table 111?

The main reason:

vyos@r4# set protocols static table 111 route 0.0.0.0/0 dhcp-interface eth2
[edit]
vyos@r4# commit
[ protocols static table 111 route 0.0.0.0/0 dhcp-interface eth2 ]
dc: can't open '0x7fffffff': No such file or directory
Error: syntax error, unexpected newline
insert rule ip mangle OUTPUT ip saddr 192.168.100.226 counter meta mark set
                                                                           ^

[[protocols static]] failed
Commit failed

It should be fixed in commit

vyos@r4:~$ show ip route table 111
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF default table 111:
S>* 0.0.0.0/0 [1/0] via 192.168.100.1, eth2, weight 1, 00:00:01
vyos@r4:~$

Try to build a new 1.3 image or do changes as mentioned in commit

I don't know what I'm building. How can I be sure I'm actually building 1.3.0 rather than 1.4? I ask because when I boot off the build I compiled I get the following message at the start of the boot process. Is it 1.3.0 or sagitta (1.4)?

Welcome to VyOS 1.3.0 (sagitta)!

I'm using the guide at the following page starting from the Native Build section.

I have a VM with Debian Buster installed with the latest updates.

I ran the following commands to build an image.

git clone -b current --single-branch https://github.com/vyos/vyos-build
cd vyos-build
docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos/vyos-build:current bash

In the docker instance I run the following.

./configure --architecture amd64 --build-by "Me" --build-type release --version 1.3.0
sudo make iso

Upon first boot I noticed things were not working as expected and rebooted. Upon reboot I noticed the failed message below. I'm not sure what failed as I couldn't find an error in syslog.

[ OK ] Finished Permit User Sessions.
[ 34.284286] vyos-router[699]: Waiting for NICs to settle down: settled in 4sec..
[ 51.833268] vyos-router[832]: Started watchfrr.
[ 61.266446] vyos-router[699]: Mounting VyOS Config...done.
[ 171.164605] vyos-router[699]: Starting VyOS router: migrate configure
[ 171.173956] vyos-router[5659]: failed!
[ 171.486425] vyos-config[2120]: Configuration error

Welcome to VyOS - nenyas-edge-01 ttyS0

nenyas-edge-01 login:

The issues this time was that the IP never showed up for my FiOS connection in show interfaces and the route was not added to the vrf table. The IP for the FiOS interface is obtained via DHCP. Eventually traffic was being passed on the interface and a route showed up for it in the main table, but the IP still didn't show up in show interfaces and the route didn't show up in the vrf table. I deleted the configuration line below, committed, added the line back and committed. I didn't get an error, but the route still didn't show in the vrf table.

set protocols static table 111 route 0.0.0.0/0 dhcp-interface 'bond1.111'

dmbaturin changed Is it a breaking change? from Unspecified (possibly destroys the router) to Perfectly compatible.
dmbaturin added a subscriber: dmbaturin.

I have confirmed that this issue is now resolved when building from equuleus. I've attached a screenshot showing the table for the dynamic interface has a default route after the interface get's an IP. I used the following commands to build the ISO.

git clone -b equuleus --single-branch https://github.com/vyos/vyos-build

docker run --rm -it --privileged -v $(pwd):/vyos -w /vyos vyos/vyos-build:equuleus bash

./configure --architecture amd64 --build-by "Rhongomiant" --build-type release --version "1.3.1-LTS-20220822-054024UTC"

sudo make iso

image.png (340×868 px, 18 KB)