Hello all,
sometimes it's not possible to do dynamic routing because not all peers supports it.
As fallback static routes are used.
I would like to see the possibility to monitor static routes by some kind of health checks like ping.
(Like Palo Alto does)
It's not the same as WAN load balancing because the PBR would add other complexity.
Regards
Markus
Hi adestis, what you descripe is possible to do today with the help of a shellscript and the crontab, if you are interested i could help you create a script that does this for you, the one drawback is that the failover-time is in the ballpark of minutes, and the routes are not present in the configuration... Also, cron fills the log with messages every time it executed
Hello runar,
I know that it's possible to do it manually.
But I really would like to see a more integrated solution where you can add a check for the next hop inside the configuration.
A solution based on cron might be not so ideal because of the minimum time of 1 minute.
MikroTik RouterOS supports something like this:
/ip route add gateway=192.0.2.1,192.0.2.2 check-gateway=ping
or check-gateway=arp for boxes that don't ping very well.
It would be really nifty to find a way to add this to VyOS, but it would also have to interact well with FRR to ensure these "semi-static" routes propagate through to IGP/EGP where there is a redistribute static in effect.
Would it be reasonable to use BFD for this? Since BFD is already implemented we might be able to use that as well?
@Cheeze_It BFD for static routes would be nice as well but sometimes the target you test against is not under your control and/or does not support BFD.
@adestis yes, that is true....but that can be worked around. Any option can be used (either BFD, or ARP, or ICMP). I just wanted to give more ideas so that hopefully can get a working implementation for all 3.
So far I have seen that BFD for static routes in FRR is currently under development:
https://github.com/FRRouting/frr/issues/3369
(Seems like tests are only missing).
But so far I have not seen anything like @maznu mentioned what MikroTik has.
That really would be nice.
The way I was thinking is on this Juniper page here.
If you guys would like, I can mock it up in my lab, test it, and show you the configuration I used and maybe it would be possible for us to see if we can make something similar or at least with similar functionality.
PR https://github.com/vyos/vyos-1x/pull/1358
set protocols failover route 203.0.113.1/32 next-hop 192.168.100.1 check target '192.168.100.1' set protocols failover route 203.0.113.1/32 next-hop 192.168.100.1 check timeout '10' set protocols failover route 203.0.113.1/32 next-hop 192.168.100.1 check type 'icmp' set protocols failover route 203.0.113.1/32 next-hop 192.168.100.1 interface 'eth1' set protocols failover route 203.0.113.1/32 next-hop 192.168.100.1 metric '2'
At first look, at least it works, but it requires more tests and improvements
set protocols failover route 203.0.113.1/32 next-hop 192.168.122.1 check target '192.168.122.1' set protocols failover route 203.0.113.1/32 next-hop 192.168.122.1 check timeout '5' set protocols failover route 203.0.113.1/32 next-hop 192.168.122.1 check type 'icmp' set protocols failover route 203.0.113.1/32 next-hop 192.168.122.1 interface 'eth0'
show
vyos@r14:~$ show ip route 203.0.113.1 Routing entry for 203.0.113.1/32 Known via "kernel", distance 0, metric 1, best Last update 00:04:42 ago * 192.168.122.1, via eth0 vyos@r14:~$ vyos@r14:~$ vyos@r14:~$ sudo ip route show proto failover 203.0.113.1 via 192.168.122.1 dev eth0 metric 1 vyos@r14:~$ `
Hello everyone,
It works but has a little problem, if you set 2 routes to the same destination using different metrics and the main link dies it will change to the backup link but it wont change back to the main link when it come alive again, so you need to "kill" the backup link to make the main route active again.
Heres my tests:
IP 10.100.1.1/32 - Mikrotik Router Loopback
eth2 - main link - 172.25.30.9/30 (Vyos) - 172.25.30.10/30 (MK)
eth3 - backup link - 172.25.40.9/30 (Vyos) - 172.25.40.10/30 (MK)
route 10.100.1.1/32 {
next-hop 172.25.30.10 {
check {
target 172.25.30.10
timeout 1
type icmp
}
interface eth2
}
next-hop 172.25.40.10 {
check {
target 172.25.40.10
timeout 1
type icmp
}
interface eth3
metric 100
}
}
vyos@vyos:~$ show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
S>* 0.0.0.0/0 [210/0] via 10.10.1.254, eth0, weight 1, 1d23h01m
C>* 10.10.1.0/24 is directly connected, eth0, 1d23h01m
K>* 10.100.1.1/32 [0/1] via 172.25.30.10, eth2, 00:00:29
C>* 10.100.1.255/32 is directly connected, lo, 1d23h01m
C>* 10.250.250.0/30 is directly connected, vti10, 1d23h01m
C>* 172.20.30.0/24 is directly connected, eth1, 1d23h01m
C>* 172.25.30.8/30 is directly connected, eth2, 1d23h01m
C>* 172.25.40.8/30 is directly connected, eth3, 1d23h01mNow disabling the Mikrotik IP 172.25.30.10:
vyos@vyos:~$ ping 172.25.30.10
PING 172.25.30.10 (172.25.30.10) 56(84) bytes of data.
From 172.25.30.9 icmp_seq=1 Destination Host Unreachable
From 172.25.30.9 icmp_seq=2 Destination Host Unreachable
From 172.25.30.9 icmp_seq=3 Destination Host Unreachable
From 172.25.30.9 icmp_seq=4 Destination Host Unreachable
From 172.25.30.9 icmp_seq=5 Destination Host Unreachable
^C
--- 172.25.30.10 ping statistics ---
7 packets transmitted, 0 received, +5 errors, 100% packet loss, time 6134ms
pipe 3
vyos@vyos:~$ show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
S>* 0.0.0.0/0 [210/0] via 10.10.1.254, eth0, weight 1, 1d23h03m
C>* 10.10.1.0/24 is directly connected, eth0, 1d23h03m
K>* 10.100.1.1/32 [0/100] via 172.25.40.10, eth3, 00:00:28
C>* 10.100.1.255/32 is directly connected, lo, 1d23h03m
C>* 10.250.250.0/30 is directly connected, vti10, 1d23h03m
C>* 172.20.30.0/24 is directly connected, eth1, 1d23h03m
C>* 172.25.30.8/30 is directly connected, eth2, 1d23h03m
C>* 172.25.40.8/30 is directly connected, eth3, 1d23h03mNow enabling the IP 172.25.30.10 again in the Mikrotik:
vyos@vyos:~$ ping 172.25.30.10
PING 172.25.30.10 (172.25.30.10) 56(84) bytes of data.
64 bytes from 172.25.30.10: icmp_seq=1 ttl=64 time=0.331 ms
64 bytes from 172.25.30.10: icmp_seq=2 ttl=64 time=0.328 ms
64 bytes from 172.25.30.10: icmp_seq=3 ttl=64 time=0.343 ms
64 bytes from 172.25.30.10: icmp_seq=4 ttl=64 time=0.315 ms
^C
--- 172.25.30.10 ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3081ms
rtt min/avg/max/mdev = 0.315/0.329/0.343/0.010 ms
vyos@vyos:~$ show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
S>* 0.0.0.0/0 [210/0] via 10.10.1.254, eth0, weight 1, 1d23h07m
C>* 10.10.1.0/24 is directly connected, eth0, 1d23h07m
K>* 10.100.1.1/32 [0/100] via 172.25.40.10, eth3, 00:04:15
C>* 10.100.1.255/32 is directly connected, lo, 1d23h07m
C>* 10.250.250.0/30 is directly connected, vti10, 1d23h07m
C>* 172.20.30.0/24 is directly connected, eth1, 1d23h07m
C>* 172.25.30.8/30 is directly connected, eth2, 1d23h07m
C>* 172.25.40.8/30 is directly connected, eth3, 1d23h07mLets disable the backup link next-hop in the MK router:
vyos@vyos:~$ ping 172.25.40.10
PING 172.25.40.10 (172.25.40.10) 56(84) bytes of data.
From 172.25.40.9 icmp_seq=1 Destination Host Unreachable
From 172.25.40.9 icmp_seq=2 Destination Host Unreachable
From 172.25.40.9 icmp_seq=3 Destination Host Unreachable
From 172.25.40.9 icmp_seq=4 Destination Host Unreachable
From 172.25.40.9 icmp_seq=5 Destination Host Unreachable
^C
--- 172.25.40.10 ping statistics ---
6 packets transmitted, 0 received, +5 errors, 100% packet loss, time 5129ms
pipe 4
vyos@vyos:~$ show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
S>* 0.0.0.0/0 [210/0] via 10.10.1.254, eth0, weight 1, 1d23h08m
C>* 10.10.1.0/24 is directly connected, eth0, 1d23h08m
K>* 10.100.1.1/32 [0/1] via 172.25.30.10, eth2, 00:00:24
C>* 10.100.1.255/32 is directly connected, lo, 1d23h08m
C>* 10.250.250.0/30 is directly connected, vti10, 1d23h08m
C>* 172.20.30.0/24 is directly connected, eth1, 1d23h08m
C>* 172.25.30.8/30 is directly connected, eth2, 1d23h08m
C>* 172.25.40.8/30 is directly connected, eth3, 1d23h08mNow the route changes back to the main link. This is the only problem I found while testing.
@Viacheslav, where is best place to discuss the feature (ask a question or report a bug)?
@Harliff It is better to write to this task if you find bugs or propose new features.
So anyone could claim/fix it.
Thanks.
@Viacheslav Ok!
Minor bug: due to using "sudo ip route show" insted of just "ip route show", the system journal flooded with messages from sudo subsystem.
Apr 04 14:31:29 R1001-new sudo[16102]: root : PWD=/ ; USER=root ; COMMAND=/usr/sbin/ip --json route show protocol failover 10.0.0.3/32 via 10.0.2.1 dev eth2 metric 20 Apr 04 14:31:29 R1001-new sudo[16102]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0) Apr 04 14:31:29 R1001-new sudo[16102]: pam_unix(sudo:session): session closed for user root
I recommend to use just "ip route show" (without sudo).
Tested on VyOS 1.4-rolling-202304020811
Bug: unable to rename a failover route:
[edit protocols failover] vyos@R1001-new# rename route 10.0.0.3/32 to route 10.0.0.4/32 Rename failed
Is it possible to implement multiple test targets instead of just one?
Arguments:
We may chose a well-known reliable host (e.g. 8.8.8.8) as ping target. But the host would become unavailable when primary ISP goes down (because there would be static route to e.g. 8.8.8.8/32 via ISP1).
We have targets-checks 203.0.113.1, 192.0.2.1, and if any of these targets are unreachable, we delete this route.
Is it correct?
Sorry, missed some messages.
It is not correct. I think it would be better to remove the route if ALL of corresponding targets are unreachable.
A target may become unreachable due to a problem of its own rather than an uplink failure. This is the reason why I asked to add multiple targets per uplink.
Maybe adding check policy any|all will be suitable for all cases
something like
set protocols failover route 192.0.2.55/32 next-hop 203.0.113.1 check policy any-available|all-available|any-fail|all-fail
and any-fail as the default behavior
PR https://github.com/vyos/vyos-1x/pull/1966
set protocols failover route 192.0.2.55/32 next-hop 192.168.122.1 check policy 'any-available' set protocols failover route 192.0.2.55/32 next-hop 192.168.122.1 check target '192.168.122.1' set protocols failover route 192.0.2.55/32 next-hop 192.168.122.1 check target '192.168.122.11' set protocols failover route 192.0.2.55/32 next-hop 192.168.122.1 check timeout '3' set protocols failover route 192.0.2.55/32 next-hop 192.168.122.1 interface 'eth0'
@Harliff Could you check it? Available in the latest rolling release
vyos@r14# set protocols failover route 192.0.2.55/32 next-hop 192.168.122.1 check policy Possible completions: all-available All targets must be alive any-available Any target must be alive (default)
Could you check it? Available in the latest rolling release
Sorry for late answer. I've tested the feature - it working fine. Thanks for your job!
It will be great to see op-mode command to see current failover status.
Is it OK to discuss such topics here?
BTW, do you have any plans to use this mechanism for WAN failover?
It will be great to see op-mode command to see current failover status.
BTW, do you have any plans to use this mechanism for WAN failover?
Could you explain or send some examples?
Currently the WAN load balancing / WAN failover built using wan_lb program.
I've spend some time recently trying to get it working as it documented (on the 1.4-rc3) and achive only partial success:
Also, I'm suspecting that disable-source-nat option does nothing.
I haven't written a bug report yet.
So I thought: what if the VyOS will use iproute2 instead of the wan_lb for WAN failover?
Found task named wan load balance issues with 3 or more WANs.
Although I've found nothing related to WANs count in the task description and comments, I want to note that I'm using 4 WANs in my test environment.
I've made mistake while configuring my test stand so the WLB in my GNS3 was affected by contrack on host machine.
Looks like the WLB works as it should.
Sorry to bother you!
I've spend some time recently trying to get [WLB] working as it documented (on the 1.4-rc3) and achive only partial success:
- failover is working
- failover status display not working (show wan failover command show that all uplinks are down while really one of them working)
- I've failed to make failover switch time less than a minute
Also, I'm suspecting that disable-source-nat option does nothing.
I haven't written a bug report yet.