so you mean that new WLB implementation(on which I assume we're discussing here) would not mark incoming packets/sessions to allow vyos to DNAT/send replies to correct WAN like pfsense for example does?

PR https://github.com/vyos/vyos-1x/pull/1604

@Nova_Logic I understand your frustration with the old WLB, it is not compatible with policy routes, DNAT, or fwmarks due to the way it's implemented. However WLB or this new implementation are not ingress capable tools. That is, these fill a niche in SMB setups where BGP peering is not possible (due to the use of commodity ISPs), or the cost and/or complexity of operating an IGP or even physically connecting into something like enterprise ethernet, is just completely out of the question. Despite the limitations these setups still need a way to switch over from faulted links quickly and reliably so you don't have an office full of people twiddling there thumbs while the internet is down.

Also it seems, that’s issue appears on 3 or more wans, as I remember it worked with 2 WAN interfaces

The problem is that failover route will not solve multiwan scenarios where you have 2 or more links for incoming traffic, I.e web. Most good infrastructures would have dedicated management uplink, and also multiple WANs for serving client traffic. That approach increases infrastructure security and provide much more cleaner way to define zone policies. But to do that all traffic, especially incoming one must be correctly marked. I’ve tried a lot of ways to configure wlb, but every time vyos had tried to reply from the wrong interface, that’s why I had crated a bug task here

@tioan , Have you assigned your local-zone to the firewall rule ? Please use the latest version and share the error

PR: https://github.com/vyos/vyos-1x/pull/1603

PR:
https://github.com/vyos/vyos-1x/pull/1602

PR for VyOS 1.3 https://github.com/vyos/vyatta-cfg-system/pull/187

Tested with next configuration:

vyos@r14:~$ sudo cat /etc/pam.d/common-auth 
auth  required      pam_env.so
auth  required      pam_faillock.so preauth silent audit deny=3 unlock_time=300
auth  sufficient    pam_unix.so  nullok  try_first_pass
auth  [default=die] pam_faillock.so  authfail  audit  deny=3  unlock_time=300
auth  requisite     pam_succeed_if.so uid >= 1000 quiet_success
auth  required      pam_deny.so
vyos@r14:~$

PR https://github.com/vyos/vyatta-cfg-system/pull/186

PR https://github.com/vyos/vyos-1x/pull/1601

set service ssh hostkey-algorithm 'sk-ssh-ed25519@openssh.com'
set service ssh hostkey-algorithm 'ssh-rsa'

Added more bgpd/ospfd events to the log. The VRF Id seem to be correct. But the events look curious. After session start the interface is first created in vrf default (vrf default, id:0) followed by bgpd/ospfd events, then accel-ppp process moves it to destination vrf (vrf client, id:5) which is follwed by the bgpd/ospfd errors.
Finally, with more or less than 5000 sessions bgpd accidentally becomes unresponsive and utilizes 200% cpu (8 cores are used on VM). Accel-pppd process having all network destinations unreachable also goes unresponsive a bit later.
After that we have to reboot.

@CuBiC3D There is a comment of the commit https://github.com/vyos/vyos-1x/commit/373227e717fac82af5ea8d71e611a3df1c59054e

I am finding out, it seems OSPF SR doesn't work properly :(

https://github.com/vyos/vyos-1x/pull/1599

Here is ISIS segment routing working:

Basically,
all commercial hooks need to be implemented

I have been thinking about this over the weekend and looked into your failover implementation, there's nothing wrong with it and should serve most peoples needs. That said I am not too good with python so it was more straight forward to start from scratch.

I confirm this is still an issue in 1.4-rolling-202207250217 trying to download 1.4-rolling-202210150526:

@SrividyaA
The documentation at https://docs.vyos.io/en/latest/configuration/firewall/zone.html currently contains the following regarding local-zone:

Why does the image has to be added manually and can not be pulled from the registry if not locally available?