@Nova_Logic As a workaround can you try the following configuration, this should ensure the behaviour you're after:

policy {
    local-route {
        rule 1 {
            fwmark 201
            inbound-interface eth0
            set {
                table 201
            }
        }
        rule 2 {
            fwmark 202
            inbound-interface eth0
            set {
                table 202
            }
        }
        rule 3 {
            fwmark 203
            inbound-interface eth0
            set {
                table 203
            }
        }
    }
}

I had a closer look at this, the commits here and here should have brought this back from the dead but there is a small issue where the table "ip nat" is used when at the same time all of the nat related chains were refactored into the vyos_nat table. The normal functionality of WLB isn't affected because the mangle table isn't changed.

@Nova_Logic no it would not function as intended, the reason is say you have 3 interfaces, and interface 1 has a metric of 1, 2 a metric of 2, etc. If a packet comes in off one of these interfaces it will be routed to it's destination with the appropriate DNAT rule, the source address is the initiators global unicast address with the mac of the router itself. Now when your service replies it's hosts routing table looks like

0.0.0.0/0 via routers-localaddr

The destination address is of course the remote global unicast address of the client and the source is the local area network address (information is still sufficient at this point), however the routers table will look like

0.0.0.0 via iface1 metric 1
0.0.0.0 via iface2 metric 2
0.0.0.0 via iface3 metric 3

So the reply will always go out iface1, source NAT happens post routing, so the source address of the reply packet from the initiating clients perspective has changed, thus being invalid and dropped by any correctly configured firewall.

@Nova_Logic I understand your frustration with the old WLB, it is not compatible with policy routes, DNAT, or fwmarks due to the way it's implemented. However WLB or this new implementation are not ingress capable tools. That is, these fill a niche in SMB setups where BGP peering is not possible (due to the use of commodity ISPs), or the cost and/or complexity of operating an IGP or even physically connecting into something like enterprise ethernet, is just completely out of the question. Despite the limitations these setups still need a way to switch over from faulted links quickly and reliably so you don't have an office full of people twiddling there thumbs while the internet is down.

I have been thinking about this over the weekend and looked into your failover implementation, there's nothing wrong with it and should serve most peoples needs. That said I am not too good with python so it was more straight forward to start from scratch.

I have used this feature in the past but not anymore due to the issues listed in the regressions task. We are now running pfsense purely for LB since this (mostly) works as advertised. Looking back at this current implementation there are some very useful features that are missing.