For exclude type of rules in load-balancing the firewall verdict statement is accept. In nftables, this is terminal statement and means that no other action with a packet will be performed in the same hook, if only we do not have other chain with higher priority.
The simple config that can be used to show the problem:
set interfaces ethernet eth0 address '192.168.0.2/24' set interfaces ethernet eth1 address '192.168.1.1/24' set interfaces tunnel tun0 address '192.168.72.20/31' set interfaces tunnel tun0 description 'magic-wan' set interfaces tunnel tun0 encapsulation 'gre' set interfaces tunnel tun0 mtu '1476' set interfaces tunnel tun0 remote '192.168.0.1' set interfaces tunnel tun0 source-address '192.168.0.2' set load-balancing wan disable-source-nat set load-balancing wan flush-connections set load-balancing wan interface-health eth0 failure-count '2' set load-balancing wan interface-health eth0 nexthop '192.168.0.1' set load-balancing wan interface-health eth0 success-count '1' set load-balancing wan interface-health eth0 test 10 resp-time '5' set load-balancing wan interface-health eth0 test 10 target '8.8.8.8' set load-balancing wan interface-health eth0 test 10 ttl-limit '1' set load-balancing wan interface-health eth0 test 10 type 'ping' set load-balancing wan rule 20 destination address '0.0.0.0/0' set load-balancing wan rule 20 exclude set load-balancing wan rule 20 inbound-interface 'eth1' set load-balancing wan rule 20 protocol 'all' set load-balancing wan rule 40 failover set load-balancing wan rule 40 inbound-interface 'eth1' set load-balancing wan rule 40 interface eth0 weight '10' set load-balancing wan rule 40 protocol 'all' set load-balancing wan sticky-connections inbound set nat source rule 100 outbound-interface 'eth0' set nat source rule 100 translation address 'masquerade' set nat source rule 200 outbound-interface 'tun0' set nat source rule 200 translation address 'masquerade' set policy route magic-wan rule 100 set table '100' set protocols static route 0.0.0.0/0 next-hop 192.168.0.1 set protocols static table 100 route 0.0.0.0/0 next-hop 192.168.72.21
Here we exclude all the traffic from load-balancing for clear view. This config will generate the next nftables table:
table ip mangle { chain PREROUTING { type filter hook prerouting priority mangle; policy accept; iifname "eth0" ct state new counter packets 0 bytes 0 jump ISP_eth0_IN counter packets 68 bytes 5339 jump WANLOADBALANCE_PRE counter packets 52 bytes 4336 jump VYATTA_FW_IN_HOOK } chain INPUT { type filter hook input priority mangle; policy accept; } chain FORWARD { type filter hook forward priority mangle; policy accept; } chain OUTPUT { type route hook output priority mangle; policy accept; counter packets 35 bytes 2484 jump VYATTA_FW_LOCALOUT_HOOK } chain POSTROUTING { type filter hook postrouting priority mangle; policy accept; counter packets 74 bytes 5699 jump VYATTA_FW_OUT_HOOK } chain VYATTA_FW_OUT_HOOK { } chain VYATTA_FW_IN_HOOK { iifname "eth1" counter packets 0 bytes 0 jump magic-wan } chain VYATTA_FW_LOCALOUT_HOOK { } chain magic-wan { counter packets 0 bytes 0 jump VYATTA_PBR_100 comment "magic-wan-100" counter packets 0 bytes 0 return comment "magic-wan-1000000 default-action accept" } chain VYATTA_PBR_100 { counter packets 0 bytes 0 meta mark set 0x80000063 counter packets 0 bytes 0 accept } chain WANLOADBALANCE_PRE { iifname "eth1" counter packets 21 bytes 1435 accept iifname "eth1" ct state new counter packets 0 bytes 0 jump ISP_eth0 iifname "eth1" counter packets 0 bytes 0 meta mark set ct mark } chain ISP_eth0 { counter packets 0 bytes 0 ct mark set 0xc9 counter packets 0 bytes 0 meta mark set 0xc9 counter packets 0 bytes 0 accept } chain ISP_eth0_IN { counter packets 0 bytes 0 ct mark set 0xc9 } }
In the table we have only one base chain in a hook prerouting that contains both load-balancing rules and PBR. Since load-balancing is executed first, packets accepted by exclude rules will never reach PBR.