Page MenuHomeVyOS Platform

loadbalancing does not work with one pppoe connection and another connection of either dhcp or static
Closed, ResolvedPublic

Description

when using load balancing with a pppoe connection and another static or dhcp connection load balancing will never use the pppoe connection due to the way pppoe uses the iptables vs the other connection.

static routes with a different metric does not alleviate this as it would if one connection was dhcp and another static.

issue has persisted since original pull from vyatta codebase

i believe that in edgeos this issue is the primary reason load balancing is handled completely differently.

Details

Version
-
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Event Timeline

in edgeos each loadbalanced interface has it's ip table set directly at the interface-health section of the loadbalancing config. ex:

load-balance {
	group <name> {
		interface <name> {
			route {
				default
				table <name>
			}
			weight <number>
			failover-only
			route-test {
				type {
					default
					ping {
						target <host>
					}
					script <name>
				}
				initial-delay <number>
				interval <number>
				count {
					success <number>
					failure <number>
				}
			}
		}
	}
}

UBNT Loadbalancing Guide

What version have you been using?

This can be achieved in VyOS 1.2 albeit it's harder to resolve. I unfortunately lost a config example with pppoe, static and dhcp over ethernet working correctly. It requires using static interface routes for dhcp to fix weights; pppoe force-gateway option; and an adjustment of the static routes weight to match the dhcp ones (this was four years ago).

priochange: I'd like to first see where the PR stems from, before proceeding.

EwaldvanGeffen lowered the priority of this task from Urgent! to Requires assessment.May 21 2017, 5:22 PM

been using the VyOS 1.2.0-beta1

i've never had this work successfully even on vyatta. back then i know they were working on a patch for this issue but it was never released.

static routes for the pppoe interface can't be done if the gateway changes (interface-routes don't work) not sure if static route x.x.x.x/x dhcp-interface would work either, i could not get either working with a pppoe tunnel.

the weights are only one issue, the other being even when the static routes were setup with the weight of 2 for both interfaces, the nat for both, and the loadbalancing configuration; only the dhcp interface was routable. it seems that the pppoe interface was not able to share the same iptable. (rather there was no way to set the weight of the pppoe interface to be the same as the dhcp one(a route of 0.0.0.0 with a weight of 2 did not work))

if there was a way to configure which iptable each interface used for outbound traffic i believe this would resolve the issue.

I added a force-gateway option some time ago. Regardless it's somewhat expected on 1.2, it needs testing and review. I meant 1.1.7 in my previous post (yes, confusion).

syncer triaged this task as Normal priority.Aug 1 2017, 3:04 AM
syncer changed the edit policy from "Task Author" to "Custom Policy".
syncer removed a project: VyOS 2.0.x.
syncer set Version to -.
syncer subscribed.
syncer changed the task status from Open to On hold.Oct 13 2018, 7:22 PM
syncer assigned this task to masterit.

please retest on the latest rolling

Tried editing our config. Could not find a force gateway option in the load balance sections. Can you confirm where the force gateway option would be?

https://community.ubnt.com/t5/EdgeRouter/Issue-NAT-Exclude-with-load-balance/m-p/628739#U628739 this example shows how edgeos has dealt with this issue.

effectively binding a route table to each wan interface. probably an iptable modify in the background.

dmbaturin set Is it a breaking change? to Unspecified (possibly destroys the router).
erkin set Issue type to Bug (incorrect behavior).Sep 1 2021, 10:59 AM
masterit changed the task status from On hold to Open.May 25 2022, 10:57 PM
masterit raised the priority of this task from Normal to High.

Test #1:

  • Simple configuration with one interface pppoe and other static, WLB working as expected:
vyos@vyos# run show int
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface        IP Address                        S/L  Description
---------        ----------                        ---  -----------
eth0             -                                 u/u  
eth1             203.0.113.2/24                    u/u  
eth2             10.0.0.1/24                       u/u  
eth3             -                                 u/u  
lo               127.0.0.1/8                       u/u  
                 ::1/128                                
pppoe0           198.51.100.101/32                 u/u  

[edit]
vyos@vyos# run show config comm | grep "ppp\|load"
set interfaces pppoe pppoe0 access-concentrator 'ACN'
set interfaces pppoe pppoe0 authentication password 'foo'
set interfaces pppoe pppoe0 authentication username 'foo'
set interfaces pppoe pppoe0 source-interface 'eth0'
set load-balancing wan interface-health eth1 nexthop '203.0.113.1'
set load-balancing wan interface-health pppoe0 nexthop '198.51.100.1'
set load-balancing wan rule 10 inbound-interface 'eth2'
set load-balancing wan rule 10 interface eth1
set load-balancing wan rule 10 interface pppoe0
[edit]
vyos@vyos# sudo ip rule
0:      from all lookup local
32764:  from all fwmark 0xca lookup 202
32765:  from all fwmark 0xc9 lookup 201
32766:  from all lookup main
32767:  from all lookup default
[edit]
vyos@vyos# sudo nft list table ip mangle
# Warning: table ip mangle is managed by iptables-nft, do not touch!
table ip mangle {
        chain WANLOADBALANCE_PRE {
                iifname "eth2" ct state new meta random & 2147483647 < 1073741824 counter packets 277 bytes 17631 jump ISP_eth1
                iifname "eth2" ct state new counter packets 260 bytes 16623 jump ISP_pppoe0
                iifname "eth2" counter packets 1 bytes 84 meta mark set ct mark
        }

        chain PREROUTING {
                type filter hook prerouting priority mangle; policy accept;
                counter packets 1161 bytes 85454 jump WANLOADBALANCE_PRE
        }

        chain ISP_eth1 {
                counter packets 277 bytes 17631 ct mark set 0xc9
                counter packets 277 bytes 17631 meta mark set 0xc9
                counter packets 277 bytes 17631 accept
        }

        chain ISP_pppoe0 {
                counter packets 260 bytes 16623 ct mark set 0xca
                counter packets 260 bytes 16623 meta mark set 0xca
                counter packets 260 bytes 16623 accept
        }
}
[edit]
vyos@vyos# sudo nft list table ip nat
# Warning: table ip nat is managed by iptables-nft, do not touch!
table ip nat {
        chain VYOS_PRE_SNAT_HOOK {
                type nat hook postrouting priority srcnat - 1; policy accept;
                counter packets 513 bytes 32846 jump WANLOADBALANCE
        }

        chain WANLOADBALANCE {
                ct mark 0xc9 counter packets 266 bytes 17003 snat to 203.0.113.2
                ct mark 0xca counter packets 245 bytes 15723 snat to 198.51.100.101
        }  
}

And traceroute from host, with IP 10.0.0.X attached to eth2:

vyos@HOST# run traceroute www.google.com
traceroute to www.google.com (216.58.202.68), 30 hops max, 60 byte packets
 1  10.0.0.1 (10.0.0.1)  0.353 ms  0.320 ms  0.314 ms
 2  203.0.113.1 (203.0.113.1)  1.034 ms 198.51.100.1 (198.51.100.1)  1.013 ms  1.006 ms
 3  elazul-p0r1nro01.elazul.local (192.168.0.1)  1.409 ms  1.476 ms  1.902 ms
 4  192.168.100.1 (192.168.100.1)  2.255 ms  2.349 ms  2.399 ms
 5  host2.186-13-10.telmex.net.ar (186.13.10.2)  4.828 ms  5.295 ms  5.227 ms
 6  be5-2.c1900-br-05.claro.net.ar (170.51.254.172)  27.922 ms be5-2.cf223-br-05.claro.net.ar (170.51.254.176)  22.535 ms be5-2.c1900-br-05.claro.net.ar (170.51.254.172)  33.019 ms
 7  142.250.47.162 (142.250.47.162)  26.209 ms  23.478 ms 142.250.165.154 (142.250.165.154)  25.694 ms
 8  * * *
 9  142.251.239.190 (142.251.239.190)  22.661 ms 142.251.239.154 (142.251.239.154)  25.749 ms 74.125.242.209 (74.125.242.209)  23.170 ms
10  74.125.242.200 (74.125.242.200)  24.448 ms 108.170.237.241 (108.170.237.241)  28.069 ms 74.125.242.217 (74.125.242.217)  28.035 ms
11  gru10s11-in-f4.1e100.net (216.58.202.68)  23.321 ms 172.253.53.17 (172.253.53.17)  24.143 ms 172.253.53.33 (172.253.53.33)  22.769 ms
[edit]
vyos@HOST# run traceroute www.google.com
traceroute to www.google.com (216.58.202.68), 30 hops max, 60 byte packets
 1  10.0.0.1 (10.0.0.1)  0.697 ms  0.665 ms *
 2  198.51.100.1 (198.51.100.1)  1.334 ms 203.0.113.1 (203.0.113.1)  0.848 ms  0.845 ms
 3  elazul-p0r1nro01.elazul.local (192.168.0.1)  1.588 ms  1.582 ms  1.718 ms
 4  192.168.100.1 (192.168.100.1)  2.520 ms *  2.503 ms
 5  host2.186-13-10.telmex.net.ar (186.13.10.2)  5.224 ms  5.660 ms  5.699 ms
 6  be5-2.c1900-br-05.claro.net.ar (170.51.254.172)  31.393 ms be5-2.cf223-br-05.claro.net.ar (170.51.254.176)  31.993 ms be5-2.c1900-br-05.claro.net.ar (170.51.254.172)  26.143 ms
 7  142.250.47.162 (142.250.47.162)  23.695 ms  23.489 ms  27.413 ms
 8  * * *
 9  74.125.242.209 (74.125.242.209)  25.237 ms 142.251.61.46 (142.251.61.46)  24.253 ms 142.251.239.146 (142.251.239.146)  27.112 ms
10  142.250.46.111 (142.250.46.111)  27.108 ms 108.170.237.241 (108.170.237.241)  28.657 ms  27.356 ms
11  eze06s09-in-f4.1e100.net (216.58.202.68)  24.586 ms  21.382 ms 172.253.53.17 (172.253.53.17)  25.747 ms
[edit]
vyos@HOST#

Also, correct behavior when 1 interface is pppoe, and other dhcp-client:

vyos@vyos# run show int
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface        IP Address                        S/L  Description
---------        ----------                        ---  -----------
eth0             -                                 u/u  
eth1             203.0.113.151/24                  u/u  
eth2             10.0.0.1/24                       u/u  
eth3             -                                 u/u  
lo               127.0.0.1/8                       u/u  
                 ::1/128                                
pppoe0           198.51.100.101/32                 u/u  
[edit]

vyos@vyos# run show config comm | grep "addr\|ppp\|wan"
set interfaces ethernet eth1 address 'dhcp'
set interfaces ethernet eth2 address '10.0.0.1/24'
set interfaces pppoe pppoe0 access-concentrator 'ACN'
set interfaces pppoe pppoe0 authentication password 'foo'
set interfaces pppoe pppoe0 authentication username 'foo'
set interfaces pppoe pppoe0 source-interface 'eth0'
set load-balancing wan interface-health eth1 nexthop 'dhcp'
set load-balancing wan interface-health pppoe0 nexthop '198.51.100.1'
set load-balancing wan rule 10 inbound-interface 'eth2'
set load-balancing wan rule 10 interface eth1
set load-balancing wan rule 10 interface pppoe0
[edit]

And test from client 10.0.0.X:

vyos@HOST# run traceroute vyos.dev
traceroute to vyos.dev (104.18.20.61), 30 hops max, 60 byte packets
 1  10.0.0.1 (10.0.0.1)  0.420 ms  0.471 ms  0.410 ms
 2  203.0.113.1 (203.0.113.1)  1.080 ms 198.51.100.1 (198.51.100.1)  1.687 ms 203.0.113.1 (203.0.113.1)  1.110 ms
 3  elazul-p0r1nro01.elazul.local (192.168.0.1)  1.972 ms  1.779 ms  1.813 ms
 4  192.168.100.1 (192.168.100.1)  2.626 ms  2.620 ms  2.524 ms
 5  host2.186-13-10.telmex.net.ar (186.13.10.2)  6.073 ms  6.068 ms  6.123 ms
 6  10.2.187.165 (10.2.187.165)  25.672 ms 10.2.187.161 (10.2.187.161)  25.036 ms  25.476 ms
 7  host59.170-51-254.telmex.net.ar (170.51.254.59)  24.340 ms  22.982 ms  23.588 ms
 8  104.18.20.61 (104.18.20.61)  24.399 ms  24.760 ms  23.823 ms
[edit]
vyos@HOST# run traceroute vyos.dev
traceroute to vyos.dev (104.18.20.61), 30 hops max, 60 byte packets
 1  10.0.0.1 (10.0.0.1)  0.239 ms  0.214 ms  0.226 ms
 2  198.51.100.1 (198.51.100.1)  1.681 ms  1.606 ms  1.536 ms
 3  elazul-p0r1nro01.elazul.local (192.168.0.1)  1.833 ms  1.851 ms  2.022 ms
 4  192.168.100.1 (192.168.100.1)  2.645 ms  2.589 ms  2.443 ms
 5  host2.186-13-10.telmex.net.ar (186.13.10.2)  4.778 ms  5.145 ms  4.683 ms
 6  10.2.187.165 (10.2.187.165)  25.074 ms 10.2.187.161 (10.2.187.161)  25.864 ms  26.215 ms
 7  host59.170-51-254.telmex.net.ar (170.51.254.59)  25.331 ms  25.097 ms  25.010 ms
 8  104.18.20.61 (104.18.20.61)  25.975 ms  25.776 ms  30.063 ms
[edit]
vyos@HOST#
This comment was removed by masterit.