SCENARIO:
This is a textbook implementation of WAN failover. Two internet circuits from different providers. The primary circuit is on eth1. The backup circuit is on eth2. LAN is eth0. The router provides a NAT and does masquerading from the LAN to the internet.
When the primary circuit is in good health and up, only the primary circuit should be used for traffic between the internet and the LAN. The backup circuit should be like a hot spare, otherwise not in use.
When a health check determines that the primary interface has failed, the router should fail over to the backup circuit. When on the backup circuit, only the backup circuit should be used for traffic between the internet and the LAN.
PROBLEMS:
- The router's primary address on eth1 is found in traffic on eth2. The router's backup interface's IP address is sometimes found in traffic on eth1.
- The router's own WAN IP address is replacing the IP address of remote hosts accessing services on a LAN host. This makes it impossible to know where the packet actually came from. Applications such as SIP and my VPN need to know the real peer IP address, not the address of my primary WAN interface.
- Traffic destined for a LAN host going through the router via the address on the primary interface (eth1) will reach the LAN host, and the LAN host will respond correctly, but the router may try to send the replies out the backup interface (eth2) instead of the interface that the traffic arrived on (eth1).
CONFIGURATION:
interfaces {
ethernet eth0 {
address 192.168.192.1/24
description LAN_LAN_LAN
duplex auto
hw-id 8c:89:a5:99:4a:8a
smp-affinity auto
speed auto
}
ethernet eth1 {
address dhcp
description Cable_Primary_WAN
duplex auto
hw-id 8c:89:a5:99:4a:8b
smp-affinity auto
speed auto
}
ethernet eth2 {
address dhcp
description DSL_Backup_WAN
duplex auto
hw-id 8c:89:a5:99:4a:8c
smp-affinity auto
speed auto
}}
load-balancing {
wan {
enable-local-traffic
flush-connections
interface-health eth1 {
failure-count 5
nexthop dhcp
success-count 1
test 10 {
resp-time 5
target 8.8.8.8
ttl-limit 1
type ping
}
test 20 {
resp-time 5
target 4.2.2.1
ttl-limit 1
type ping
}
}
interface-health eth2 {
failure-count 4
nexthop dhcp
success-count 1
test 10 {
resp-time 5
target 8.8.4.4
ttl-limit 1
type ping
}
test 20 {
resp-time 5
target 4.2.2.2
ttl-limit 1
type ping
}
}
rule 10 {
failover
inbound-interface eth0
interface eth1 {
weight 10
}
interface eth2 {
weight 1
}
protocol all
}
sticky-connections {
inbound
}
}}
nat {
destination {
rule 40 {
description "Preroute IAX2"
destination {
port iax
}
inbound-interface eth1
protocol udp
translation {
address 192.168.192.242
}
}
rule 44 {
description "Preroute SIP"
destination {
port 5060-5061
}
inbound-interface eth1
protocol tcp_udp
translation {
address 192.168.192.242
}
}
rule 49 {
description "Preroute RTP"
destination {
port 10000-20000
}
inbound-interface eth1
protocol udp
translation {
address 192.168.192.242
}
}
rule 2655 {
description "Preroute tinc VPN to HP"
destination {
port 2655
}
inbound-interface eth1
protocol tcp_udp
translation {
address 192.168.192.58
port tinc
}
}
rule 7040 {
description "Preroute IAX2"
destination {
port iax
}
inbound-interface eth2
protocol udp
translation {
address 192.168.192.242
}
}
rule 7044 {
description "Preroute SIP"
destination {
port 5060-5061
}
inbound-interface eth2
protocol tcp_udp
translation {
address 192.168.192.242
}
}
rule 7049 {
description "Preroute RTP"
destination {
port 10000-20000
}
inbound-interface eth2
protocol udp
translation {
address 192.168.192.242
}
}
rule 7655 {
description "Preroute tinc VPN to HP"
destination {
port 2655
}
inbound-interface eth2
protocol tcp_udp
translation {
address 192.168.192.58
port tinc
}
}}
source {
rule 110 {
description "Hairpin NAT"
destination {
address 192.168.192.0/24
}
outbound-interface eth0
source {
address 192.168.192.0/24
}
translation {
address masquerade
}
}
rule 192 {
description NAT
outbound-interface eth1
source {
address 192.168.192.0/24
}
translation {
address masquerade
}
}
rule 193 {
description NAT
outbound-interface eth2
source {
address 192.168.192.0/24
}
translation {
address masquerade
}
}}
}
EVIDENCE:
rob@Rt-9877:~$ sh int e
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Description
eth0 192.168.192.1/24 u/u LAN_LAN_LAN
eth1 24.217.88.23/21 u/u Cable_Primary_WAN
eth2 162.205.147.147/22 u/u DSL_Backup_WAN
eth3 - u/D
rob@Rt-9877:~$ sh wan
Interface: eth1
Status: active Last Status Change: Thu Aug 31 05:26:42 2017 +Test: ping Target: 8.8.8.8 Test: ping Target: 4.2.2.1 Last Interface Success: 0s Last Interface Failure: n/a # Interface Failure(s): 0
Interface: eth2
Status: active Last Status Change: Thu Aug 31 05:26:42 2017 +Test: ping Target: 8.8.4.4 Test: ping Target: 4.2.2.2 Last Interface Success: 0s Last Interface Failure: n/a # Interface Failure(s): 0
Here you can see the primary interface's IP address talking on the backup interface:
rob@Rt-9877:~$ tshark -i eth2 -f "host 24.217.88.23"
Capturing on 'eth2'
1 0.000000 24.217.88.23 -> 8.8.4.4 ICMP 146 Destination unreachable (Port unreachable) 2 0.020403 24.217.88.23 -> 208.67.222.220 ICMP 146 Destination unreachable (Port unreachable) 3 25.137963 24.217.88.23 -> 208.67.222.222 ICMP 127 Destination unreachable (Port unreachable) 4 25.139023 24.217.88.23 -> 8.8.4.4 ICMP 127 Destination unreachable (Port unreachable) 5 25.151657 24.217.88.23 -> 208.67.222.220 ICMP 127 Destination unreachable (Port unreachable) 6 69.506506 24.217.88.23 -> 208.67.222.220 ICMP 164 Destination unreachable (Port unreachable) 7 69.512728 24.217.88.23 -> 208.67.220.220 ICMP 164 Destination unreachable (Port unreachable) 8 69.518716 24.217.88.23 -> 208.67.220.222 ICMP 164 Destination unreachable (Port unreachable) 9 91.828302 24.217.88.23 -> 208.67.220.220 ICMP 294 Destination unreachable (Port unreachable)
10 91.855232 24.217.88.23 -> 208.67.222.220 ICMP 294 Destination unreachable (Port unreachable)
11 124.915055 24.217.88.23 -> 8.8.4.4 ICMP 127 Destination unreachable (Port unreachable)
12 124.929002 24.217.88.23 -> 208.67.222.220 ICMP 127 Destination unreachable (Port unreachable)
13 124.953739 24.217.88.23 -> 208.67.220.220 ICMP 127 Destination unreachable (Port unreachable)
14 150.098830 24.217.88.23 -> 208.67.220.220 ICMP 121 Destination unreachable (Port unreachable)
15 150.106534 24.217.88.23 -> 208.67.222.220 ICMP 121 Destination unreachable (Port unreachable)
16 150.106603 24.217.88.23 -> 208.67.220.222 ICMP 121 Destination unreachable (Port unreachable)
17 182.941209 24.217.88.23 -> 71.10.216.2 ICMP 147 Destination unreachable (Port unreachable)
18 182.957277 24.217.88.23 -> 8.8.4.4 ICMP 147 Destination unreachable (Port unreachable)
19 182.984679 24.217.88.23 -> 208.67.222.220 ICMP 147 Destination unreachable (Port unreachable)
20 204.003832 24.217.88.23 -> 208.67.222.220 ICMP 143 Destination unreachable (Port unreachable)
21 204.003906 24.217.88.23 -> 208.67.220.222 ICMP 143 Destination unreachable (Port unreachable)
22 204.022680 24.217.88.23 -> 208.67.222.222 ICMP 143 Destination unreachable (Port unreachable)
23 229.996563 24.217.88.23 -> 8.8.4.4 ICMP 127 Destination unreachable (Port unreachable)
24 230.012135 24.217.88.23 -> 208.67.222.220 ICMP 127 Destination unreachable (Port unreachable)
^C24 packets captured
Here is an example of how the real IP address has been improperly replaced with my own primary WAN address instead of the real IP address of the peer.
[Aug 31 11:58:20] NOTICE[14058][C-000005f8]: chan_sip.c:25653 handle_request_invite: Call from '' (24.217.88.23:5071) to extension '+48914472532' rejected because extension not found in context 'unauthenticated'.
[Aug 31 12:10:23] NOTICE[21430][C-000005f9]: Ext. 01148914472532:2 @ unauthenticated: "1" <[email protected]> is attempting to make unauthorized calls
[Aug 31 13:28:40] NOTICE[22717][C-000005fa]: Ext. 0048914472532:2 @ unauthenticated: "1" <[email protected]> is attempting to make unauthorized calls
Here is an example of how those packets SHOULD look. Notice the IP address is not my own in this one.
[Aug 31 14:29:38] NOTICE[24035][C-00000634]: Ext. 1807706966:2 @ unauthenticated: "1807706966" <[email protected]> is attempting to make unauthorized calls
[Aug 31 14:29:38] NOTICE[24036][C-00000635]: Ext. 3182044188:2 @ unauthenticated: "3182044188" <[email protected]> is attempting to make unauthorized calls
[Aug 31 14:29:38] NOTICE[24037][C-00000636]: Ext. 100:2 @ unauthenticated: "100" <[email protected]> is attempting to make unauthorized calls
[Aug 31 14:29:38] NOTICE[24038][C-00000637]: Ext. 101:2 @ unauthenticated: "101" <[email protected]> is attempting to make unauthorized calls
Additional tests can be performed on request.