VPP node drops packets coming from ipsec tunnel destined to its local interfaces.
LEFT:
set interfaces ethernet eth0 address '10.0.0.1/24' set interfaces ethernet eth1 address '10.1.0.1/24' set interfaces vti vti1 address '192.168.255.1/30' set protocols static route 10.2.0.0/24 interface vti1 set vpn ipsec authentication psk psk1 id 'A' set vpn ipsec authentication psk psk1 id 'B' set vpn ipsec authentication psk psk1 secret 'AB' set vpn ipsec esp-group esp1 mode 'tunnel' set vpn ipsec esp-group esp1 pfs 'disable' set vpn ipsec esp-group esp1 proposal 10 encryption 'aes256' set vpn ipsec esp-group esp1 proposal 10 hash 'sha256' set vpn ipsec ike-group ike1 close-action 'none' set vpn ipsec ike-group ike1 dead-peer-detection action 'clear' set vpn ipsec ike-group ike1 proposal 10 encryption 'camellia256ccm96' set vpn ipsec ike-group ike1 proposal 10 hash 'sha256' set vpn ipsec interface 'eth0' set vpn ipsec site-to-site peer B authentication local-id 'A' set vpn ipsec site-to-site peer B authentication mode 'pre-shared-secret' set vpn ipsec site-to-site peer B authentication remote-id 'B' set vpn ipsec site-to-site peer B connection-type 'initiate' set vpn ipsec site-to-site peer B default-esp-group 'esp1' set vpn ipsec site-to-site peer B ike-group 'ike1' set vpn ipsec site-to-site peer B local-address '10.0.0.1' set vpn ipsec site-to-site peer B remote-address '10.0.0.2' set vpn ipsec site-to-site peer B vti bind 'vti1' set vpp settings interface eth0 driver 'dpdk' set vpp settings interface eth1 driver 'dpdk' set vpp settings ipsec
RIGHT:
set interfaces ethernet eth0 address '10.0.0.2/24' set interfaces ethernet eth1 address '10.2.0.1/24' set interfaces vti vti1 address '192.168.255.2/30' set protocols static route 10.1.0.0/24 interface vti1 set vpn ipsec authentication psk psk1 id 'B' set vpn ipsec authentication psk psk1 id 'A' set vpn ipsec authentication psk psk1 secret 'AB' set vpn ipsec esp-group esp1 mode 'tunnel' set vpn ipsec esp-group esp1 pfs 'disable' set vpn ipsec esp-group esp1 proposal 10 encryption 'aes256' set vpn ipsec esp-group esp1 proposal 10 hash 'sha256' set vpn ipsec ike-group ike1 close-action 'none' set vpn ipsec ike-group ike1 dead-peer-detection action 'clear' set vpn ipsec ike-group ike1 proposal 10 encryption 'camellia256ccm96' set vpn ipsec ike-group ike1 proposal 10 hash 'sha256' set vpn ipsec interface 'eth0' set vpn ipsec site-to-site peer A authentication local-id 'B' set vpn ipsec site-to-site peer A authentication mode 'pre-shared-secret' set vpn ipsec site-to-site peer A authentication remote-id 'A' set vpn ipsec site-to-site peer A connection-type 'none' set vpn ipsec site-to-site peer A default-esp-group 'esp1' set vpn ipsec site-to-site peer A ike-group 'ike1' set vpn ipsec site-to-site peer A local-address '10.0.0.2' set vpn ipsec site-to-site peer A remote-address '10.0.0.1' set vpn ipsec site-to-site peer A vti bind 'vti1'
Packets get dropped with 'ip4 spoofed local-address packet' error on vpp node. Example of failed cases, pings from router RIGHT:
- ping 10.1.0.1
Packet 1
00:25:15:087588: dpdk-input
eth0 rx queue 0
buffer 0x91c25: current data 0, length 170, buffer-pool 0, ref-count 1, trace handle 0x0
ext-hdr-valid
PKT MBUF: port 0, nb_segs 1, pkt_len 170
buf_len 2176, data_len 170, ol_flags 0x0, data_off 128, phys_addr 0x642709c0
packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
rss 0x0 fdir.hi 0x0 fdir.lo 0x0
IP4: 0c:ff:0c:aa:00:00 -> 0c:53:c9:db:00:00
IPSEC_ESP: 10.0.0.2 -> 10.0.0.1
tos 0x00, ttl 64, length 156, checksum 0x262e dscp CS0 ecn NON_ECN
fragment id 0x0000, flags DONT_FRAGMENT
00:25:15:087611: ethernet-input
frame: flags 0x1, hw-if-index 1, sw-if-index 1
IP4: 0c:ff:0c:aa:00:00 -> 0c:53:c9:db:00:00
00:25:15:087618: ip4-input
IPSEC_ESP: 10.0.0.2 -> 10.0.0.1
tos 0x00, ttl 64, length 156, checksum 0x262e dscp CS0 ecn NON_ECN
fragment id 0x0000, flags DONT_FRAGMENT
00:25:15:087622: ip4-lookup
fib 0 dpo-idx 12 flow hash: 0x00000000
IPSEC_ESP: 10.0.0.2 -> 10.0.0.1
tos 0x00, ttl 64, length 156, checksum 0x262e dscp CS0 ecn NON_ECN
fragment id 0x0000, flags DONT_FRAGMENT
00:25:15:087624: ip4-receive
fib:0 adj:12 flow:0x00000000
IPSEC_ESP: 10.0.0.2 -> 10.0.0.1
tos 0x00, ttl 64, length 156, checksum 0x262e dscp CS0 ecn NON_ECN
fragment id 0x0000, flags DONT_FRAGMENT
00:25:15:087626: ipsec4-tun-input
IPSec: remote:10.0.0.2 spi:3402481557 (0xcacdbf95) sa:0 tun:0 seq 75
00:25:15:087627: esp4-decrypt-tun
esp: crypto aes-cbc-256 integrity sha-256-128 pkt-seq 75 sa-seq 75 pkt-seq-hi
0
00:25:15:087644: ip4-input-no-checksum
ICMP: 192.168.255.2 -> 10.1.0.1
tos 0x00, ttl 64, length 84, checksum 0x1c7c dscp CS0 ecn NON_ECN
fragment id 0x5480, flags DONT_FRAGMENT
ICMP echo_request checksum 0x9dfd id 6063
00:25:15:087646: ip4-lookup
fib 0 dpo-idx 14 flow hash: 0x00000000
ICMP: 192.168.255.2 -> 10.1.0.1
tos 0x00, ttl 64, length 84, checksum 0x1c7c dscp CS0 ecn NON_ECN
fragment id 0x5480, flags DONT_FRAGMENT
ICMP echo_request checksum 0x9dfd id 6063
00:25:15:087647: ip4-receive
fib:0 adj:14 flow:0x00000000
ICMP: 192.168.255.2 -> 10.1.0.1
tos 0x00, ttl 64, length 84, checksum 0x1c7c dscp CS0 ecn NON_ECN
fragment id 0x5480, flags DONT_FRAGMENT
ICMP echo_request checksum 0x9dfd id 6063
00:25:15:087648: ip4-drop
fib:0 adj:9 flow:0x00000000
ICMP: 192.168.255.2 -> 10.1.0.1
tos 0x00, ttl 64, length 84, checksum 0x1c7c dscp CS0 ecn NON_ECN
fragment id 0x5480, flags DONT_FRAGMENT
ICMP echo_request checksum 0x9dfd id 6063
00:25:15:087649: error-drop
rx:ipsec1
00:25:15:087650: drop
ip4-local: ip4 spoofed local-address packet drops
Packet 2- ping 192.168.255.1
Packet 1
00:26:05:071632: dpdk-input
eth0 rx queue 0
buffer 0x92a02: current data 0, length 170, buffer-pool 0, ref-count 1, trace handle 0x0
ext-hdr-valid
PKT MBUF: port 0, nb_segs 1, pkt_len 170
buf_len 2176, data_len 170, ol_flags 0x0, data_off 128, phys_addr 0x642a8100
packet_type 0x0 l2_len 0 l3_len 0 outer_l2_len 0 outer_l3_len 0
rss 0x0 fdir.hi 0x0 fdir.lo 0x0
IP4: 0c:ff:0c:aa:00:00 -> 0c:53:c9:db:00:00
IPSEC_ESP: 10.0.0.2 -> 10.0.0.1
tos 0x00, ttl 64, length 156, checksum 0x262e dscp CS0 ecn NON_ECN
fragment id 0x0000, flags DONT_FRAGMENT
00:26:05:071642: ethernet-input
frame: flags 0x1, hw-if-index 1, sw-if-index 1
IP4: 0c:ff:0c:aa:00:00 -> 0c:53:c9:db:00:00
00:26:05:071649: ip4-input
IPSEC_ESP: 10.0.0.2 -> 10.0.0.1
tos 0x00, ttl 64, length 156, checksum 0x262e dscp CS0 ecn NON_ECN
fragment id 0x0000, flags DONT_FRAGMENT
00:26:05:071652: ip4-lookup
fib 0 dpo-idx 12 flow hash: 0x00000000
IPSEC_ESP: 10.0.0.2 -> 10.0.0.1
tos 0x00, ttl 64, length 156, checksum 0x262e dscp CS0 ecn NON_ECN
fragment id 0x0000, flags DONT_FRAGMENT
00:26:05:071655: ip4-receive
fib:0 adj:12 flow:0x00000000
IPSEC_ESP: 10.0.0.2 -> 10.0.0.1
tos 0x00, ttl 64, length 156, checksum 0x262e dscp CS0 ecn NON_ECN
fragment id 0x0000, flags DONT_FRAGMENT
00:26:05:071657: ipsec4-tun-input
IPSec: remote:10.0.0.2 spi:3402481557 (0xcacdbf95) sa:0 tun:0 seq 122
00:26:05:071658: esp4-decrypt-tun
esp: crypto aes-cbc-256 integrity sha-256-128 pkt-seq 122 sa-seq 122 pkt-seq-h
i 0
00:26:05:071674: ip4-input-no-checksum
ICMP: 192.168.255.2 -> 192.168.255.1
tos 0x00, ttl 64, length 84, checksum 0xc2ec dscp CS0 ecn NON_ECN
fragment id 0xf866, flags DONT_FRAGMENT
ICMP echo_request checksum 0xb042 id 6064
00:26:05:071676: ip4-lookup
fib 0 dpo-idx 9 flow hash: 0x00000000
ICMP: 192.168.255.2 -> 192.168.255.1
tos 0x00, ttl 64, length 84, checksum 0xc2ec dscp CS0 ecn NON_ECN
fragment id 0xf866, flags DONT_FRAGMENT
ICMP echo_request checksum 0xb042 id 6064
00:26:05:071676: ip4-receive
fib:0 adj:9 flow:0x00000000
ICMP: 192.168.255.2 -> 192.168.255.1
tos 0x00, ttl 64, length 84, checksum 0xc2ec dscp CS0 ecn NON_ECN
fragment id 0xf866, flags DONT_FRAGMENT
ICMP echo_request checksum 0xb042 id 6064
00:26:05:071677: ip4-drop
fib:0 adj:9 flow:0x00000000
ICMP: 192.168.255.2 -> 192.168.255.1
tos 0x00, ttl 64, length 84, checksum 0xc2ec dscp CS0 ecn NON_ECN
fragment id 0xf866, flags DONT_FRAGMENT
ICMP echo_request checksum 0xb042 id 6064
00:26:05:071678: error-drop
rx:ipsec1
00:26:05:071680: drop
ip4-local: ip4 spoofed local-address packet dropsVPP node kernel routes:
vyos@vyos# run sh ip ro
Codes: K - kernel route, C - connected, L - local, S - static,
R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric, t - Table-Direct,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
IPv4 unicast VRF default:
C>* 10.0.0.0/24 is directly connected, eth0, weight 1, 00:26:57
L * 10.0.0.1/32 is directly connected, eth0, weight 1, 00:26:57
L>* 10.0.0.1/32 is directly connected, eth0, weight 1, 00:26:57
L 10.0.0.1/32 is directly connected, unknown inactive, weight 1, 00:27:31
L 10.0.0.1/32 is directly connected, unknown inactive, weight 1, 00:48:44
L 10.0.0.1/32 is directly connected, unknown inactive, weight 1, 00:52:30
C>* 10.1.0.0/24 is directly connected, eth1, weight 1, 00:26:56
L * 10.1.0.1/32 is directly connected, eth1, weight 1, 00:26:56
L>* 10.1.0.1/32 is directly connected, eth1, weight 1, 00:26:56
L 10.1.0.1/32 is directly connected, unknown inactive, weight 1, 00:48:43
L 10.1.0.1/32 is directly connected, unknown inactive, weight 1, 00:52:29
S>* 10.2.0.0/24 [1/0] is directly connected, vti1, weight 1, 00:21:14
C>* 192.168.255.0/30 is directly connected, vti1, weight 1, 00:21:14
L>* 192.168.255.1/32 is directly connected, vti1, weight 1, 00:21:14How vpp installs the VTI route:
vpp# sh ip fib 192.168.255.0/30
ipv4-VRF:0, fib_index:0, flow hash:[src dst sport dport proto flowlabel ] epoch:0 flags:none locks:[adjacency:1, default-route:1, lcp-rt:1, ]
192.168.255.0/30 fib:0 index:8 locks:2
lcp-rt refs:2 entry-flags:local, src-flags:added,contributing,active,
path-list:[20] locks:2 flags:local, uPRF-list:9 len:0 itfs:[]
path:[20] pl-index:20 ip4 weight=1 pref=0 receive: oper-flags:resolved, cfg-flags:local,
[@0]: dpo-receive: 0.0.0.0 on local0
forwarding: unicast-ip4-chain
[@0]: dpo-load-balance: [proto:ip4 index:9 buckets:1 uRPF:9 to:[144:12096]]
[0] [@12]: dpo-receive: 0.0.0.0 on local0
vpp#
vpp# sh ip fib 192.168.255.2
ipv4-VRF:0, fib_index:0, flow hash:[src dst sport dport proto flowlabel ] epoch:0 flags:none locks:[adjacency:1, default-route:1, lcp-rt:1, ]
192.168.255.0/30 fib:0 index:8 locks:2
lcp-rt refs:2 entry-flags:local, src-flags:added,contributing,active,
path-list:[20] locks:2 flags:local, uPRF-list:9 len:0 itfs:[]
path:[20] pl-index:20 ip4 weight=1 pref=0 receive: oper-flags:resolved, cfg-flags:local,
[@0]: dpo-receive: 0.0.0.0 on local0
forwarding: unicast-ip4-chain
[@0]: dpo-load-balance: [proto:ip4 index:9 buckets:1 uRPF:9 to:[157:13188]]
[0] [@12]: dpo-receive: 0.0.0.0 on local0
vpp#Entire VTI prefix is installed as entry-flags:local as we can see, which triggers ip4-local spoofed local-address for decrypted traffic arriving on ipsec1 in vpp.
If we disable VPP connectivity is restored:
vyos@vyos# run ping 192.168.255.1 source-address 192.168.255.2 PING 192.168.255.1 (192.168.255.1) from 192.168.255.2 : 56(84) bytes of data. 64 bytes from 192.168.255.1: icmp_seq=1 ttl=64 time=5.28 ms 64 bytes from 192.168.255.1: icmp_seq=2 ttl=64 time=1.04 ms 64 bytes from 192.168.255.1: icmp_seq=3 ttl=64 time=0.885 ms ^C --- 192.168.255.1 ping statistics --- 3 packets transmitted, 3 received, 0% packet loss, time 2003ms rtt min/avg/max/mdev = 0.885/2.398/5.275/2.035 ms vyos@vyos# run ping 10.1.0.1 source-address 10.2.0.1 PING 10.1.0.1 (10.1.0.1) from 10.2.0.1 : 56(84) bytes of data. 64 bytes from 10.1.0.1: icmp_seq=1 ttl=64 time=4.67 ms 64 bytes from 10.1.0.1: icmp_seq=2 ttl=64 time=1.12 ms ^C --- 10.1.0.1 ping statistics --- 2 packets transmitted, 2 received, 0% packet loss, time 1002ms rtt min/avg/max/mdev = 1.121/2.893/4.665/1.772 ms