Summary
Routes that VPP creates automatically for REMOTE_TS in IPsec are less preferred than normal routes imported by linux-cp. This leads to leaking unencrypted traffic even when all IPsec components are properly installed in VPP.
How to reproduce
Basic config is:
set interfaces ethernet eth0 address '192.168.0.1/24' set interfaces ethernet eth1 address '192.168.1.1/24' set protocols static route 192.168.2.0/24 next-hop 192.168.0.2 set system option kernel cpu disable-nmi-watchdog set system option kernel cpu isolate-cpus '2-3' set system option kernel cpu nohz-full '2-3' set system option kernel cpu rcu-no-cbs '2-3' set system option kernel disable-hpet set system option kernel disable-mce set system option kernel disable-mitigations set system option kernel disable-softlockup set system option kernel memory hugepage-size 2M hugepage-count '1024' set vpn ipsec authentication psk PSK1 id 'peer1' set vpn ipsec authentication psk PSK1 id 'peer2' set vpn ipsec authentication psk PSK1 secret 'secret' set vpn ipsec esp-group ESP1 lifetime '60' set vpn ipsec esp-group ESP1 proposal 10 encryption 'aes256' set vpn ipsec esp-group ESP1 proposal 10 hash 'sha256' set vpn ipsec ike-group IKE1 close-action 'start' set vpn ipsec ike-group IKE1 dead-peer-detection action 'restart' set vpn ipsec ike-group IKE1 lifetime '600' set vpn ipsec ike-group IKE1 proposal 10 encryption 'aes256gcm128' set vpn ipsec ike-group IKE1 proposal 10 hash 'sha256' set vpn ipsec interface 'eth0' set vpn ipsec options disable-route-autoinstall set vpn ipsec site-to-site peer peer1 authentication local-id 'peer1' set vpn ipsec site-to-site peer peer1 authentication mode 'pre-shared-secret' set vpn ipsec site-to-site peer peer1 authentication remote-id 'peer2' set vpn ipsec site-to-site peer peer1 connection-type 'initiate' set vpn ipsec site-to-site peer peer1 default-esp-group 'ESP1' set vpn ipsec site-to-site peer peer1 ike-group 'IKE1' set vpn ipsec site-to-site peer peer1 local-address '192.168.0.1' set vpn ipsec site-to-site peer peer1 remote-address '192.168.0.2' set vpn ipsec site-to-site peer peer1 tunnel 10 local prefix '192.168.1.0/24' set vpn ipsec site-to-site peer peer1 tunnel 10 remote prefix '192.168.2.0/24' set vpp settings interface eth0 set vpp settings interface eth1 set vpp settings ipsec-acceleration set vpp settings poll-sleep-usec '1000' set vpp settings resource-allocation memory main-heap-size '1G'
Key places are:
set protocols static route 192.168.2.0/24 next-hop 192.168.0.2 set vpn ipsec site-to-site peer peer1 tunnel 10 remote prefix '192.168.2.0/24'
Such a config is perfectly valid and even expected for a normal kernel IPsec. But when it is synchronized to the VPP dataplane, the route to 192.168.2.0/24 is doubled:
vyos@vyos:~$ sudo vppctl show ip fib 192.168.2.0/24
ipv4-VRF:0, fib_index:0, flow hash:[src dst sport dport proto flowlabel ] epoch:0 flags:none locks:[adjacency:1, default-route:1, lcp-rt:1, ]
192.168.2.0/24 fib:0 index:22 locks:3
lcp-rt-dynamic refs:1 src-flags:added,contributing,active,
path-list:[34] locks:2 flags:shared, uPRF-list:26 len:1 itfs:[1, ]
path:[45] pl-index:34 ip4 weight=1 pref=20 attached-nexthop: oper-flags:resolved,
192.168.0.2 eth0
[@0]: ipv4 via 192.168.0.2 eth0: mtu:1500 next:5 flags:[] 0cbad6e100000c7b9eb500000800
API refs:1 entry-flags:attached, src-flags:added,
path-list:[35] locks:1 flags:shared, uPRF-list:27 len:1 itfs:[5, ]
path:[51] pl-index:35 ip4 weight=1 pref=0 attached: oper-flags:resolved, cfg-flags:glean,
ipsec1
forwarding: unicast-ip4-chain
[@0]: dpo-load-balance: [proto:ip4 index:23 buckets:1 uRPF:26 to:[274:23016]]
[0] [@5]: ipv4 via 192.168.0.2 eth0: mtu:1500 next:5 flags:[] 0cbad6e100000c7b9eb500000800And the route via ipsec1 is less preferred.
Expected behavior
It is expected that IPsec routes are always preferred, because the security policy should not be possible to bypass.
Potential issues
If we are going to hardcode some values for IPsec-installed routes, they must be better than similar values for routes received via linux-cp. Dynamic routing protocols may create routes in the kernel that have any values, modified by route-maps or received from peers - we need to take this into account.