We are currently seeing an issue with IPsec tunnels where after the ESP Lifetime expires traffic is unable to route down the tunnel to it's intended destination down the VTI interface. The only way to make traffic traverse down the tunnel is to restart the IPsec process, and this then allows the expected behaviour for the ESP lifetime, then once this expires, traffic stops working again.
We believe this is a VyOS issue as after the ESP time out the IPsec tunnel remains up and traffic is still sent down the tunnel, however this traffic never reaches the other side of the tunnel, and we also do not see any back and forth ESP traffic being sent to the IPsec tunnel's remote gateway after the ESP timeout and can only see this after restarting the IPsec process on the VyOS device.
When we monitor traffic, it still goes out via the vti interfaces that is binded to the site-to-site VPN however when we monitor for ESP packets towards the public endpoint of the VPN tunnel, this stops after the lifetime expires.
We also would expect that the SA should restart after the lifetime (in our case its set to 60 seconds for testing but in production this would be 3600), we just see the VPN SA continue showing as up (longer than 60 seconds) but the ESP traffic stops, we have also set a keep alive on the VPN but with this being IKEv2 I don't believe this feature works as per the documentation.
Packet capture when traffic is working;
vti0:
root@csg19:/home/jack.littlewood# tcpdump -i vti0
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on vti0, link-type RAW (Raw IP), snapshot length 262144 bytes
11:56:20.171172 IP 172.24.48.128.49955 > page.surfprotect.exa.net.uk.http: Flags [S], seq 2994231420, win 28720, options [mss 1300,sackOK,TS val 1174863400 ecr 0,nop,wscale 7], length 0
11:56:20.171360 IP page.surfprotect.exa.net.uk.http > 172.24.48.128.49955: Flags [S.], seq 2018934677, ack 2994231421, win 42340, options [mss 1460,nop,nop,sackOK,nop,wscale 12], length 0
11:56:20.182959 IP 172.24.48.128.49955 > page.surfprotect.exa.net.uk.http: Flags [.], ack 1, win 225, length 0
11:56:20.183110 IP 172.24.48.128.49955 > page.surfprotect.exa.net.uk.http: Flags [P.], seq 1:93, ack 1, win 225, length 92: HTTP: GET / HTTP/1.1
11:56:20.183204 IP page.surfprotect.exa.net.uk.http > 172.24.48.128.49955: Flags [.], ack 93, win 11, length 0
11:56:20.183529 IP page.surfprotect.exa.net.uk.http > 172.24.48.128.49955: Flags [P.], seq 1:278, ack 93, win 11, length 277: HTTP: HTTP/1.1 302 Found
11:56:20.195067 IP 172.24.48.128.49955 > page.surfprotect.exa.net.uk.http: Flags [.], ack 278, win 233, length 0
11:56:20.196704 IP 172.24.48.128.49955 > page.surfprotect.exa.net.uk.http: Flags [F.], seq 93, ack 278, win 233, length 0
11:56:20.196851 IP page.surfprotect.exa.net.uk.http > 172.24.48.128.49955: Flags [F.], seq 278, ack 94, win 11, length 0
11:56:20.208375 IP 172.24.48.128.49955 > page.surfprotect.exa.net.uk.http: Flags [.], ack 279, win 233, length 0
11:56:22.806908 IP 172.24.48.128.57117 > page.surfprotect.exa.net.uk.http: Flags [S], seq 3151778621, win 28720, options [mss 1300,sackOK,TS val 1174863664 ecr 0,nop,wscale 7], length 0
11:56:22.807083 IP page.surfprotect.exa.net.uk.http > 172.24.48.128.57117: Flags [S.], seq 2313620486, ack 3151778622, win 42340, options [mss 1460,nop,nop,sackOK,nop,wscale 12], length 0
11:56:22.818644 IP 172.24.48.128.57117 > page.surfprotect.exa.net.uk.http: Flags [.], ack 1, win 225, length 0
11:56:22.818809 IP 172.24.48.128.57117 > page.surfprotect.exa.net.uk.http: Flags [P.], seq 1:93, ack 1, win 225, length 92: HTTP: GET / HTTP/1.1
11:56:22.818901 IP page.surfprotect.exa.net.uk.http > 172.24.48.128.57117: Flags [.], ack 93, win 11, length 0
11:56:22.819299 IP page.surfprotect.exa.net.uk.http > 172.24.48.128.57117: Flags [P.], seq 1:278, ack 93, win 11, length 277: HTTP: HTTP/1.1 302 Found
11:56:22.830830 IP 172.24.48.128.57117 > page.surfprotect.exa.net.uk.http: Flags [.], ack 278, win 233, length 0
11:56:22.831222 IP 172.24.48.128.57117 > page.surfprotect.exa.net.uk.http: Flags [F.], seq 93, ack 278, win 233, length 0
11:56:22.831350 IP page.surfprotect.exa.net.uk.http > 172.24.48.128.57117: Flags [F.], seq 278, ack 94, win 11, length 0
11:56:22.842862 IP 172.24.48.128.57117 > page.surfprotect.exa.net.uk.http: Flags [.], ack 279, win 233, length 0
11:56:23.911252 IP 172.24.48.128.49063 > page.surfprotect.exa.net.uk.http: Flags [S], seq 2577670235, win 28720, options [mss 1300,sackOK,TS val 1174863774 ecr 0,nop,wscale 7], length 0
11:56:23.911426 IP page.surfprotect.exa.net.uk.http > 172.24.48.128.49063: Flags [S.], seq 1832504976, ack 2577670236, win 42340, options [mss 1460,nop,nop,sackOK,nop,wscale 12], length 0
11:56:23.923070 IP 172.24.48.128.49063 > page.surfprotect.exa.net.uk.http: Flags [.], ack 1, win 225, length 0
11:56:23.923226 IP 172.24.48.128.49063 > page.surfprotect.exa.net.uk.http: Flags [P.], seq 1:93, ack 1, win 225, length 92: HTTP: GET / HTTP/1.1
11:56:23.923308 IP page.surfprotect.exa.net.uk.http > 172.24.48.128.49063: Flags [.], ack 93, win 11, length 0
11:56:23.923758 IP page.surfprotect.exa.net.uk.http > 172.24.48.128.49063: Flags [P.], seq 1:278, ack 93, win 11, length 277: HTTP: HTTP/1.1 302 Found
11:56:23.935294 IP 172.24.48.128.49063 > page.surfprotect.exa.net.uk.http: Flags [.], ack 278, win 233, length 0
11:56:23.935769 IP 172.24.48.128.49063 > page.surfprotect.exa.net.uk.http: Flags [F.], seq 93, ack 278, win 233, length 0
11:56:23.935910 IP page.surfprotect.exa.net.uk.http > 172.24.48.128.49063: Flags [F.], seq 278, ack 94, win 11, length 0
11:56:23.947449 IP 172.24.48.128.49063 > page.surfprotect.exa.net.uk.http: Flags [.], ack 279, win 233, length 0
eth0/1(WAN):
root@csg19:/home/jack.littlewood# tcpdump -i any 'esp'
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
11:56:20.171194 eth1 Out IP 82.219.122.120 > 82.219.26.244: ESP(spi=0xc370571a,seq=0x1), length 100
11:56:20.171350 eth0 In IP 82.219.26.244 > 82.219.122.120: ESP(spi=0xc349551f,seq=0x1), length 100
11:56:20.182969 eth1 Out IP 82.219.122.120 > 82.219.26.244: ESP(spi=0xc370571a,seq=0x2), length 84
11:56:20.183115 eth1 Out IP 82.219.122.120 > 82.219.26.244: ESP(spi=0xc370571a,seq=0x3), length 180
11:56:20.183196 eth0 In IP 82.219.26.244 > 82.219.122.120: ESP(spi=0xc349551f,seq=0x2), length 84
11:56:20.183523 eth0 In IP 82.219.26.244 > 82.219.122.120: ESP(spi=0xc349551f,seq=0x3), length 356
11:56:20.195080 eth1 Out IP 82.219.122.120 > 82.219.26.244: ESP(spi=0xc370571a,seq=0x4), length 84
11:56:20.196712 eth1 Out IP 82.219.122.120 > 82.219.26.244: ESP(spi=0xc370571a,seq=0x5), length 84
11:56:20.196843 eth0 In IP 82.219.26.244 > 82.219.122.120: ESP(spi=0xc349551f,seq=0x4), length 84
11:56:20.208382 eth1 Out IP 82.219.122.120 > 82.219.26.244: ESP(spi=0xc370571a,seq=0x6), length 84
11:56:22.806927 eth1 Out IP 82.219.122.120 > 82.219.26.244: ESP(spi=0xc370571a,seq=0x7), length 100
11:56:22.807073 eth0 In IP 82.219.26.244 > 82.219.122.120: ESP(spi=0xc349551f,seq=0x5), length 100
11:56:22.818650 eth1 Out IP 82.219.122.120 > 82.219.26.244: ESP(spi=0xc370571a,seq=0x8), length 84
11:56:22.818815 eth1 Out IP 82.219.122.120 > 82.219.26.244: ESP(spi=0xc370571a,seq=0x9), length 180
11:56:22.818896 eth0 In IP 82.219.26.244 > 82.219.122.120: ESP(spi=0xc349551f,seq=0x6), length 84
11:56:22.819295 eth0 In IP 82.219.26.244 > 82.219.122.120: ESP(spi=0xc349551f,seq=0x7), length 356
11:56:22.830837 eth1 Out IP 82.219.122.120 > 82.219.26.244: ESP(spi=0xc370571a,seq=0xa), length 84
11:56:22.831227 eth1 Out IP 82.219.122.120 > 82.219.26.244: ESP(spi=0xc370571a,seq=0xb), length 84
11:56:22.831343 eth0 In IP 82.219.26.244 > 82.219.122.120: ESP(spi=0xc349551f,seq=0x8), length 84
11:56:22.842871 eth1 Out IP 82.219.122.120 > 82.219.26.244: ESP(spi=0xc370571a,seq=0xc), length 84
11:56:23.911270 eth1 Out IP 82.219.122.120 > 82.219.26.244: ESP(spi=0xc370571a,seq=0xd), length 100
11:56:23.911416 eth0 In IP 82.219.26.244 > 82.219.122.120: ESP(spi=0xc349551f,seq=0x9), length 100
11:56:23.923077 eth1 Out IP 82.219.122.120 > 82.219.26.244: ESP(spi=0xc370571a,seq=0xe), length 84
11:56:23.923230 eth1 Out IP 82.219.122.120 > 82.219.26.244: ESP(spi=0xc370571a,seq=0xf), length 180
11:56:23.923303 eth0 In IP 82.219.26.244 > 82.219.122.120: ESP(spi=0xc349551f,seq=0xa), length 84
11:56:23.923752 eth0 In IP 82.219.26.244 > 82.219.122.120: ESP(spi=0xc349551f,seq=0xb), length 356
11:56:23.935303 eth1 Out IP 82.219.122.120 > 82.219.26.244: ESP(spi=0xc370571a,seq=0x10), length 84
11:56:23.935774 eth1 Out IP 82.219.122.120 > 82.219.26.244: ESP(spi=0xc370571a,seq=0x11), length 84
11:56:23.935900 eth0 In IP 82.219.26.244 > 82.219.122.120: ESP(spi=0xc349551f,seq=0xc), length 84
11:56:23.947457 eth1 Out IP 82.219.122.120 > 82.219.26.244: ESP(spi=0xc370571a,seq=0x12), length 84
Packet capture when traffic is broken;
vti0:
root@csg19:/home/jack.littlewood# tcpdump -i vti0
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on vti0, link-type RAW (Raw IP), snapshot length 262144 bytes
11:54:44.434520 IP 172.24.48.128.42271 > page.surfprotect.exa.net.uk.http: Flags [S], seq 4024688645, win 28720, options [mss 1300,sackOK,TS val 1174853826 ecr 0,nop,wscale 7], length 0
11:54:49.282686 IP 172.24.48.128.59333 > page.surfprotect.exa.net.uk.http: Flags [S], seq 3640303055, win 28720, options [mss 1300,sackOK,TS val 1174854311 ecr 0,nop,wscale 7], length 0
11:54:49.974029 IP 172.24.48.128.48929 > page.surfprotect.exa.net.uk.http: Flags [S], seq 3388062991, win 28720, options [mss 1300,sackOK,TS val 1174854380 ecr 0,nop,wscale 7], length 0
11:54:50.737946 IP 172.24.48.128.33129 > page.surfprotect.exa.net.uk.http: Flags [S], seq 3904427069, win 28720, options [mss 1300,sackOK,TS val 1174854457 ecr 0,nop,wscale 7], length 0
11:54:51.465052 IP 172.24.48.128.53051 > page.surfprotect.exa.net.uk.http: Flags [S], seq 1668219073, win 28720, options [mss 1300,sackOK,TS val 1174854529 ecr 0,nop,wscale 7], length 0
11:54:52.182279 IP 172.24.48.128.44197 > page.surfprotect.exa.net.uk.http: Flags [S], seq 2340681241, win 28720, options [mss 1300,sackOK,TS val 1174854601 ecr 0,nop,wscale 7], length 0
11:54:52.851438 IP 172.24.48.128.35155 > page.surfprotect.exa.net.uk.http: Flags [S], seq 2628254242, win 28720, options [mss 1300,sackOK,TS val 1174854668 ecr 0,nop,wscale 7], length 0
11:54:53.381413 IP 172.24.48.128.52183 > page.surfprotect.exa.net.uk.http: Flags [S], seq 468708520, win 28720, options [mss 1300,sackOK,TS val 1174854721 ecr 0,nop,wscale 7], length 0
eth0/1(WAN):
root@csg19:/home/jack.littlewood# tcpdump -i any 'esp'
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
Full VPN configuration
daniel.bertram@csg19.tcw.man# edit vpn ipsec
[edit vpn ipsec]
daniel.bertram@csg19.tcw.man# show
esp-group SPQ {
lifetime 60
mode tunnel
pfs disable
proposal 1 {
encryption aes128
hash sha1
}
}
ike-group SPQ {
close-action restart
dead-peer-detection {
action restart
interval 5
timeout 60
}
ikev2-reauth
key-exchange ikev2
lifetime 60
proposal 1 {
dh-group 14
encryption aes128
hash sha1
}
}
interface lo
options {
disable-route-autoinstall
interface lo
virtual-ip
}
site-to-site {
peer OUR-0005 {
authentication {
mode x509
x509 {
ca-certificate OUR-0005
certificate OUR-0005
}
}
connection-type initiate
default-esp-group SPQ
ike-group SPQ
local-address 82.219.122.120
remote-address 82.219.26.244
vti {
bind vti0
}
}
}
[edit vpn ipsec]
daniel.bertram@csg19.tcw.man#
Can you look into this and advise further?