I've built VyOs 1.2 from source and am having this issue on all of the vyos routers that I use as virtual endpoints for various networks.
The configuration there are a pair of routers with public (eth1) and private (eth0) IP addresses, both of which have VRRP enabled for HA for sevices.
We also have IPSec enabled on the public interface for IKEv2 site to site vpn connections to other routers.
A thing we have noticed is that when the VRRP on the public interface changes state, intended for private ip addresseses go out of the public interface.
Here is the interface config:
high-availability { vrrp { group eth0-1 { advertise-interval 1 interface eth0 priority 32 virtual-address 10.xx.xx.254/24 vrid 1 } group eth1-1 { advertise-interval 1 interface eth1 peer-address xx.xx.xx.34 priority 32 virtual-address xx.xx.xx.2/26 vrid 2 } } } interfaces { ethernet eth0 { address 10.xx.xx.253/24 duplex auto smp-affinity auto speed auto } ethernet eth1 { address xx.xx.xx.35/26 duplex auto smp-affinity auto speed auto } loopback lo { } vti vti7001 { address 10.70.255.0/31 description "Tunnel to endpoint" mtu 1400 } } vpn { ipsec { esp-group ESP-AES256-SHA1-DH5 { compression disable lifetime 3600 mode tunnel pfs dh-group5 proposal 1 { encryption aes256 hash sha1 } } ike-group IKE2-AES265-SHA1-DH5 { dead-peer-detection { action restart interval 10 timeout 30 } ikev2-reauth no key-exchange ikev2 lifetime 86400 proposal 1 { dh-group 5 encryption aes256 hash sha1 } } ipsec-interfaces { interface eth1 } nat-traversal enable site-to-site { peer vpn.endpoint { authentication { mode pre-shared-secret pre-shared-secret **************** } connection-type initiate ike-group IKE2-AES265-SHA1-DH5 ikev2-reauth inherit local-address xx.xx.xx.35 vti { bind vti7001 esp-group ESP-AES256-SHA1-DH5 } } } } }
Once the VRRP state changes, packets meant to go out of etho go out of eth1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes 15:46:09.525007 IP RouterA > 10.xx.xx.11: ICMP echo request, id 37020, seq 1, length 12 15:46:10.200648 IP RouterA > 10.xx.xx.11: ICMP echo request, id 30211, seq 29, length 64 15:46:11.224580 IP RouterA > 10.xx.xx.11: ICMP echo request, id 30211, seq 30, length 64 15:46:12.248571 IP RouterA > 10.xx.xx.11: ICMP echo request, id 30211, seq 31, length 64
This is reproduceable by just manually changing the priority of VRRP on the public interface and changing it back again.
The weird part is that in order to fix this all I have to do is to run restart vpn to restart the ipsec process and packets go back to running out of the correct interfaces again.
This is on both VyOS 1.2.4 and 1.2.5.
The version of strongSwan is 5.7.2-1+vyos2
AI have done a bit of searching and the only other task i can find that relates to this issue is https://phabricator.vyos.net/T342 where a user mentions that he has to restart the ipsec service when the VRRP becomes master.