Page MenuHomeVyOS Platform

VRRP change on IPSec interface causes packet routing issues
Closed, ResolvedPublic

Description

I've built VyOs 1.2 from source and am having this issue on all of the vyos routers that I use as virtual endpoints for various networks.

The configuration there are a pair of routers with public (eth1) and private (eth0) IP addresses, both of which have VRRP enabled for HA for sevices.

We also have IPSec enabled on the public interface for IKEv2 site to site vpn connections to other routers.

A thing we have noticed is that when the VRRP on the public interface changes state, intended for private ip addresseses go out of the public interface.

Here is the interface config:

high-availability {
    vrrp {
        group eth0-1 {
            advertise-interval 1
            interface eth0
            priority 32
            virtual-address 10.xx.xx.254/24
            vrid 1
        }
        group eth1-1 {
            advertise-interval 1
            interface eth1
            peer-address xx.xx.xx.34
            priority 32
            virtual-address xx.xx.xx.2/26
            vrid 2
        }
    }
}
interfaces {
    ethernet eth0 {
        address 10.xx.xx.253/24
        duplex auto
        smp-affinity auto
        speed auto
    }
    ethernet eth1 {
        address xx.xx.xx.35/26
        duplex auto
        smp-affinity auto
        speed auto
    }
    loopback lo {
    }
    vti vti7001 {
        address 10.70.255.0/31
        description "Tunnel to endpoint"
        mtu 1400
    }
}
vpn {
    ipsec {
        esp-group ESP-AES256-SHA1-DH5 {
            compression disable
            lifetime 3600
            mode tunnel
            pfs dh-group5
            proposal 1 {
                encryption aes256
                hash sha1
            }
        }
        ike-group IKE2-AES265-SHA1-DH5 {
            dead-peer-detection {
                action restart
                interval 10
                timeout 30
            }
            ikev2-reauth no
            key-exchange ikev2
            lifetime 86400
            proposal 1 {
                dh-group 5
                encryption aes256
                hash sha1
            }
        }
        ipsec-interfaces {
            interface eth1
        }
        nat-traversal enable
        site-to-site {
            peer vpn.endpoint {
                authentication {
                    mode pre-shared-secret
                    pre-shared-secret ****************
                }
                connection-type initiate
                ike-group IKE2-AES265-SHA1-DH5
                ikev2-reauth inherit
                local-address xx.xx.xx.35
                vti {
                    bind vti7001
                    esp-group ESP-AES256-SHA1-DH5
                }
            }
        }
    }
}

Once the VRRP state changes, packets meant to go out of etho go out of eth1

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
15:46:09.525007 IP RouterA > 10.xx.xx.11: ICMP echo request, id 37020, seq 1, length 12
15:46:10.200648 IP RouterA > 10.xx.xx.11: ICMP echo request, id 30211, seq 29, length 64
15:46:11.224580 IP RouterA > 10.xx.xx.11: ICMP echo request, id 30211, seq 30, length 64
15:46:12.248571 IP RouterA > 10.xx.xx.11: ICMP echo request, id 30211, seq 31, length 64

This is reproduceable by just manually changing the priority of VRRP on the public interface and changing it back again.

The weird part is that in order to fix this all I have to do is to run restart vpn to restart the ipsec process and packets go back to running out of the correct interfaces again.

This is on both VyOS 1.2.4 and 1.2.5.

The version of strongSwan is 5.7.2-1+vyos2

AI have done a bit of searching and the only other task i can find that relates to this issue is https://phabricator.vyos.net/T342 where a user mentions that he has to restart the ipsec service when the VRRP becomes master.

Details

Difficulty level
Unknown (require assessment)
Version
1.2.5
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Related Objects

Event Timeline

erkin set Issue type to Bug (incorrect behavior).Aug 29 2021, 12:31 PM
erkin removed a subscriber: Active contributors.
syncer changed the task status from Open to Needs testing.Oct 17 2021, 3:25 PM
syncer assigned this task to n.fort.
syncer triaged this task as Normal priority.
syncer removed a project: VyOS 1.3 Equuleus.

Tests were done using 1.2.8 and 1.3.0-rc6 version.
Same inconvenient present as described.
Problems is that a default route is added in table 220

vyos@RAA-41:~$ show ip route table 220
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued route, r - rejected route

VRF default table 220:
K>* 0.0.0.0/0 [0/0] via 192.0.2.1, eth1, 00:03:04

This is solved with command

set vpn ipsec options disable-route-autoinstall

Output with bug fixed:

vyos@RAA-41:~$ show vrrp
Name    Interface      VRID  State    Last Transition
------  -----------  ------  -------  -----------------
eth0-1  eth0              1  MASTER   1m30s
eth1-1  eth1              2  MASTER   1m30s
vyos@RAA-41:~$ show vpn ipsec sa
Connection                    State    Up          Bytes In/Out    Remote address    Remote ID    Proposal
----------------------------  -------  ----------  --------------  ----------------  -----------  ------------------------------------------------
peer-198.51.100.2-tunnel-vti  up       99 seconds  0B/0B           198.51.100.2      N/A          AES_CBC_256/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1536
vyos@RAA-41:~$ show ip route table 220
vyos@RAA-41:~$ ping 10.0.0.2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=0.561 ms
64 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=0.827 ms
^C
--- 10.0.0.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1016ms
rtt min/avg/max/mdev = 0.561/0.694/0.827/0.133 ms

Then, modifying VRRP priority and forcing it to backup state in "public" interface, everything working as expected.

vyos@RAA-41:~$ show vrrp
Name    Interface      VRID  State    Last Transition
------  -----------  ------  -------  -----------------
eth0-1  eth0              1  MASTER   15s
eth1-1  eth1              2  BACKUP   11s
vyos@RAA-41:~$ show vpn ipsec sa
Connection                    State    Up         Bytes In/Out    Remote address    Remote ID    Proposal
----------------------------  -------  ---------  --------------  ----------------  -----------  ------------------------------------------------
peer-198.51.100.2-tunnel-vti  up       3 minutes  0B/0B           198.51.100.2      N/A          AES_CBC_256/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1536
vyos@RAA-41:~$ show ip route table 220
vyos@RAA-41:~$ ping 10.0.0.2
PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
64 bytes from 10.0.0.2: icmp_seq=1 ttl=64 time=0.411 ms
64 bytes from 10.0.0.2: icmp_seq=2 ttl=64 time=0.791 ms
64 bytes from 10.0.0.2: icmp_seq=3 ttl=64 time=0.723 ms
^C
--- 10.0.0.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2037ms
rtt min/avg/max/mdev = 0.411/0.641/0.791/0.168 ms

Resolved using

set vpn ipsec options disable-route-autoinstall

Also, documentation is up to date: "When using site-to-site IPsec with VTI interfaces, be sure to disable route autoinstall" ==> https://docs.vyos.io/en/equuleus/configuration/interfaces/vti.html