Page MenuHomeVyOS Platform

Wireguard does not pass traffic if VRFs are used
Needs reporter action, NormalPublicBUG

Description

I am using the following setup:

WAN VRF, Interface in VLAN 999, gets IP assigned via DHCP (possibly overlaps with other IPs, hence separate VRF)
Simple wireguard tunnel to server at xxx.xxx.74.149

This wireguard tunnel does not pass any traffic.

ping 10.255.1.1 vrf wan

gives no replies. tcpdump on the server shows no inbound packets. tcpdump on VyOS shows packets being sent

monitor traffic interface wg0
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on wg0, link-type RAW (Raw IP), snapshot length 262144 bytes
16:33:23.491616 IP 10.255.1.2 > 10.255.1.1: ICMP echo request, id 15476, seq 1, length 64
16:33:24.500153 IP 10.255.1.2 > 10.255.1.1: ICMP echo request, id 15476, seq 2, length 64
16:33:25.524026 IP 10.255.1.2 > 10.255.1.1: ICMP echo request, id 15476, seq 3, length 64
16:33:26.548167 IP 10.255.1.2 > 10.255.1.1: ICMP echo request, id 15476, seq 4, length 64

Version:

Version:          VyOS 1.5-rolling-202404280021
Release train:    current

Config (unncessary stuff stripped)

interfaces {
    ethernet eth1 {
        description "LAN and WAN interface"
        vif 999 {
            address dhcp
            description WAN
            vrf wan
        }
    }
    loopback lo {
        address 10.0.0.5/32
    }
    wireguard wg0 {
        address 10.255.1.2/29
        description VPN-to-VPS
        peer to-VPS {
            address xxx.xxx.74.149
            allowed-ips 10.255.1.0/29
            persistent-keepalive 25
            port 51822
            public-key ****************
        }
        private-key xxxxxx
        vrf wan
    }
}
protocols {
    static {
        route xxx.xxx.74.149/32 {
            next-hop 192.168.128.1 {
                vrf wan
            }
        }
    }
}
vrf {
    name wan {
        table 1100
    }
}
sudo wg show wg0

interface: wg0
  public key: ****************
  private key: (hidden)
  listening port: 37998

peer: ****************
  endpoint: xxx.xxx.74.149:51822
  allowed ips: 10.255.1.0/29
  transfer: 0 B received, 32.95 KiB sent
  persistent keepalive: every 25 seconds
show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

C>* 10.0.0.5/32 is directly connected, lo, 00:24:16
S>* xxx.xxx.74.149/32 [1/0] via 192.168.128.1, eth1.999 (vrf wan), weight 1, 00:24:13
show ip route vrf wan
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

VRF wan:
S>* 0.0.0.0/0 [210/0] via 192.168.128.1, eth1.999, weight 1, 00:24:46
C>* 10.255.1.0/29 is directly connected, wg0, 00:24:45
C>* 192.168.128.0/17 is directly connected, eth1.999, 00:24:46

Details

Difficulty level
Unknown (require assessment)
Version
VyOS 1.5-rolling-202404280021
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Event Timeline

Please note that the Wireguard tunnel itself is sourced from the default VRF. Only the "inner side" of the tunnel runs in VRF wan.

There is no possibility to source the wireguard interface from

  • Another VRf
  • A discrete source IP
  • A discrete source interface

This can only be handled by applying fwmark values and policy based routing - this is a WireGuard design thing.

In T6281#185394, @c-po wrote:

Please note that the Wireguard tunnel itself is sourced from the default VRF. Only the "inner side" of the tunnel runs in VRF wan.

There is no possibility to source the wireguard interface from

  • Another VRf
  • A discrete source IP
  • A discrete source interface

This can only be handled by applying fwmark values and policy based routing - this is a WireGuard design thing.

But this is not related to problem, right?
It's the reason why I need

static {
        route xxx.xxx.74.149/32 {
            next-hop 192.168.128.1 {
                vrf wan
            }
        }
    }

in order that wireguard can establish the tunnel. But I cannot ping from any vrf, no matter whether I place wireguard in the default vrf or the wan vrf itself

You probably should get the wireguard interface running in your default VRF first and see if traffic properly passes - once that's working for oyu you can move it into a VRF. Please not only the decrypted side of the WireGuard interface will reside in the VRF. The side passing encrypted packets ALWAYS is in the default VRF (Linux Kernel)

In T6281#185620, @c-po wrote:

You probably should get the wireguard interface running in your default VRF first and see if traffic properly passes - once that's working for oyu you can move it into a VRF. Please not only the decrypted side of the WireGuard interface will reside in the VRF. The side passing encrypted packets ALWAYS is in the default VRF (Linux Kernel)

Tried that (thought I mentioned it), won't work.

Wireguard in WAN VRF and output WAN interface in WAN VRF (route leak via snippet posted above): Tunnel is established, but no traffic is passed
Wireguard in default VRF and outbound WAN interface in WAN VRF (route leak via snipped posted above): Tunnel is established, but no traffic is passed

Well looks like you hit a WireGuard limitation here (route-leaking the default nexthop to another VRF)

Did you try it with any vanilla Linux distribution if that works at all?

What you're trying to achieve seems to be a derivate of https://www.wireguard.com/netns/

NETNS was removed from the 1.4 series the other day so hopefully that feature can be worked on for 1.5 since its needed:

https://vyos.dev/T6295

Specially since the VRF config in VyOS (due to Linux backend) is different from VRF in other NOS (like Arista, Cisco, Juniper etc).

Other NOS definition of VRF is more like what Linux calls NETNS for process segmentation in combination with VRF as dedicated routing table.

Viacheslav triaged this task as Normal priority.Wed, May 8, 6:26 AM
c-po changed the task status from Open to Needs reporter action.Wed, May 8, 4:05 PM