Page MenuHomeVyOS Platform
Create Task
Maniphest T4570

Exception when trying to set up VXLAN over Wireguard
Closed, ResolvedPublicBUG
Actions

Description

vyos@vyos-lab# compare
[edit interfaces vxlan vxlan0]
+address 2407:c280:ee:15::100/64
+mtu 1370
+remote 2407:c280:ee:ffff::1
+source-interface wg2
+vni 100
vyos@vyos-lab# commit

WARNING: RFC7348 recommends VXLAN tunnels preserve a 1500 byte MTU
[ interfaces vxlan vxlan0 ]

WARNING: RFC7348 recommends VXLAN tunnels preserve a 1500 byte MTU
VyOS had an issue completing a command.

We are sorry that you encountered a problem while using VyOS.
There are a few things you can do to help us (and yourself):
- Contact us using the online help desk if you have a subscription:
  https://support.vyos.io/
- Make sure you are running the latest version of VyOS available at:
  https://vyos.net/get/
- Consult the community forum to see how to handle this issue:
  https://forum.vyos.io
- Join us on Slack where our users exchange help and advice:
  https://vyos.slack.com

When reporting problems, please include as much information as possible:
- do not obfuscate any data (feel free to contact us privately if your
  business policy requires it)
- and include all the information presented below

Report time:      2022-07-26 01:51:08
Image version:    VyOS 1.4-rolling-202207250217
Release train:    sagitta

Built by:         autobuild@vyos.net
Built on:         Mon 25 Jul 2022 02:17 UTC
Build UUID:       2ac7cf68-8f44-4a3f-862d-ecc6a336110e
Build commit ID:  30e90e1f9f8edd

Architecture:     x86_64
Boot via:         installed image
System type:      VMware guest

Hardware vendor:  VMware, Inc.
Hardware model:   VMware Virtual Platform
Hardware S/N:     VMware-56 4d 1c 86 ab 92 b7 fd-81 53 b7 ce 69 1c 64 6f
Hardware UUID:    861c4d56-92ab-fdb7-8153-b7ce691c646f

Traceback (most recent call last):
  File "/usr/libexec/vyos/conf_mode/interfaces-vxlan.py", line 176, in <module>
    apply(c)
  File "/usr/libexec/vyos/conf_mode/interfaces-vxlan.py", line 167, in apply
    v.update(vxlan)
  File "/usr/lib/python3/dist-packages/vyos/ifconfig/interface.py", line 1539, in update
    self.set_mtu(config.get('mtu'))
  File "/usr/lib/python3/dist-packages/vyos/ifconfig/interface.py", line 443, in set_mtu
    return self.set_interface('mtu', mtu)
  File "/usr/lib/python3/dist-packages/vyos/ifconfig/control.py", line 183, in set_interface
    return self._set_command(self.config, name, value)
  File "/usr/lib/python3/dist-packages/vyos/ifconfig/control.py", line 110, in _set_command
    return self._command_set[name].get('format', lambda _: _)(self._cmd(cmd))
  File "/usr/lib/python3/dist-packages/vyos/ifconfig/control.py", line 52, in _cmd
    return cmd(command, self.debug)
  File "/usr/lib/python3/dist-packages/vyos/util.py", line 161, in cmd
    raise OSError(code, feedback)
FileNotFoundError: [Errno 2] failed to run command: ip link set dev vxlan0 mtu 1370
returned:
exit code: 2

noteworthy:
cmd 'nft -c delete element inet vrf_zones ct_iface_map { "vxlan0" }'
returned (out):

returned (err):
Error: Could not process rule: No such file or directory
delete element inet vrf_zones ct_iface_map { vxlan0 }
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
cmd 'ip link set dev vxlan0 mtu 1370'
returned (out):

returned (err):
RTNETLINK answers: Invalid argument

[[interfaces vxlan vxlan0]] failed
Commit failed

Details

Version
1.4-rolling-202207250217
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Related Objects

Event Timeline

aderouineau created this task.Jul 26 2022, 2:01 AM
aderouineau created this object in space S1 VyOS Public.
Viacheslav changed the subtype of this task from "Task" to "Bug".Jul 26 2022, 9:48 AM
Viacheslav subscribed.Jul 26 2022, 10:00 AM

@aderouineau Describe please all steps of how to reproduce it (with commands set xxx)
I don't have any issues with it

set interfaces vxlan vxlan0 group '239.0.0.241'
set interfaces vxlan vxlan0 mtu '1370'
set interfaces vxlan vxlan0 port '4789'
set interfaces vxlan vxlan0 source-interface 'wg0'
set interfaces vxlan vxlan0 vni '123'
set interfaces wireguard wg0 address '100.64.0.1/24'
set interfaces wireguard wg0 peer PEER01 allowed-ips '0.0.0.0/0'
set interfaces wireguard wg0 peer PEER01 public-key 'VVfR5S0yi+QPEJRLr25ZAfzFnwZM40G5WCZ/7ou7h3k='
set interfaces wireguard wg0 private-key 'yGOy08Kv8KUe8rsO6WHeo5jC7YdOAzQK0SJkDFQWlmA='

Commit

vyos@r14# commit
[ interfaces vxlan vxlan0 ]

WARNING: RFC7348 recommends VXLAN tunnels preserve a 1500 byte MTU

[edit]
vyos@r14# run show int
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface        IP Address                        S/L  Description
---------        ----------                        ---  -----------
dum0             -                                 u/u  
eth0             192.168.122.14/24                 u/u  WAN
eth1             192.0.2.1/24                      u/u  
                 2001:db8::1/64                         

lo               127.0.0.1/8                       u/u  
                 ::1/128                                
vxlan0           -                                 u/u  
wg0              100.64.0.1/24                     u/u  
[edit]
vyos@r14#
aderouineau added a comment.EditedJul 26 2022, 11:41 PM

Here is my WG config:

set interfaces wireguard wg2 address 'REDACTED_IPV6/64'
set interfaces wireguard wg2 peer mypeer address 'REDACTED_IPV4'
set interfaces wireguard wg2 peer mypeer allowed-ips '::/0'
set interfaces wireguard wg2 peer mypeer persistent-keepalive '60'
set interfaces wireguard wg2 peer mypeer port '51820'
set interfaces wireguard wg2 peer mypeer public-key 'REDACTED'
set interfaces wireguard wg2 private-key 'REDACTED'
set interfaces wireguard wg2 vrf 'test'
pasik subscribed.Jul 27 2022, 9:24 AM
Viacheslav added a comment.EditedJul 27 2022, 10:34 AM

I can reproduce it:

set vrf name test table '1010'
set interfaces vxlan vxlan0 address 2001:db8:2020::1/64
set interfaces vxlan vxlan0 remote '2001:db8:2222::1'
set interfaces vxlan vxlan0 mtu '1370'
set interfaces vxlan vxlan0 port '4789'
set interfaces vxlan vxlan0 source-interface 'wg0'
set interfaces vxlan vxlan0 vni '123'
set interfaces wireguard wg0 address '2001:db8:4411::1/64'
set interfaces wireguard wg0 peer PEER01 allowed-ips '::/0'
set interfaces wireguard wg0 peer PEER01 public-key 'VVfR5S0yi+QPEJRLr25ZAfzFnwZM40G5WCZ/7ou7h3k='
set interfaces wireguard wg0 private-key 'yGOy08Kv8KUe8rsO6WHeo5jC7YdOAzQK0SJkDFQWlmA='
set interfaces wireguard wg0 vrf 'test'

Commit:

Traceback (most recent call last):
  File "/usr/libexec/vyos/conf_mode/interfaces-vxlan.py", line 176, in <module>
    apply(c)
  File "/usr/libexec/vyos/conf_mode/interfaces-vxlan.py", line 167, in apply
    v.update(vxlan)
  File "/usr/lib/python3/dist-packages/vyos/ifconfig/interface.py", line 1539, in update
    self.set_mtu(config.get('mtu'))
  File "/usr/lib/python3/dist-packages/vyos/ifconfig/interface.py", line 443, in set_mtu
    return self.set_interface('mtu', mtu)
  File "/usr/lib/python3/dist-packages/vyos/ifconfig/control.py", line 183, in set_interface
    return self._set_command(self.config, name, value)
  File "/usr/lib/python3/dist-packages/vyos/ifconfig/control.py", line 110, in _set_command
    return self._command_set[name].get('format', lambda _: _)(self._cmd(cmd))
  File "/usr/lib/python3/dist-packages/vyos/ifconfig/control.py", line 52, in _cmd
    return cmd(command, self.debug)
  File "/usr/lib/python3/dist-packages/vyos/util.py", line 161, in cmd
    raise OSError(code, feedback)
FileNotFoundError: [Errno 2] failed to run command: ip link set dev vxlan0 mtu 1370
returned: 
exit code: 2

noteworthy:
cmd 'nft -c delete element inet vrf_zones ct_iface_map { "vxlan0" }'
returned (out):

returned (err):
Error: Could not process rule: No such file or directory
delete element inet vrf_zones ct_iface_map { vxlan0 }
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
cmd 'ip link set dev vxlan0 mtu 1370'
returned (out):

It doesn't like MTU 1370 for vxlanX + IPv6 peer

vyos@r14# sudo ip link set dev vxlan0 mtu 1370
RTNETLINK answers: Invalid argument
[edit]
vyos@r14#

Try to change MTU to 1350 or not set this parameter, is should be auto-calculated

set interfaces vxlan vxlan0 mtu '1350'

Require more checks https://github.com/vyos/vyos-1x/blob/8bbaafd554d003078607121e2de662c9c8edee40/src/conf_mode/interfaces-vxlan.py#L111-L124

As it checks if source_address in the config, but it seems not used in this case for vxlanX
As result it minus MTU - 50 bytes, expected MTU - 70 bytes

As an option, we can check the list of remote addresses, and if one of them is IPv6 add +20 to overhead

Viacheslav changed the task status from Open to In progress.Jul 27 2022, 11:37 AM
Viacheslav claimed this task.
Viacheslav added a project: VyOS 1.4 Sagitta.
Viacheslav added a comment.Jul 27 2022, 12:02 PM

PR https://github.com/vyos/vyos-1x/pull/1440

set vrf name test table '1010'
set interfaces vxlan vxlan0 address 2001:db8:2020::1/64
set interfaces vxlan vxlan0 remote '2001:db8:2222::1'
set interfaces vxlan vxlan0 mtu '1370'
set interfaces vxlan vxlan0 port '4789'
set interfaces vxlan vxlan0 source-interface 'wg0'
set interfaces vxlan vxlan0 vni '123'
set interfaces wireguard wg0 address '2001:db8:4411::1/64'
set interfaces wireguard wg0 peer PEER01 allowed-ips '::/0'
set interfaces wireguard wg0 peer PEER01 public-key 'VVfR5S0yi+QPEJRLr25ZAfzFnwZM40G5WCZ/7ou7h3k='
set interfaces wireguard wg0 private-key 'yGOy08Kv8KUe8rsO6WHeo5jC7YdOAzQK0SJkDFQWlmA='
set interfaces wireguard wg0 vrf 'test'

Commit:

vyos@r14# commit
[ interfaces vxlan vxlan0 ]

WARNING: RFC7348 recommends VXLAN tunnels preserve a 1500 byte MTU
Underlaying device MTU is to small (1420 bytes) for VXLAN overhead (70
bytes!)

[[interfaces vxlan vxlan0]] failed
Commit failed
[edit]
vyos@r14#
aderouineau added a comment.EditedJul 28 2022, 5:58 PM

The reason I set an MTU is because I get the following error when unset:

WARNING: RFC7348 recommends VXLAN tunnels preserve a 1500 byte MTU

Underlaying device MTU is to small (1420 bytes) for VXLAN overhead (50
bytes!)

Setting MTU to 1350 does apply without issues. I guess there are issues with overhead calculation when checking the MTU?

As a side note, "underlying" and "too small" are spelled incorrectly in the error message.

Restricted Repository Identity mentioned this in rVYOSONEX9310ed82e1e3: Merge pull request #1440 from sever-sever/T4570.Jul 29 2022, 4:15 PM
Viacheslav mentioned this in rVYOSONEX319102b7f9ea: vxlan: T4570: Verify MTU for remote address if source not defined.Jul 29 2022, 4:15 PM
Viacheslav changed the task status from In progress to Needs testing.Jul 30 2022, 9:37 AM
Viacheslav closed this task as Resolved.Aug 18 2022, 7:38 PM
Viacheslav moved this task from Open to Finished on the VyOS 1.4 Sagitta board.