Page MenuHomeVyOS Platform
Create Task
Maniphest T4120

[VXLAN] add ability to set multiple unicast-remotes
Closed, ResolvedPublicFEATURE REQUEST
Actions

Description

Hi,

with vxlan it is possible to use multiple remote addresses to connect 3 or more hosts to one vxlan-device.

Vyos currently allows only one remote peer (or a multicast group or L3VPN).
Maybe it's because the "ip"-command only allows one.

If you want to join multiple hosts then with Vyos there is no other solution than L3VPN (or multicast).
For 10+ hosts I really do recommend L3VPN. But for only 3-5 hosts multiple remotes would be much simpler. Also if you want to join other linux hosts without FRR.

Using Debian and iproute2 this can be achieved by using something like this in /etc/network/interfaces:

iface vxlan10
        vxlan-id 10
        vxlan_remoteip 10.0.0.1
        vxlan_remoteip 10.0.0.2
        mtu 1450

Proxmox also uses that for their SDN feature.

On the shell you had to add the Interface without the remote parameter

ip link add vxlan10 type vxlan id 10 dev eth0 dstport 0

and then add the remotes using “bridge” commands like this:

bridge fdb append to 00:00:00:00:00:00 dst 10.0.0.1 dev vxlan10
bridge fdb append to 00:00:00:00:00:00 dst 10.0.0.2 dev vxlan10

(it's also possible to define the port for every remote)

Nice documented source from 2017:
https://joejulian.name/post/how-to-configure-linux-vxlans-with-multiple-unicast-endpoints/
(he added a patch for systemd to get this feature there)

At current it is only possible to add one remote like this on Vyos:

set interfaces vxlan vxlan10 remote '10.0.0.1'
set interfaces vxlan vxlan10 vni '10'

Then the "Forwarding Database" (fdb) looks like this:

vyos@vyos# bridge fdb show dev vxlan10
00:00:00:00:00:00 dst 10.0.0.1 self permanent

It would be nice to be able to set something like this:

set interfaces vxlan vxlan10 remote '10.0.0.1'
set interfaces vxlan vxlan10 remote '10.0.0.2'
set interfaces vxlan vxlan10 vni '10'

To get such a forwarding database:

vyos@vyos# bridge fdb show dev vxlan10
00:00:00:00:00:00 dst 10.0.0.1 self permanent
00:00:00:00:00:00 dst 10.0.0.2 self permanent

I already mentioned on the forum:
https://forum.vyos.io/t/using-vxlan-with-multiple-unicast-remotes/

And I already wrote a small patch to implement this feature.
Immediate after getting a Task ID I'll do a PR.

Details

Difficulty level
Hard (possibly days)
Version
-
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Perfectly compatible
Issue type
Improvement (missing useful functionality)

Related Objects

Mentioned In
1.3.3
1.3.1

Event Timeline

aha created this task.Dec 29 2021, 4:43 PM
pasik added a subscriber: pasik.Dec 29 2021, 5:01 PM
aha added a comment.Dec 29 2021, 5:30 PM

PR started:
https://github.com/vyos/vyos-1x/pull/1127

I already tested that feature for some weeks but if it's requested I can also do some tcpdumps and post the results from a lab with 5 routers.

Maybe there is a better solution than copying the template tunnel-remote.xml.i to template tunnel-remotes.xml.i and only add "<multi/>".
And maybe there is also a better solution to execute multiple bridge commands inside vxlan.py.
So please consider my PR as a first idea ;)

aha added a comment.EditedDec 30 2021, 12:13 AM

During multiple tests on my testlab I found two (or three) possible bugs:
1.)
vyos-cli does not prevent to mix IPv4 and IPv6 remotes. Mixing them is not possible with vxlan.

Exampe:

vyos@vyos# show | commands |grep vxlan
set interfaces vxlan vxlan100 remote '10.0.0.1'
set interfaces vxlan vxlan100 remote '10.0.0.2'
set interfaces vxlan vxlan100 vni '100'
[edit]
vyos@vyos# bridge fdb show dev vxlan100
00:00:00:00:00:00 dst 10.0.0.1 self permanent
00:00:00:00:00:00 dst 10.0.0.2 self permanent
vyos@vyos# set interfaces vxlan vxlan100 remote fd42:1983:1::2
[edit]
vyos@vyos# commit
[ interfaces vxlan vxlan100 ]
VyOS had an issue completing a command.
[..]
OSError: [Errno 255] failed to run command: bridge fdb append to 00:00:00:00:00:00 dst fd42:1983:1::2 port 8472 dev vxlan100
returned:
exit code: 255

noteworthy:
cmd 'bridge fdb append to 00:00:00:00:00:00 dst fd42:1983:1::2 port 8472 dev vxlan100'
returned (out):

returned (err):
RTNETLINK answers: Address family not supported by protocol

[[interfaces vxlan vxlan100]] failed
Commit failed
[edit]
vyos@vyos# bridge fdb show dev vxlan100
00:00:00:00:00:00 dst 10.0.0.1 self permanent
00:00:00:00:00:00 dst 10.0.0.2 self permanent

I'm searching now how other services do syntax checking to prevent mixing IPv4 and IPv6.
If I find a solution then I will add a syntax check for that.

2.)
There is a problem when adding multiple IPv6 remotes. This problem does not occur when setting "source-interface".
I am working on it.

It's because with only one remote the vxlan interface will be created through another script (not /usr/lib/python3/dist-packages/vyos/ifconfig/vxlan.py - maybe interface.py) with remote param:

DEBUG/IFCONFIG cmd 'ip link add vxlan100 type vxlan dstport 8472 tos inherit ttl 16 remote fd42:1983:1::2 id 10'

vyos@vyos# ip -d link show dev vxlan100
64: vxlan100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether ae:1c:6a:72:c7:51 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu 68 maxmtu 65535
    vxlan id 10 remote fd42:1983:1::2 srcport 0 0 dstport 8472 tos inherit ttl 16 ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx addrgenmode none numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535

with multiple remotes it is first created with an array as remote:

DEBUG/IFCONFIG cmd 'ip link add vxlan100 type vxlan dstport 8472 tos inherit ttl 16 remote ['fd42:1983:1::3', 'fd42:1983:1::4'] id 10'

(which does not work)
and then again on "vxlan.py" without remote param

DEBUG/IFCONFIG cmd 'ip link add vxlan100 type vxlan dstport 8472 tos inherit ttl 16 id 10'

Without "remote" it is considered to be IPv4 and no other IPv6 address could be added by "bridge" command:

vyos@vyos# sudo bridge fdb append to 00:00:00:00:00:00 dst fd42:1983:1::3 port 8472 dev vxlan100
RTNETLINK answers: Address family not supported by protocol

I am not very happy with that and working on it.
Using a IPv6 "source-interface" is only a workaround.

3.)
It's not possible to use link local addresses. The bridge command also does not accept something like "fe80::2%eth0".
But that maybe generally a problem of vxlan.

PS:

touch /tmp/vyos.ifconfig.debug

was a big help :)

aha added a comment.EditedDec 30 2021, 11:19 AM

Problem (2) with multiple IPv6 remotes fixed.

Any idea how to fix problem (1) to prohibit adding IPv4 and IPv6 remotes at the same time?

EDIT: done

     vxlan vxlan10 {
         remote 10.0.0.1
+        remote fd42:1983:1::2
         source-address 10.0.0.2
         vni 10
     }
vyos@vyos# commit
[ interfaces vxlan vxlan10 ]
IPv4 and IPV6 cannot be mixed

[[interfaces vxlan vxlan10]] failed
Commit failed
Viacheslav changed the task status from Open to In progress.Jan 3 2022, 9:14 AM
Viacheslav assigned this task to aha.
danhusan added a subscriber: danhusan.Feb 4 2022, 10:04 PM
danhusan awarded a token.Feb 4 2022, 10:11 PM
c-po closed this task as Resolved.Feb 21 2022, 6:11 PM
c-po triaged this task as Low priority.
c-po added a project: VyOS 1.3 Equuleus ( 1.3.1).
c-po changed Difficulty level from Unknown (require assessment) to Hard (possibly days).
c-po changed Is it a breaking change? from Config syntax change (migratable) to Perfectly compatible.
c-po moved this task from Need Triage to Finished on the VyOS 1.3 Equuleus ( 1.3.1) board.
c-po moved this task from Need Triage to Finished on the VyOS 1.4 Sagitta board.
dmbaturin mentioned this in 1.3.1.Jun 11 2022, 8:40 AM
dmbaturin mentioned this in 1.3.3.Sep 22 2022, 10:56 AM