Page MenuHomeVyOS Platform

Attempts to delete vlan-to-vni option causes an unhandled exception
Closed, ResolvedPublicBUG

Description

The issue below has been encountered on these builds:

  • 1.4-rolling-202403240523
  • 1.5-rolling-202405240020

Staring from a fresh 1.5-rolling-202405240020 install, I have the following committed config:

interfaces {
    bridge br0 {
        enable-vlan
        member {
            interface vxlan0 {
            }
        }
    }
    dummy dum0 {
        address "1.1.1.1/32"
    }
    ethernet eth0 {
        hw-id "bc:24:11:0b:0c:db"
    }
    loopback lo {
    }
    vxlan vxlan0 {
        parameters {
            external
        }
        source-address "1.1.1.1"
        vlan-to-vni 10 {
            vni "10"
        }
    }
}
system {
    config-management {
        commit-revisions "100"
    }
    host-name "vyos"
    login {
        [removed]
    }
    syslog {
        global {
            facility all {
                level "info"
            }
            facility local7 {
                level "debug"
            }
        }
    }
}


// Warning: Do not remove the following line.
// vyos-config-version: "bgp@5:broadcast-relay@1:cluster@2:config-management@1:conntrack@5:conntrack-sync@2:container@2:dhcp-relay@2:dhcp-server@11:dhcpv6-server@5:dns-dynamic@4:dns-forwarding@4:firewall@15:flow-accounting@1:https@6:ids@1:interfaces@32:ipoe-server@3:ipsec@13:isis@3:l2tp@9:lldp@2:mdns@1:monitoring@1:nat@8:nat66@3:ntp@3:openconnect@3:openvpn@1:ospf@2:pim@1:policy@8:pppoe-server@10:pptp@5:qos@2:quagga@11:rip@1:rpki@2:salt@1:snmp@3:ssh@2:sstp@6:system@27:vrf@3:vrrp@4:vyos-accel-ppp@2:wanloadbalance@3:webproxy@2"
// Release version: 1.5-rolling-202405240020

When I attempt to commit delete interfaces vxlan vxlan0 vlan-to-vni, I receive this error:

[ interfaces vxlan vxlan0 ]
VyOS had an issue completing a command.

We are sorry that you encountered a problem while using VyOS.
There are a few things you can do to help us (and yourself):
- Contact us using the online help desk if you have a subscription:
  https://support.vyos.io/
- Make sure you are running the latest version of VyOS available at:
  https://vyos.net/get/
- Consult the community forum to see how to handle this issue:
  https://forum.vyos.io
- Join us on Slack where our users exchange help and advice:
  https://vyos.slack.com

When reporting problems, please include as much information as possible:
- do not obfuscate any data (feel free to contact us privately if your
  business policy requires it)
- and include all the information presented below

Report time:      2024-05-25 23:26:05
Image version:    VyOS 1.5-rolling-202405240020
Release train:    current

Built by:         [email protected]
Built on:         Fri 24 May 2024 03:00 UTC
Build UUID:       4e05c3b5-48bc-42c4-9231-2b29cfb85fcf
Build commit ID:  869792ab01d9b4

Architecture:     x86_64
Boot via:         installed image
System type:      KVM guest

Hardware vendor:  QEMU
Hardware model:   Standard PC (Q35 + ICH9, 2009)
Hardware S/N:
Hardware UUID:    96e83f01-0838-4f5d-8e02-a2f17edda36c

Traceback (most recent call last):
  File "/usr/libexec/vyos/conf_mode/interfaces_vxlan.py", line 231, in <module>
    apply(c)
  File "/usr/libexec/vyos/conf_mode/interfaces_vxlan.py", line 222, in apply
    v.update(vxlan)
  File "/usr/lib/python3/dist-packages/vyos/ifconfig/vxlan.py", line 188, in update
    self.set_vlan_vni_mapping(dict_search('vlan_to_vni', config) != None)
  File "/usr/lib/python3/dist-packages/vyos/ifconfig/vxlan.py", line 141, in set_vlan_vni_mapping
    cur_vni_filter = get_vxlan_vni_filter(self.ifname)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/vyos/utils/network.py", line 508, in get_vxlan_vni_filter
    tmp = loads(cmd(f'bridge --json vni show dev {interface}'))
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/vyos/utils/process.py", line 155, in cmd
    raise OSError(code, feedback)
PermissionError: [Errno 1] failed to run command: bridge --json vni show dev vxlan0
returned: [
exit code: 1

noteworthy:
cmd 'bridge --json vni show dev vxlan0'
returned (out):
[
returned (err):
RTNETLINK answers: Invalid argument
Dump ternminated

[[interfaces vxlan vxlan0]] failed
Commit failed

Details

Difficulty level
Unknown (require assessment)
Version
1.4-rolling-202403240523 through 1.5-rolling-202405240020
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Event Timeline

On testing, it looks like vyos.utils.network.get_vxlan_vni_filter() doesn't know how to handle when there are no vni filters installed.

I'm able to replicate the issue with your config, and adding set interfaces vxlan vxlan0 parameters vni-filter appears to negate the issue entirely.

I'll whip up a PR with a suggested fix when I have a moment. It might be a problem with bridge itself misbehaving - that'll be a bit more fun to fix.

PR opened: https://github.com/vyos/vyos-1x/pull/3573

If possible, can you try duplicating with more than one vni configured, then delete one of them? I'm pretty sure I tried that case a few weeks ago on an older 1.4 rolling with same issue. This was just the minimum config to reproduce.

This is without my patch applied, confirming both the problem and a workaround:

vyos@vyos# compare
[interfaces]
+ bridge br0 {
+     enable-vlan
+     member {
+         interface vxlan0 {
+         }
+     }
+ }
+ dummy dum0 {
+     address "1.1.1.1/32"
+ }
+ vxlan vxlan0 {
+     parameters {
+         external
+     }
+     source-address "1.1.1.1"
+     vlan-to-vni 10 {
+         vni "10"
+     }
+     vlan-to-vni 20 {
+         vni "20"
+     }
+     vlan-to-vni 30 {
+         vni "30"
+     }
+ }

[edit]
vyos@vyos# commit
[edit]
vyos@vyos# delete interfaces vxlan vxlan0 vlan-to-vni 30
[edit]
vyos@vyos# commit
[ interfaces vxlan vxlan0 ]
VyOS had an issue completing a command.

[...snipped big long error message...]

Commit failed
[edit]
vyos@vyos# discard

  Changes have been discarded

[edit]
vyos@vyos# set interfaces vxlan vxlan0 parameters vni-filter 
[edit]
vyos@vyos# commit
[edit]
vyos@vyos# delete interfaces vxlan vxlan0 vlan-to-vni 30
[edit]
vyos@vyos# commit
[edit]
vyos@vyos# rollback-soft 1
Rollback diff has been applied.
Use "compare" to review the changes or "commit" to apply them.
[edit]
vyos@vyos# commit
[edit]
vyos@vyos#

With multiple vxlan interfaces with parameter external, vni-filter is required, and I'm unable to get it to fault out through scenarios with one or multiple VNI adds/removes from either or any combination of vxlan interfaces.

EDIT: A full reboot followed by deleting a vlan-to-vni mapping with vni-filter enabled is also perfectly fine - it doesn't appear like the error leaves inconsistent state behind that allows the workaround to work. I'd already assumed this from the code but just ran through to double check.

With the patch applied, I also cannot get it to fault out. While my detailed understanding of VXLAN operation under Linux isn't complete, it looks like the underlying is adding/removing all the right state for everything to fit together, and the smoketests pass without issue.

Are there any specific scenarios you'd like me to run it through to double check?

Great work, thanks for checking that case too.

c-po moved this task from Need Triage to Finished on the VyOS 1.5 Circinus board.
c-po edited projects, added VyOS 1.4 Sagitta (1.4.1); removed VyOS 1.4 Sagitta.
c-po moved this task from Need Triage to Finished on the VyOS 1.4 Sagitta (1.4.1) board.
c-po moved this task from Finished to In Progress on the VyOS 1.4 Sagitta (1.4.1) board.
c-po moved this task from In Progress to Finished on the VyOS 1.4 Sagitta (1.4.1) board.
dmbaturin renamed this task from Cannot delete vxlan vlan-to-vni to Attempts to delete vlan-to-vni option causes an unhandled exception.Jul 1 2024, 2:27 PM
dmbaturin removed a project: VyOS 1.5 Circinus.