Page MenuHomeVyOS Platform

VPP interfaces are not returned to the original dataplane in some cases
Closed, ResolvedPublicBUG

Description

VPP interfaces are not returned to the original dataplane if partly commit was applied, but another part is not applied

Original driver:

vyos@r16:~$ sudo ethtool -i eth1
driver: virtio_net
version: 1.0.0

To reproduce:

set vpp settings interface eth1 driver dpdk 
set vpp settings interface eth1 rx-mode adaptive
commit

commit:

vyos@r16# commit
[ vpp ]
WARNING: offload option in eth1 settings is not supported by VPP interfaces. It will be ignored.
Traceback (most recent call last):
  File "/usr/libexec/vyos/services/vyos-configd", line 139, in run_script
    script.apply(c)
  File "/usr/libexec/vyos//conf_mode/vpp.py", line 560, in apply
    vpp_control.iface_rxmode(iface, rx_mode)
  File "/usr/lib/python3/dist-packages/vyos/vpp/control_vpp.py", line 81, in check_retval_wrapper
    raise VPPValueError(f'VPP API call failed: {return_value.retval}')
vpp_papi.vpp_papi.VPPValueError: VPP API call failed: -9

[[vpp]] failed
Commit failed
[edit]
vyos@r16#

In this step vpp attached the interface eth1 to vpp dataplane

vyos@r16# sudo vppctl show int
              Name               Idx    State  MTU (L3/IP4/IP6/MPLS)     Counter          Count     
eth1                              1     down         9000/0/0/0     tx-error                       1
local0                            0     down          0/0/0/0       
tap1                              2      up          9000/0/0/0     rx packets                     1
                                                                    rx bytes                      90
                                                                    drops                          1
                                                                    ip6                            1
[edit]
vyos@r16#

Delete vpp

vyos@r16# compare 
+ vpp {
+     settings {
+         interface eth1 {
+             driver "dpdk"
+             rx-mode "adaptive"
+         }
+     }
+ }

[edit]
vyos@r16# delete vpp 
[edit]
vyos@r16# commit
[edit]
vyos@r16#

In this step it does nothing and eth1 still in the vpp dataplane

vyos@r16# sudo ethtool -i eth1
driver: tun

vyos@r16# sudo vppctl show int
              Name               Idx    State  MTU (L3/IP4/IP6/MPLS)     Counter          Count     
eth1                              1     down         9000/0/0/0     tx-error                       1
local0                            0     down          0/0/0/0       
tap1                              2      up          9000/0/0/0     rx packets                     1
                                                                    rx bytes                      90
                                                                    drops                          1
                                                                    ip6                            1
[edit]
vyos@r16#

Of course we cannot attach this interface again to the VPP dataplane

vyos@r16# set vpp settings interface eth1 driver dpdk 
[edit]
vyos@r16# commit
[ vpp ]
WARNING: offload option in eth1 settings is not supported by VPP interfaces. It will be ignored.
Traceback (most recent call last):
  File "/usr/libexec/vyos/services/vyos-configd", line 137, in run_script
    script.verify(c)
  File "/usr/libexec/vyos//conf_mode/vpp.py", line 330, in verify
    if not verify_dev_driver(iface, iface_config['driver']):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/vyos/vpp/config_verify.py", line 149, in verify_dev_driver
    driver: str = get_eth_driver(iface_name)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/vyos/vpp/control_host.py", line 271, in get_eth_driver
    raise FileNotFoundError(f'PCI device {iface} not found in ethernet interfaces')
FileNotFoundError: PCI device eth1 not found in ethernet interfaces

[[vpp]] failed
Commit failed
[edit]
vyos@r16#

Only reboot can help in this case

Details

Version
VyOS 1.5-current-vpp-202501291400
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Event Timeline

Viacheslav triaged this task as High priority.

Additional related logs:

Jan 30 12:35:46 r16 netplugd[1022]: eth1: state INNING flags 0x00011043 UP,BROADCAST,RUNNING,MULTICAST,10000 -> 0x00001002 BROADCAST,MULTICAST
Jan 30 12:35:46 r16 vpp[2756]: interface/rx-queue: setting rx mode on the interface eth1 queue-id 0 failed.
                                  dpdk_interface_rx_mode_change: unsupported op (is the interface up?)
Jan 30 12:35:46 r16 vyos-configd[751]: Traceback (most recent call last):
Jan 30 12:35:46 r16 vyos-configd[751]:   File "/usr/libexec/vyos/services/vyos-configd", line 139, in run_script
Jan 30 12:35:46 r16 vyos-configd[751]:     script.apply(c)
Jan 30 12:35:46 r16 vyos-configd[751]:   File "/usr/libexec/vyos//conf_mode/vpp.py", line 560, in apply
Jan 30 12:35:46 r16 vyos-configd[751]:     vpp_control.iface_rxmode(iface, rx_mode)
Jan 30 12:35:46 r16 vyos-configd[751]:   File "/usr/lib/python3/dist-packages/vyos/vpp/control_vpp.py", line 81, in check_retval_wrapper
Jan 30 12:35:46 r16 vyos-configd[751]:     raise VPPValueError(f'VPP API call failed: {return_value.retval}')
Jan 30 12:35:46 r16 vyos-configd[751]: vpp_papi.vpp_papi.VPPValueError: VPP API call failed: -9
Jan 30 12:35:46 r16 vpp[2756]: set_hw_interface_change_rx_mode: unable to set rx-mode on interface eth1 queue-id 0.
Jan 30 12:35:46 r16 vyos-configd[751]: Sending reply: error_code 2 with output
Jan 30 12:35:46 r16 vyos-configd[751]: scripts_called: ['vpp']
Jan 30 12:35:47 r16 systemd[1]: opt-vyatta-config-tmp-new_config_2587.mount: Deactivated successfully.
Jan 30 12:35:47 r16 netplugd[1022]: eth1: state INNING flags 0x00001043 UP,BROADCAST,RUNNING,MULTICAST -> 0x00001002 BROADCAST,MULTICAST
Jan 30 12:35:47 r16 netplugd[2825]: /etc/netplug/netplug eth1 probe -> pid 2825
Jan 30 12:35:48 r16 netplugd[2826]: /etc/netplug/netplug eth1 probe -> pid 2826
Jan 30 12:35:48 r16 netplugd[1022]: eth1: state PROBING pid 2826 exited status 256
Jan 30 12:35:48 r16 netplugd[1022]: Could not bring eth1 back up
Jan 30 12:37:04 r16 systemd[1]: opt-vyatta-config-tmp-new_config_2587.mount: Deactivated successfully.
Jan 30 12:38:55 r16 vyos-configd[751]: Received message: {"type": "init"}
Jan 30 12:38:55 r16 vyos-configd[751]: config session pid is 2587
Jan 30 12:38:55 r16 vyos-configd[751]: config session sudo_user is vyos
Jan 30 12:38:55 r16 vyos-configd[751]: commit_scripts: ['vpp']
Jan 30 12:38:55 r16 vyos-configd[751]: Received message: {"type": "node", "last": true, "data": "/usr/libexec/vyos/conf_mode/vpp.py"}
Jan 30 12:38:55 r16 vyos-configd[751]: Traceback (most recent call last):
Jan 30 12:38:55 r16 vyos-configd[751]:   File "/usr/libexec/vyos/services/vyos-configd", line 137, in run_script
Jan 30 12:38:55 r16 vyos-configd[751]:     script.verify(c)
Jan 30 12:38:55 r16 vyos-configd[751]:   File "/usr/libexec/vyos//conf_mode/vpp.py", line 330, in verify
Jan 30 12:38:55 r16 vyos-configd[751]:     if not verify_dev_driver(iface, iface_config['driver']):
Jan 30 12:38:55 r16 vyos-configd[751]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Jan 30 12:38:55 r16 vyos-configd[751]:   File "/usr/lib/python3/dist-packages/vyos/vpp/config_verify.py", line 149, in verify_dev_driver
Jan 30 12:38:55 r16 vyos-configd[751]:     driver: str = get_eth_driver(iface_name)
Jan 30 12:38:55 r16 vyos-configd[751]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
Jan 30 12:38:55 r16 vyos-configd[751]:   File "/usr/lib/python3/dist-packages/vyos/vpp/control_host.py", line 271, in get_eth_driver
Jan 30 12:38:55 r16 vyos-configd[751]:     raise FileNotFoundError(f'PCI device {iface} not found in ethernet interfaces')
Jan 30 12:38:55 r16 vyos-configd[751]: FileNotFoundError: PCI device eth1 not found in ethernet interfaces
Jan 30 12:38:55 r16 vyos-configd[751]: Sending reply: error_code 2 with output
Jan 30 12:38:55 r16 vyos-configd[751]: scripts_called: ['vpp']
Jan 30 12:38:55 r16 systemd[1]: opt-vyatta-config-tmp-new_config_2587.mount: Deactivated successfully.
Jan 30 12:45:07 r16 systemd[1]: Starting systemd-tmpfiles-clean.service - Cleanup of Temporary Directories...
Jan 30 12:45:07 r16 systemd[1]: systemd-tmpfiles-clean.service: Deactivated successfully.
Jan 30 12:45:07 r16 systemd[1]: Finished systemd-tmpfiles-clean.service - Cleanup of Temporary Directories.
Jan 30 12:45:07 r16 systemd[1]: run-credentials-systemd\x2dtmpfiles\x2dclean.service.mount: Deactivated successfully.
Viacheslav claimed this task.

Fixed, as we use a daemon that returns to the previous config if something is wrong.