Page MenuHomeVyOS Platform
Create Task
Maniphest T3939

When xdp is enabled on ethernet interface, interface crashes on reboot and is not started
Closed, InvalidPublicBUG
Actions

Description

Hardware is a generic Vmware VM, ethernet eth0 is VMXNET3, but as this appears to be a permissions problem, it's probably not relevant.

vyos@xdp-check:~$ show version

Version:          VyOS 1.4-rolling-202110220645
Release train:    sagitta

Built by:         autobuild@vyos.net
Built on:         Fri 22 Oct 2021 06:45 UTC
Build UUID:       66b5f3d4-440a-4f16-a6e9-559b123a820d
Build commit ID:  f6c1a927ca63da

Architecture:     x86_64
Boot via:         installed image
System type:      VMware guest

Hardware vendor:  VMware, Inc.
Hardware model:   VMware Virtual Platform
Hardware S/N:     VMware-42 27 c2 7a db f9 f0 4c-ce 2e 6c 7b 07 b2 41 c1
Hardware UUID:    7ac22742-f9db-4cf0-ce2e-6c7b07b241c1

Copyright:        VyOS maintainers and contributors
vyos@xdp-check:~$ show interfaces ethernet eth0 physical
Settings for eth0:
        Supported ports: [ TP ]
        Supported link modes:   1000baseT/Full
                                10000baseT/Full
        Supported pause frame use: No
        Supports auto-negotiation: No
        Supported FEC modes: Not reported
        Advertised link modes:  Not reported
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: off
        MDI-X: Unknown
        Supports Wake-on: uag
        Wake-on: d
        Link detected: yes
Ring parameters for eth0:
Pre-set maximums:
RX:             4096
RX Mini:        2048
RX Jumbo:       4096
TX:             4096
Current hardware settings:
RX:             1024
RX Mini:        128
RX Jumbo:       256
TX:             512

driver: vmxnet3
version: 1.5.0.0-k-NAPI
firmware-version:
expansion-rom-version:
bus-info: 0000:0b:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no
vyos@xdp-check:~$ show interfaces ethernet eth0
eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:a7:7a:1e brd ff:ff:ff:ff:ff:ff
    inet 203.7.226.10/24 brd 203.7.226.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:fea7:7a1e/64 scope link
       valid_lft forever preferred_lft forever

    RX:   bytes  packets  errors  dropped  overrun       mcast
         290951     4692       0        0        0           0
    TX:   bytes  packets  errors  dropped  carrier  collisions
          36016      352       0        0        0           0
vyos@xdp-check:~$

The interface is up and working fine. I can enable xdp without a problem.

vyos@xdp-check:~$ conf
[edit]
vyos@xdp-check# set interfaces ethernet eth0 xdp
[edit]
vyos@xdp-check# commit
[edit]
vyos@xdp-check# save
Saving configuration to '/config/config.boot'...
Done
[edit]
vyos@xdp-check# exit
exit
vyos@xdp-check:~$ show interfaces ethernet eth0
eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 xdpgeneric/id:85 qdisc mq state UP group default qlen 1000
    link/ether 00:50:56:a7:7a:1e brd ff:ff:ff:ff:ff:ff
    inet 203.7.226.10/24 brd 203.7.226.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::250:56ff:fea7:7a1e/64 scope link
       valid_lft forever preferred_lft forever

    RX:   bytes  packets  errors  dropped  overrun       mcast
         410959     6659       0        0        0           0
    TX:   bytes  packets  errors  dropped  carrier  collisions
          46234      471       0        0        0           0
vyos@xdp-check:~$

At this point, everything appears to be working. However, after a reboot, vyos-config fais and it won't bring up eth0

Console screenshot: https://i.imgur.com/0u3yGDX.png

I can load the /config/boot.config without a problem: https://i.imgur.com/8DVtulD.png

And the interface then comes up: https://i.imgur.com/agw5Tjo.png

In /var/log/messages, the xdp_loader command is failing:

Oct 24 22:58:18 xdp-check vyos-configd[648]: Sending response 1
Oct 24 22:58:18 xdp-check vyos-configd[648]: Received message: {"type": "node", "data": "VYOS_TAGNODE_VALUE=eth0/usr/libexec/vyos/conf_mode/interfaces-ethernet.py"}
Oct 24 22:58:18 xdp-check isisd[1031]: circuit already disconnected
Oct 24 22:58:18 xdp-check isisd[1031]: circuit already disconnected
Oct 24 22:58:19 xdp-check vyos-configd[648]: [Errno 40] failed to run command: xdp_loader -d eth0 --auto-mode -F --progsec xdp_router --filename /usr/share/vyos/xdp/xdp_prog_kern.o && xdp_prog_user -d eth0
Oct 24 22:58:19 xdp-check vyos-configd[648]: returned:
Oct 24 22:58:19 xdp-check vyos-configd[648]: exit code: 40
Oct 24 22:58:19 xdp-check vyos-configd[648]: Sending response 4
Oct 24 22:58:19 xdp-check Report time:      2021-10-24 22:58:19
Oct 24 22:58:19 xdp-check Image version:    VyOS 1.4-rolling-202110220645
Oct 24 22:58:19 xdp-check Release train:    sagitta
Oct 24 22:58:19 xdp-check Built by:         autobuild@vyos.net
Oct 24 22:58:19 xdp-check Built on:         Fri 22 Oct 2021 06:45 UTC
Oct 24 22:58:19 xdp-check Build UUID:       66b5f3d4-440a-4f16-a6e9-559b123a820d
Oct 24 22:58:19 xdp-check Build commit ID:  f6c1a927ca63da
Oct 24 22:58:19 xdp-check Architecture:     x86_64
Oct 24 22:58:19 xdp-check Boot via:         installed image
Oct 24 22:58:19 xdp-check System type:      VMware guest
Oct 24 22:58:19 xdp-check Hardware vendor:  VMware, Inc.
Oct 24 22:58:19 xdp-check Hardware model:   VMware Virtual Platform
Oct 24 22:58:19 xdp-check Hardware S/N:     VMware-42 27 c2 7a db f9 f0 4c-ce 2e 6c 7b 07 b2 41 c1
Oct 24 22:58:19 xdp-check Hardware UUID:    7ac22742-f9db-4cf0-ce2e-6c7b07b241c1
Oct 24 22:58:19 xdp-check Traceback (most recent call last):
Oct 24 22:58:19 xdp-check   File "/usr/libexec/vyos/conf_mode/interfaces-ethernet.py", line 200, in <module>
Oct 24 22:58:19 xdp-check     apply(c)
Oct 24 22:58:19 xdp-check   File "/usr/libexec/vyos/conf_mode/interfaces-ethernet.py", line 189, in apply
Oct 24 22:58:19 xdp-check     e.update(ethernet)
Oct 24 22:58:19 xdp-check   File "/usr/lib/python3/dist-packages/vyos/ifconfig/ethernet.py", line 339, in update
Oct 24 22:58:19 xdp-check     super().update(config)
Oct 24 22:58:19 xdp-check   File "/usr/lib/python3/dist-packages/vyos/ifconfig/interface.py", line 1519, in update
Oct 24 22:58:19 xdp-check     self.set_xdp('xdp' in config)
Oct 24 22:58:19 xdp-check   File "/usr/lib/python3/dist-packages/vyos/ifconfig/interface.py", line 1338, in set_xdp
Oct 24 22:58:19 xdp-check     return self._cmd(cmd)
Oct 24 22:58:19 xdp-check   File "/usr/lib/python3/dist-packages/vyos/ifconfig/control.py", line 52, in _cmd
Oct 24 22:58:19 xdp-check     return cmd(command, self.debug)
Oct 24 22:58:19 xdp-check   File "/usr/lib/python3/dist-packages/vyos/util.py", line 161, in cmd
Oct 24 22:58:19 xdp-check     raise OSError(code, feedback)
Oct 24 22:58:19 xdp-check OSError: [Errno 40] failed to run command: xdp_loader -d eth0 --auto-mode -F --progsec xdp_router --filename /usr/share/vyos/xdp/xdp_prog_kern.o && xdp_prog_user -d eth0
Oct 24 22:58:19 xdp-check returned:
Oct 24 22:58:19 xdp-check exit code: 40
Oct 24 22:58:19 xdp-check noteworthy:
Oct 24 22:58:19 xdp-check cmd 'ethtool --show-pause eth0'
Oct 24 22:58:19 xdp-check returned (out):
Oct 24 22:58:19 xdp-check Pause parameters for eth0:
Oct 24 22:58:19 xdp-check returned (err):
Oct 24 22:58:19 xdp-check Cannot get device pause settings: Operation not supported
Oct 24 22:58:19 xdp-check cmd 'ethtool --show-pause eth0'
Oct 24 22:58:19 xdp-check returned (out):
Oct 24 22:58:19 xdp-check Pause parameters for eth0:
Oct 24 22:58:19 xdp-check returned (err):
Oct 24 22:58:19 xdp-check Cannot get device pause settings: Operation not supported
Oct 24 22:58:19 xdp-check cmd 'nft -c delete element inet vrf_zones ct_iface_map { "eth0" }'
Oct 24 22:58:19 xdp-check returned (out):
Oct 24 22:58:19 xdp-check returned (err):
Oct 24 22:58:19 xdp-check Error: No such file or directory
Oct 24 22:58:19 xdp-check delete element inet vrf_zones ct_iface_map { eth0 }
Oct 24 22:58:19 xdp-check                     ^^^^^^^^^
Oct 24 22:58:19 xdp-check cmd 'xdp_loader -d eth0 --auto-mode -F --progsec xdp_router --filename /usr/share/vyos/xdp/xdp_prog_kern.o && xdp_prog_user -d eth0'
Oct 24 22:58:19 xdp-check returned (out):
Oct 24 22:58:19 xdp-check returned (err):
Oct 24 22:58:19 xdp-check libbpf: failed to create map (name: 'tx_port'): Invalid argument(-22)
Oct 24 22:58:19 xdp-check libbpf: failed to load object '/usr/share/vyos/xdp/xdp_prog_kern.o'
Oct 24 22:58:19 xdp-check ERR: loading BPF-OBJ file(/usr/share/vyos/xdp/xdp_prog_kern.o) (-22): Invalid argument
Oct 24 22:58:19 xdp-check ERR: loading file: /usr/share/vyos/xdp/xdp_prog_kern.o
Oct 24 22:58:21 xdp-check vyos-configd[648]: Received message: {"type": "node", "data": "/usr/libexec/vyos/conf_mode/system-syslog.py"}

Hopefully this is the debug when running the load /config/boot.config from the console which ALSO has an exit 40, but appears to continue on without crashing:

Oct 24 22:59:53 xdp-check systemd[1]: opt-vyatta-config-tmp-new_config_1721.mount: Succeeded.
Oct 24 22:59:53 xdp-check login[1711]: pam_unix(login:session): session opened for user vyos(uid=1002) by LOGIN(uid=0)
Oct 24 22:59:53 xdp-check systemd[1]: Created slice User Slice of UID 1002.
Oct 24 22:59:53 xdp-check systemd[1]: Starting User Runtime Directory /run/user/1002...
Oct 24 22:59:53 xdp-check systemd-logind[786]: New session 1 of user vyos.
Oct 24 22:59:53 xdp-check systemd[1]: Finished User Runtime Directory /run/user/1002.
Oct 24 22:59:53 xdp-check systemd[1]: Starting User Manager for UID 1002...
Oct 24 22:59:53 xdp-check systemd: pam_unix(systemd-user:session): session opened for user vyos(uid=1002) by (uid=0)
Oct 24 22:59:54 xdp-check systemd[1790]: gpgconf: error running '/usr/lib/gnupg/scdaemon': probably not installed
Oct 24 22:59:54 xdp-check systemd[1785]: Queued start job for default target Main User Target.
Oct 24 22:59:54 xdp-check systemd[1785]: Created slice User Application Slice.
Oct 24 22:59:54 xdp-check systemd[1785]: Reached target Paths.
Oct 24 22:59:54 xdp-check systemd[1785]: Reached target Timers.
Oct 24 22:59:54 xdp-check systemd[1785]: Listening on GnuPG network certificate management daemon.
Oct 24 22:59:54 xdp-check systemd[1785]: Listening on GnuPG cryptographic agent and passphrase cache (access for web browsers).
Oct 24 22:59:54 xdp-check systemd[1785]: Listening on GnuPG cryptographic agent and passphrase cache (restricted).
Oct 24 22:59:54 xdp-check systemd[1785]: Listening on GnuPG cryptographic agent (ssh-agent emulation).
Oct 24 22:59:54 xdp-check systemd[1785]: Listening on GnuPG cryptographic agent and passphrase cache.
Oct 24 22:59:54 xdp-check systemd[1785]: Listening on Podman API Socket.
Oct 24 22:59:54 xdp-check systemd[1785]: Reached target Sockets.
Oct 24 22:59:54 xdp-check systemd[1785]: Reached target Basic System.
Oct 24 22:59:54 xdp-check systemd[1785]: Reached target Main User Target.
Oct 24 22:59:54 xdp-check systemd[1785]: Startup finished in 798ms.
Oct 24 22:59:54 xdp-check systemd[1]: Started User Manager for UID 1002.
Oct 24 22:59:54 xdp-check systemd[1]: Started Session 1 of user vyos.
Oct 24 23:00:12 xdp-check vyos-configd[648]: Received message: {"type": "init"}
Oct 24 23:00:12 xdp-check vyos-configd[648]: config session pid is 1927
Oct 24 23:00:12 xdp-check vyos-configd[648]: Received message: {"type": "node", "data": "VYOS_TAGNODE_VALUE=eth0/usr/libexec/vyos/conf_mode/interfaces-ethernet.py"}
Oct 24 23:00:13 xdp-check vyos-configd[648]: [Errno 40] failed to run command: xdp_loader -d eth0 --auto-mode -F --progsec xdp_router --filename /usr/share/vyos/xdp/xdp_prog_kern.o && xdp_prog_user -d eth0
Oct 24 23:00:13 xdp-check vyos-configd[648]: returned:
Oct 24 23:00:13 xdp-check vyos-configd[648]: exit code: 40
Oct 24 23:00:13 xdp-check vyos-configd[648]: Sending response 4
Oct 24 23:00:13 xdp-check kernel: [  148.395769] vmxnet3 0000:0b:00.0 eth0: intr type 3, mode 0, 5 vectors allocated
Oct 24 23:00:13 xdp-check kernel: [  148.397510] vmxnet3 0000:0b:00.0 eth0: NIC Link is Up 10000 Mbps
Oct 24 23:00:13 xdp-check netplugd[867]: eth0: state DOWN flags 0x00001002 BROADCAST,MULTICAST -> 0x00011043 UP,BROADCAST,RUNNING,MULTICAST,10000
Oct 24 23:00:13 xdp-check netplugd[2101]: /etc/netplug/netplug eth0 in -> pid 2101
Oct 24 23:00:13 xdp-check netplugd[867]: eth0: state INNING pid 2101 exited status 0
Oct 24 23:00:13 xdp-check systemd[1785]: opt-vyatta-config-tmp-new_config_1927.mount: Succeeded.
Oct 24 23:00:13 xdp-check systemd[1]: opt-vyatta-config-tmp-new_config_1927.mount: Succeeded.
Oct 24 23:00:14 xdp-check commit: Successful change to active configuration by user vyos on /dev/tty1
Oct 24 23:00:16 xdp-check ntpd[1616]: Listen normally on 5 eth0 203.7.226.10:123
Oct 24 23:00:16 xdp-check ntpd[1616]: Listen normally on 6 eth0 [fe80::250:56ff:fea7:7a1e%2]:123
Oct 24 23:00:22 xdp-check systemd[1785]: opt-vyatta-config-tmp-new_config_1927.mount: Succeeded.
Oct 24 23:00:22 xdp-check systemd[1]: opt-vyatta-config-tmp-new_config_1927.mount: Succeeded.

Details

Version
VyOS 1.4-rolling-202110220645
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)