Page MenuHomeVyOS Platform

VPP fails to commit on AWS unless vfio module is manually loaded
Closed, ResolvedPublicBUG

Description

Commands:

set system option kernel disable-mitigations
set system option kernel memory hugepage-size 2M hugepage-count '2100'

set vpp settings interface eth1 dpdk-options promisc
set vpp settings interface eth1 driver 'dpdk'
set vpp settings lcp netlink batch-delay-ms '10'
set vpp settings lcp netlink batch-size '1024'
set vpp settings lcp netlink rx-buffer-size '8388608'
set vpp settings memory main-heap-page-size '2M'
set vpp settings memory main-heap-size '3G'
set vpp settings statseg size '128M'
set vpp settings unix poll-sleep-usec '222'

After the commit:

vyos@VyOS-on-AWS# set vpp settings interface eth1 dpdk-options promisc
[edit]
vyos@VyOS-on-AWS# set vpp settings interface eth1 driver 'dpdk'
[edit]
vyos@VyOS-on-AWS# set vpp settings lcp netlink batch-delay-ms '10'
[edit]
vyos@VyOS-on-AWS# set vpp settings lcp netlink batch-size '1024'
[edit]
vyos@VyOS-on-AWS# set vpp settings lcp netlink rx-buffer-size '8388608'
[edit]
vyos@VyOS-on-AWS# set vpp settings memory main-heap-page-size '2M'
[edit]
vyos@VyOS-on-AWS# set vpp settings memory main-heap-size '3G'
[edit]
vyos@VyOS-on-AWS# set vpp settings statseg size '128M'
[edit]
vyos@VyOS-on-AWS# set vpp settings unix poll-sleep-usec '222'
[edit]
vyos@VyOS-on-AWS# commit
[ vpp ]
Traceback (most recent call last):
  File "/usr/libexec/vyos/services/vyos-configd", line 156, in run_script
    script.apply(c)
  File "/usr/libexec/vyos/conf_mode/vpp.py", line 538, in apply
    control_host.unsafe_noiommu_mode(True)
  File "/usr/lib/python3/dist-packages/vyos/vpp/control_host.py", line 289, in unsafe_noiommu_mode
    current_status: str = param_path.read_text().strip()
                          ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/pathlib.py", line 1059, in read_text
    with self.open(mode='r', encoding=encoding, errors=errors) as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/pathlib.py", line 1045, in open
    return io.open(self, mode, buffering, encoding, errors, newline)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/sys/module/vfio/parameters/enable_unsafe_noiommu_mode'

[[vpp]] failed
Commit failed
[edit]
vyos@VyOS-on-AWS# run sh version
Version:          VyOS 2025.07.25-0021-rolling
Release train:    current
Release flavor:   generic

Built by:         autobuild@vyos.net
Built on:         Fri 25 Jul 2025 00:21 UTC
Build UUID:       62d2b667-236e-4448-80af-a822c3b6bbc5
Build commit ID:  779c3429e1d9d4

Architecture:     x86_64
Boot via:         installed image
System type:      KVM guest
Secure Boot:      n/a (BIOS)

Hardware vendor:  Amazon EC2
Hardware model:   c6in.2xlarge
Hardware S/N:     ec21ff9d-8d07-f0e4-3e29-0ba6265c7f82
Hardware UUID:    ec21ff9d-8d07-f0e4-3e29-0ba6265c7f82

Copyright:        VyOS maintainers and contributors
[edit]
vyos@VyOS-on-AWS#

When we load the vfio module, commit completes successfully, and VPP interfaces come up as expected:

vyos@VyOS-on-AWS# sudo modprobe vfio
[edit]
vyos@VyOS-on-AWS# sudo lsmod | grep vfio
vfio_iommu_type1       45056  0
vfio                   65536  1 vfio_iommu_type1
[edit]
vyos@VyOS-on-AWS# sudo ls /sys/module/vfio/parameters/
enable_unsafe_noiommu_mode
[edit]
[edit]
vyos@VyOS-on-AWS# set vpp settings interface eth1 dpdk-options promisc
[edit]
vyos@VyOS-on-AWS# set vpp settings interface eth1 driver 'dpdk'
[edit]
vyos@VyOS-on-AWS# set vpp settings lcp netlink batch-delay-ms '10'
[edit]
vyos@VyOS-on-AWS# set vpp settings lcp netlink batch-size '1024'
[edit]
vyos@VyOS-on-AWS# set vpp settings lcp netlink rx-buffer-size '8388608'
[edit]
vyos@VyOS-on-AWS# set vpp settings memory main-heap-page-size '2M'
[edit]
vyos@VyOS-on-AWS# set vpp settings memory main-heap-size '3G'
[edit]
vyos@VyOS-on-AWS# set vpp settings statseg size '128M'
[edit]
vyos@VyOS-on-AWS# set vpp settings unix poll-sleep-usec '222'
[edit]
vyos@VyOS-on-AWS# commit
[edit]
vyos@VyOS-on-AWS# ru sh vpp interfaces
Kernel    Dataplane    Type    IP Address    MAC                  MTU  State
--------  -----------  ------  ------------  -----------------  -----  -------
          eth1         dpdk                  06:40:b2:b3:dd:33   1500  up
          local0       local                 00:00:00:00:00:00      0  down
eth1      tap4096      virtio                02:fe:95:86:ab:62   9000  up
[edit]
vyos@VyOS-on-AWS#

Details

Version
1.5
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Event Timeline

Unknown Object (User) triaged this task as High priority.Jul 25 2025, 1:38 PM

Will be fixed in the next rolling release.
Nice to re-check after 21 August

Tested in VyOS 2025.08.25-0020-rolling and works as expected.
Below are the test results:

vyos@VyOS-on-AWS#
[edit]
vyos@VyOS-on-AWS# run sh version
Version:          VyOS 2025.08.25-0020-rolling
Release train:    current
Release flavor:   generic

Built by:         autobuild@vyos.net
Built on:         Mon 25 Aug 2025 00:20 UTC
Build UUID:       ede69173-0b54-4a1a-ad2c-66d600e7086f
Build commit ID:  9d533529aacd33

Architecture:     x86_64
Boot via:         installed image
System type:      KVM guest
Secure Boot:      n/a (BIOS)

Hardware vendor:  Amazon EC2
Hardware model:   c5n.2xlarge
Hardware S/N:     ec27df9e-6488-3cb3-31ac-638ab419b785
Hardware UUID:    ec27df9e-6488-3cb3-31ac-638ab419b785

Copyright:        VyOS maintainers and contributors
[edit]
vyos@VyOS-on-AWS#
[edit]
vyos@VyOS-on-AWS#
[edit]
vyos@VyOS-on-AWS# set vpp settings interface eth1 dpdk-options promisc
set vpp settings interface eth1 driver 'dpdk'
set vpp settings lcp netlink batch-delay-ms '10'
set vpp settings lcp netlink batch-size '1024'
set vpp settings lcp netlink rx-buffer-size '8388608'
set vpp settings memory main-heap-page-size '2M'
set vpp settings memory main-heap-size '3G'
set vpp settings statseg size '128M'
set vpp settings unix poll-sleep-usec '222'
[edit]
vyos@VyOS-on-AWS# commit
[edit]
vyos@VyOS-on-AWS#
[edit]
vyos@VyOS-on-AWS# run sh vpp interfaces
Kernel    Dataplane    Type    IP Address    MAC                  MTU  State
--------  -----------  ------  ------------  -----------------  -----  -------
          eth1         dpdk                  06:98:c8:62:32:8b   1500  up
          local0       local                 00:00:00:00:00:00      0  down
eth1      tap4096      virtio                02:fe:db:6b:8a:10   9000  up
[edit]
vyos@VyOS-on-AWS#
[edit]
Viacheslav moved this task from Need Triage to Completed on the VyOS Rolling board.
Viacheslav moved this task from Open to Finished on the VyOS 1.5 Circinus (1.5-stream-2025-Q3) board.