Page MenuHomeVyOS Platform
Create Task
Maniphest T9063

VPP commit fails at lcp_nl_resync() on kernel 6.18
Open, HighPublicBUG
Actions

Description

Hardware: Intel E823-L integrated NIC.
PCI ID: 8086:124d
Kernel driver when not bound to VPP: ice
Firmware information from dmesg: fw 5.5.17 api 1.7.9 nvm 2.28

VPP configuration:

vyos@vyos# show vpp
 settings {
     allow-unsupported-nics
     interface eth101 {
     }
     resource-allocation {
         buffers {
             data-size 10128
             page-size 1G
         }
         cpu-cores 6
         ipv6 {
             hash-buckets 524288
             heap-size 512M
         }
         memory {
             main-heap-size 4G
             main-heap-page-size 1G
             stats {
                 size 1G
             }
         }
     }
 }

The same machine and the same VPP configuration work correctly on: 2026.04.30-0041-rolling, which appears to be the last rolling image using the 6.6 kernel.

After upgrading to the current rolling image (2026.06.30-0048-rolling) with the 6.18 kernel, the same configuration no longer commits:

[Errno 2] VPP API client: read failed. VPP service will be restarted with the previous configuration
[[vpp]] failed
Commit failed

The failure happens during VyOS VPP apply (/usr/libexec/vyos/conf_mode/vpp.py)

Traceback shows the failing call is vpp_control.lcp_resync(), which calls lcp_nl_resync(). Traceback excerpt:

File "/usr/libexec/vyos/conf_mode/vpp.py", line 885, in apply
    vpp_control.lcp_resync()
File "/usr/lib/python3/dist-packages/vyos/vpp/control_vpp.py", line 304, in lcp_resync
    return self.__vpp_api_client.api.lcp_nl_resync()
File "/usr/lib/python3/dist-packages/vpp_papi/vpp_papi.py", line 854, in _call_vpp
    raise VPPIOError(2, "VPP API client: read failed")

If I add retry logic around lcp_resync(), the first attempt fails with:

VPPIOError(2, 'VPP API client: read failed')

but a later attempt succeeds and commit completes.

Details

Version
2026.06.30-0048-rolling
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Event Timeline

canoziia created this task.Jul 7 2026, 5:20 PM
Viacheslav triaged this task as Normal priority.Jul 13 2026, 2:01 PM
pasik subscribed.Jul 13 2026, 2:05 PM
canoziia added a comment.Aug 24 2026, 8:45 AM

Hello, I found this is because the lcp_nl_resync operation is a potentially long-running full netlink synchronization, but the VyOS wrapper invokes it with the generic VPP PAPI response timeout of 5 seconds.

diff
     @_Decorators.check_retval
     @_Decorators.api_call
-    def lcp_resync(self) -> None:
+    def lcp_resync(self, timeout: float = LCP_RESYNC_TIMEOUT_DEFAULT) -> None:
         """Resynchronize objects between kernel and VPP via Netlink
 
         This clears all routes in VPP configured by LCP and re-creates them
         based on the current state of the kernel.
+
+        Args:
+            timeout: Maximum time in seconds to wait for the resync reply.
         """
-        return self.__vpp_api_client.api.lcp_nl_resync()
+        return self.__vpp_api_client.api.lcp_nl_resync(_timeout=timeout)

Add an 60s timeout will fix the problem. Should I submit a pull request?

Viacheslav raised the priority of this task from Normal to High.Aug 25 2026, 12:42 PM
Viacheslav subscribed.Aug 25 2026, 1:54 PM

We should try to find the root cause.

a.kudientsov subscribed.Wed, Sep 9, 5:32 AM

@canoziia Hi! Please check if the latest nightly build resolves the issue you described - https://vyos.net/get/nightly-builds/