Page MenuHomeVyOS Platform
Create Task
Maniphest T7117

VPP sometimes a recursive error may occur
Closed, ResolvedPublicBUG
Actions

Description

VPP sometimes a recursive error may occur
To reproduce, set this steps several times (I tried 5 times)

set vpp settings interface eth1 driver 'dpdk'

set vpp interfaces gre gre2 remote '192.0.2.25'
set vpp interfaces gre gre2 source-address '192.0.2.1'
set vpp interfaces gre gre2 kernel-interface 'vpptun12'
set vpp kernel-interfaces vpptun12 address '10.0.0.0/31'
commit && ip link show

delete vpp
commit && ip link show

The journalctl log attached

vpp-rescursion-journalctl.tar.gz28 KBDownload

In this step commit hung

vyos@r16# set vpp interfaces gre gre2 remote '192.0.2.25'
[edit]
vyos@r16# set vpp interfaces gre gre2 source-address '192.0.2.1'
[edit]
vyos@r16# set vpp interfaces gre gre2 kernel-interface 'vpptun12'
[edit]
vyos@r16# set vpp kernel-interfaces vpptun12 address '10.0.0.0/31'
[edit]
vyos@r16# commit && ip link show

process:

vyos@r16:~$ ps ax | grep commit
   4471 pts/1    S+     0:00 /opt/vyatta/sbin/my_commit
   4696 pts/0    S+     0:00 grep commit
vyos@r16:~$

Note: I increased priority 305 => 308 for the vpp_kernel-interface while testing.

vyos@r16:~$ /usr/libexec/vyos/priority.py | grep vpp
       295  vpp.py                               ['vpp']
       304  vpp_interfaces_bonding.py            ['vpp', 'interfaces', 'bonding']
       305  vpp_interfaces_gre.py                ['vpp', 'interfaces', 'gre']
       305  vpp_interfaces_ipip.py               ['vpp', 'interfaces', 'ipip']
       305  vpp_interfaces_loopback.py           ['vpp', 'interfaces', 'loopback']
       305  vpp_interfaces_vxlan.py              ['vpp', 'interfaces', 'vxlan']
       305  vpp_interfaces_xconnect.py           ['vpp', 'interfaces', 'xconnect']
       307  vpp_interfaces_bridge.py             ['vpp', 'interfaces', 'bridge']
       308  vpp_kernel-interfaces.py             ['vpp', 'kernel-interfaces']
vyos@r16:~$

Reproduced without any changes on VyOS 1.5-rolling-202501311537

Details

Version
VyOS 1.5-rolling-202501310006
Is it a breaking change?
Perfectly compatible
Issue type
Bug (incorrect behavior)

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenBUGNoneT7070 VPP related bugs the root task
 ResolvedBUGnatali-rs1985T7117 VPP sometimes a recursive error may occur

Event Timeline

Viacheslav created this task.Jan 31 2025, 3:28 PM
Viacheslav triaged this task as Normal priority.
Viacheslav updated the task description. (Show Details)Jan 31 2025, 3:35 PM
Viacheslav updated the task description. (Show Details)Jan 31 2025, 3:58 PM
Viacheslav updated the task description. (Show Details)
Viacheslav updated the task description. (Show Details)Jan 31 2025, 4:34 PM
Viacheslav added a comment.Feb 1 2025, 11:13 AM

Also recursion sometimes occurs under bonding interfaces, or just sequences/count of the tests

bonding.png (911×2 px, 305 KB)

Viacheslav added a comment.Feb 27 2025, 5:35 PM

Reproduced again if start smoketests 17 times on VyOS 1.5-rolling-202502190007

Feb 27 16:50:13 r14 vyos-configd[794]: Received message: {"type": "node", "last": true, "data": "/usr/libexec/vyos/conf_mode/vpp.py"}
Feb 27 16:50:13 r14 vyos-configd[794]: "cpu main-core" is required but not set!
Feb 27 16:50:13 r14 vyos-configd[794]: Sending reply: ERROR_COMMIT with output
Feb 27 16:50:13 r14 vyos-configd[794]: scripts_called: ['vpp']
Feb 27 16:50:13 r14 systemd[1]: opt-vyatta-config-tmp-new_config_133953.mount: Deactivated successfully.
Feb 27 16:50:14 r14 vyos-configd[794]: Received message: {"type": "init"}
Feb 27 16:50:14 r14 vyos-configd[794]: config session pid is 133953
Feb 27 16:50:14 r14 vyos-configd[794]: config session sudo_user is vyos
Feb 27 16:50:14 r14 vyos-configd[794]: commit_scripts: ['vpp']
Feb 27 16:50:14 r14 vyos-configd[794]: Received message: {"type": "node", "last": true, "data": "/usr/libexec/vyos/conf_mode/vpp.py"}
Feb 27 16:50:14 r14 vyos-configd[794]: "cpu corelist-workers" and "cpu workers" cannot be used at the same
Feb 27 16:50:14 r14 vyos-configd[794]: time!
Feb 27 16:50:14 r14 vyos-configd[794]: Sending reply: ERROR_COMMIT with output
Feb 27 16:50:14 r14 vyos-configd[794]: scripts_called: ['vpp']
Feb 27 16:50:14 r14 systemd[1]: opt-vyatta-config-tmp-new_config_133953.mount: Deactivated successfully.
Feb 27 16:50:15 r14 vyos-configd[794]: Received message: {"type": "init"}
Feb 27 16:50:15 r14 vyos-configd[794]: config session pid is 133953
Feb 27 16:50:15 r14 vyos-configd[794]: config session sudo_user is vyos
Feb 27 16:50:15 r14 vyos-configd[794]: commit_scripts: ['vpp']
Feb 27 16:50:15 r14 vyos-configd[794]: Received message: {"type": "node", "last": true, "data": "/usr/libexec/vyos/conf_mode/vpp.py"}
Feb 27 16:50:15 r14 vyos-configd[794]: "cpu corelist-workers" is not correct
Feb 27 16:50:15 r14 vyos-configd[794]: Sending reply: ERROR_COMMIT with output
Feb 27 16:50:15 r14 vyos-configd[794]: scripts_called: ['vpp']
Feb 27 16:50:15 r14 systemd[1]: opt-vyatta-config-tmp-new_config_133953.mount: Deactivated successfully.
Feb 27 16:50:16 r14 vyos-configd[794]: Received message: {"type": "init"}
Feb 27 16:50:17 r14 vyos-configd[794]: config session pid is 133953
Feb 27 16:50:17 r14 vyos-configd[794]: config session sudo_user is vyos
Feb 27 16:50:17 r14 vyos-configd[794]: commit_scripts: ['vpp']
Feb 27 16:50:17 r14 vyos-configd[794]: Received message: {"type": "node", "last": true, "data": "/usr/libexec/vyos/conf_mode/vpp.py"}
Feb 27 16:50:17 r14 systemd[1]: Reloading.
Feb 27 16:50:17 r14 systemd[1]: Starting vpp.service - vector packet processing engine...
Feb 27 16:50:17 r14 systemd[1]: Started vpp.service - vector packet processing engine.
Feb 27 16:50:20 r14 vyos-configd[794]: ERROR:root:select failed
Feb 27 16:50:20 r14 vyos-configd[794]: CRITICAL:VyOS StdErr:ERROR:root:select failed
Feb 27 16:50:20 r14 vyos-configd[794]: CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:ERROR:root:select failed
Feb 27 16:50:20 r14 vyos-configd[794]: CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:ERROR:root:select failed
Feb 27 16:50:20 r14 vyos-configd[794]: CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:ERROR:root:select failed
Feb 27 16:50:20 r14 vyos-configd[794]: CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:ERROR:root:select failed
Feb 27 16:50:20 r14 vyos-configd[794]: CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:ERROR:root:select failed
Feb 27 16:50:20 r14 vyos-configd[794]: CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:ERROR:root:select failed
Feb 27 16:50:20 r14 vyos-configd[794]: CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:ERROR:root:select failed
Feb 27 16:50:20 r14 vyos-configd[794]: CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:ERROR:root:select failed
Feb 27 16:50:20 r14 vyos-configd[794]: CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:ERROR:root:select failed
Feb 27 16:50:20 r14 vyos-configd[794]: CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:ERROR:root:select failed
Feb 27 16:50:20 r14 vyos-configd[794]: CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:ERROR:root:select failed
Feb 27 16:50:20 r14 python3[794]: ERROR:root:select failed
Feb 27 16:50:20 r14 vyos-configd[794]: CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:ERROR:root:select failed
Feb 27 16:50:20 r14 vyos-configd[794]: CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:ERROR:root:select failed
Feb 27 16:50:20 r14 vyos-configd[794]: CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:ERROR:root:select failed
Feb 27 16:50:20 r14 vyos-configd[794]: CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:ERROR:root:select failed
Feb 27 16:50:20 r14 vyos-configd[794]: CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:ERROR:root:select failed
Feb 27 16:50:20 r14 python3[794]: CRITICAL:VyOS StdErr:ERROR:root:select failed
Feb 27 16:50:20 r14 vyos-configd[794]: CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:ERROR:root:select failed
Feb 27 16:50:20 r14 vyos-configd[794]: CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:ERROR:root:select failed
Feb 27 16:50:20 r14 vyos-configd[794]: CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:ERROR:root:select failed
Feb 27 16:50:20 r14 vyos-configd[794]: CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:ERROR:root:select failed
Feb 27 16:50:20 r14 python3[794]: CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:ERROR:root:select failed

rec.png (949×2 px, 355 KB)

Restricted Repository Identity mentioned this in rVYOSONEX3703c5452020: smoketest: temporary skip bonding tests.Aug 12 2025, 2:31 PM
Viacheslav added a comment.EditedSep 29 2025, 3:37 PM

In most cases, it could be reproduced by setting num-rx-queues :

vyos@r14# set vpp settings interface eth1 driver xdp 
[edit]
vyos@r14# commit
[edit]
vyos@r14# set vpp settings interface eth1 xdp-options num-rx-queues 
Possible completions:
   <1-65535>            Number of receive queues to connect to
   all                  All (default)
                        

      
[edit]
vyos@r14# set vpp settings interface eth1 xdp-options num-rx-queues 3
[edit]
vyos@r14# commit

logs:

OS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:CRITICAL:VyOS StdErr:Traceback (most recent call last):
Sep 29 18:33:44 r14 systemd[1]: vyos-configd.service: Main process exited, code=exited, status=1/FAILURE
Sep 29 18:33:44 r14 systemd[1]: vyos-configd.service: Failed with result 'exit-code'.
Sep 29 18:33:44 r14 systemd[1]: vyos-configd.service: Consumed 5.760s CPU time.
Sep 29 18:33:44 r14 systemd[1]: vyos-configd.service: Scheduled restart job, restart counter is at 1.
Sep 29 18:33:44 r14 systemd[1]: Stopped VyOS configuration daemon.
Sep 29 18:33:44 r14 systemd[1]: vyos-configd.service: Consumed 5.760s CPU time.
Sep 29 18:33:44 r14 systemd[1]: Started VyOS configuration daemon.
Sep 29 18:33:55 r14 zebra[1843]: [WPPMZ-G9797] if_zebra_speed_update: eth1 old speed: 10000 new speed: 0
Sep 29 18:34:34 r14 systemd[1]: opt-vyatta-config-tmp-new_config_4856.mount: Deactivated successfully.
Sep 29 18:34:34 r14 login[3387]: pam_unix(login:session): session opened for user vyos(uid=1003) by LOGIN(uid=0)
Sep 29 18:34:34 r14 systemd-logind[931]: New session 4 of user vyos.
Sep 29 18:34:34 r14 systemd[1]: Started Session 4 of User vyos.

stdout:

vyos@r14:~$ cat /tmp/vyos-configd-script-stdout

WARNING: Interface "eth1.23" does not exist!


WARNING: No certificate specified, using build-in self-signed
certificates. Do not use them in a production environment!

vyos@r14:~$
Viacheslav added a comment.Sep 29 2025, 6:21 PM

Note, replacing

# def enable(log=True):
def enable(log=False):

https://github.com/vyos/vyos-1x/blob/5248f1cebd4ac9cd8bebb62aa08ecc042a082f30/python/vyos/airbag.py#L26
Seems fixes the issue

jestabro mentioned this in T7903: Set airbag enable to default log=False.Oct 3 2025, 12:32 AM
Viacheslav mentioned this in rVYOSONEX81c2ee4b6510: T7117: VPP remove skipping for the bonding smoketest.Oct 10 2025, 7:09 PM
Restricted Repository Identity mentioned this in rVYOSONEXb28f8e73da45: Merge pull request #4787 from sever-sever/T7117-smoketest-bond.Oct 10 2025, 7:09 PM
Viacheslav added a comment.Oct 13 2025, 8:25 AM

PR https://github.com/vyos/vyos-1x/pull/4787

natali-rs1985 closed this task as Resolved.Oct 23 2025, 10:47 AM
natali-rs1985 claimed this task.
dmbaturin edited projects, added VyOS 1.5 Circinus (2025.11); removed VyOS Rolling.Wed, Nov 12, 5:08 PM
dmbaturin changed Is it a breaking change? from Unspecified (possibly destroys the router) to Perfectly compatible.