I run a VyOS 2026.09.11-1436-rolling router (vyos-1x f226af1e4) as the WAN edge for my home network, virtualized on Proxmox. I configure it through the REST API with an automation tool, and a local service on the same router polls /retrieve every few seconds. vyos-http-api dies with signal 11 several times a day, and every deploy that overlaps a poll is at risk. systemd restarts the service within about a second; meanwhile nginx returns 502 to whichever client was mid-request, so deploys fail and the poller skips its cycle.

Every crash is inside libvyosconfig. Kernel lines from one day, symbolized against the installed /usr/lib/libvyosconfig.so.0:

Sep 12 15:47:54 kernel: traps: vyos-http-api-s[92589] general protection fault ip:7f756c356aba sp:7f756cd649d8 error:0 in libvyosconfig.so.0[186aba,7f756c308000+11d000]
Sep 12 15:47:54 systemd[1]: vyos-http-api.service: Main process exited, code=killed, status=11/SEGV
Sep 12 15:48:20 kernel: traps: vyos-http-api-s[92818] general protection fault ip:7f62aaffcaba sp:7ffeaace8468 error:0 in libvyosconfig.so.0[186aba,7f62aafae000+11d000]
Sep 12 13:23:27 kernel: vyos-http-api-s[76675]: segfault at 1400 ip 00007fad35272eb1 sp 00007fad2fffd950 error 4 in libvyosconfig.so.0[1e2eb1,7fad351c8000+11d000]

Root cause

_configure_op hands the whole configure workflow to run_in_threadpool(_execute_configure_op, ...). That workflow builds Config(session_env=env), and ConfigSourceSession parses the running and session configs with ConfigTree(...), which calls from_string in libvyosconfig. retrieve_op is an async def served on the event loop; it takes no lock and also builds ConfigTree(res). ctypes releases the GIL for the duration of each foreign call, so a /retrieve that arrives while a configure worker is in Config() puts two threads inside the OCaml library at the same time.

libvyosconfig is not safe for concurrent entry. I loaded the exact libvyosconfig.so.0 from the router through ctypes, the way python/vyos/configtree.py loads it, and tested outside VyOS:

The error buffer (error_message in libvyosconfig/lib/bindings.ml) and the parser state are shared across threads, which matches the crash landing in Vyos1x_parser.error twice in two separate processes.

History

The same crash has been fixed twice for different request pairs:

• T5006: concurrent /retrieve requests segfaulted; fixed by making retrieve_op async so retrieves serialize on the event loop.
• T6069: concurrent /configure requests segfaulted; fixed in rVYOSONEX7503e419d0db ("fix allocation outside of thread lock") by building Config() after lock.acquire(). That lock is held only by configure, so it serializes configure against configure, not against /retrieve.
• T7588 (rVYOSONEXbfb2e8595140): kept _configure_op on the event loop and moved only run_commit (a subprocess, no library calls) to the threadpool, so library calls again stayed on one thread.
• T7090 (rVYOSONEX7fd4b50494d3, background configure operations): turned _configure_op into the synchronous _execute_configure_op and runs all of it in the threadpool, including Config(). From this change on, configure and retrieve enter the library from different threads.

T9015 (#5294) moved the file I/O of read_internal/write_internal into Python, which addresses thread registration for those entry points. The build I run contains it, and the crash still occurs.

Reproduction

The attached reproduce.py

Run python3 reproduce.py on a VyOS instance. With the stock python/vyos/configtree.py (exit code 1):

child process killed by SIGSEGV

With the patched configtree.py described below (exit code 0):

107140 ConfigTree operations from 4 threads in 20s
0 errors

I ran both on the 2026.09.11-1436-rolling router (4 vCPU), importing the stock module from a copy placed ahead of the installed one on PYTHONPATH for the first run.

To reproduce through the API instead: loop 3 clients on POST /retrieve (showConfig, path ["interfaces"]) and 1 client on POST /configure setting system host-name to its current value. On my router, 90 seconds of that killed the service 8 times and returned 502 for 88 of 106 requests.

Validation

I replaced /usr/lib/python3/dist-packages/vyos/configtree.py on the router with the patched file and restarted vyos-http-api, then repeated the 90-second API load:

With only the 3 readers and no writer, the router completed 10 /retrieve requests in 30 seconds with the stock module and 11 with the patched one.

One /retrieve returned 500 during the patched run. While a commit rewrites the shared session, session.show_config() can return _show_diff error (both config NULL) instead of config text, and ConfigTree(res) raised ValueError: Failed to parse config: Syntax error .... The same _show_diff error appears on the stock module; I have not looked into it further and it is not addressed here.

Fix

Serialize every libvyosconfig entry point on one process-wide threading.RLock in the _Lib wrapper in python/vyos/configtree.py. ConfigTree, ReferenceTree, DiffTree and the module-level helpers all call the library through that wrapper, so the lock covers the HTTP API, vyos-configd, and __del__ running on any thread, without changes to call sites. It is an RLock because a call made while holding it can trigger garbage collection, and ConfigTree.__del__ then calls destroy on the same thread. The lock is held for one library call (microseconds to about a millisecond), not for a request.

The wrapper also keeps error text per thread. For the entry points that reset and set error_message, it reads the buffer right after the call, still under the lock, and stores it in a threading.local; get_error() returns the calling thread's value. destroy leaves that value alone, since __del__ can run it between a failed call and the caller's get_error(). Any other call clears it, and get_error() then reads the buffer directly as before.

Unit tests

Three tests added to src/tests/test_config_tree.py: concurrent parse from four threads, per-thread error text while other threads parse, and error text surviving a destroy between a failed call and get_error(). Details in the PR.

Related

• T5006: Http api segfault with concurrent requests. Resolved (retrieve moved to the event loop).
• T6069: HTTP API segfault during concurrent configuration requests. Resolved (Config() built under the configure lock).
• T5305: REST API configure operation should not be defined as async. Resolved.
• T7588: Vyconf: Call libvyosconfig functions from main thread under http-api. Resolved.
• T7090: HTTP API upstream task timeout (504 Gateway Timeout). Introduced background configure operations and the threadpool workflow.
• T9015: Resolve thread_registration error in vyos1x-config read_internal* functions. Resolved; present in the affected build.
• T8235: VyOS HTTPS API becomes unresponsive during concurrent config-file load operations. Resolved; same area (concurrent API requests), different mechanism (deadlock in the config-file handler).

Environment

• VyOS 2026.09.11-1436-rolling (vyos-1x f226af1e4, libvyosconfig0 999.0-14869-gf226af1e4)
• REST API used by an automation tool plus a local poller of /retrieve
• KVM guest on Proxmox VE, 4 vCPU