Page MenuHomeVyOS Platform
Files F114283559

vyos-bug-report.md
All Users
Actions

Authored By
fabrizziop
May 6 2026, 11:50 AM
Size
2 KB
Referenced Files
None
Subscribers
None

vyos-bug-report.md
View Options

Title: vyos-netlinkd consuming 30-40% CPU due to O(n) psutil process scan in commit_in_progress()
## Description
After upgrading to VyOS nightly `2026.05.05-0039-rolling`, the `vyos-netlinkd` service
consumes 30-40% CPU continuously, even with no network events occurring.
## Root Cause
The `commit_in_progress()` function in `python/vyos/utils/commit.py` uses
`psutil.process_iter()` + `proc.open_files()` to detect config commits.
This iterates ALL system processes and reads `/proc/PID/fd` for each one.
This function is called for **every netlink message** in a `while` loop inside
`src/services/vyos-netlinkd`:
```python
for message in ipr.get():
while commit_in_progress():
sleep(0.250)
```
On a system with N processes, each call performs O(N) `/proc` scans.
With 139 PIDs on the affected router, this results in ~16,000 openat/readlink
syscalls per 15-second window, consuming significant CPU.
## Evidence (strace from affected router)
```
Syscall counts over 15-second strace window:
28,954 clock_gettime(CLOCK_MONOTONIC)
23,270 newfstatat
16,387 openat # /proc/PID/stat reads
16,384 close
16,335 lseek
16,334 read
13,870 getdents64 # /proc/PID/fd directory reads
9,658 newfstatat
51 recvfrom # actual netlink operations
50 readlink # /proc/PID/fd/N symlink reads
```
The process is scanning `/proc/*/stat` and `/proc/*/fd` for every PID
(kernel threads included: rcu_sched, migration/0, idle_inject/0, cpuhp/0, etc.)
in a tight loop, while only performing 51 actual netlink operations.
## Affected Versions
- VyOS 2026.05.05-0039-rolling (and likely other recent nightlies)
- Kernel: 6.6.137-vyos
## Proposed Fix
1. Replace the psutil process_iter() loop in `commit_in_progress()` with an
O(1) fcntl-based lock check, consistent with how `acquire_commit_lock_file()`
works.
2. Add a fast path: if the lock file doesn't exist, return False immediately.
3. In `vyos-netlinkd`, move the commit_in_progress() check outside the
per-message loop (check once per select() cycle).
See the attached patch: `/tmp/vyos-netlinkd-cpu-fix.patch`
## Impact
- Average case (no commit in progress): single stat() syscall instead of 100+ /proc scans
- During commit: fcntl check is ~1000x faster than psutil process iteration
- Expected CPU reduction: from ~35% to <1% for vyos-netlinkd
## Files Changed
- `python/vyos/utils/commit.py` - Replace O(n) psutil scan with O(1) fcntl check
- `src/services/vyos-netlinkd` - Move commit check outside per-message loop

File Metadata

Mime Type
text/plain
Storage Engine
blob
Storage Format
Raw Data
Storage Handle
3651102
Default Alt Text
vyos-bug-report.md (2 KB)

Event Timeline