Page MenuHomeVyOS Platform

config-mgmt commit_confirm child never exits after successful confirm; spins at 100% CPU indefinitely
Open, NormalPublic

Description

Summary

After a commit-confirm was issued and then successfully confirmed, the config-mgmt commit_confirm child process did not exit. It entered a busy state and consumed 100% of one core continuously for 12 days 17 hours, until killed manually.

The configuration itself committed and saved correctly — this is a leaked process, not a failed transaction. But on passively-cooled hardware it held the CPU package at 90 °C (crit 105 °C) for the entire period, and nothing in the system surfaced it.

Version

Version:          VyOS 2026.06.22-0055-rolling
Release train:    rolling
Release flavor:   generic
Built on:         Mon 22 Jun 2026 00:55 UTC
Build commit ID:  6f3f7b7290bb7d
Architecture:     x86_64
System type:      bare metal

vyos-1x package version 999.0-14466-g6c0fd9909.

Hardware: fanless 6-port x86_64 appliance, 4 cores, used as a router.

Expected behaviour

Once a commit-confirm is confirmed, the timer child should terminate.

Actual behaviour

The child remains in state R, consuming ~100% of one core, indefinitely.

 PID  PPID     ELAPSED         TIME %CPU STAT COMMAND
4210  4209  12-17:16:42  12-17:06:15 99.9 R    /usr/bin/python3 /usr/bin/config-mgmt commit_confirm -t=5

TIME (12d 17h 06m) against ELAPSED (12d 17h 16m) shows it was on-CPU for essentially its whole lifetime — a spin, not periodic work.

Parent, sleeping normally:

 PID  PPID     ELAPSED STAT COMMAND
4209  4014  12-17:17:10 S    sudo sg vyattacfg /usr/bin/config-mgmt commit_confirm -t=5

Evidence the confirm succeeded

The child was spawned at approximately 2026-07-27 17:42, about five minutes after boot, with -t=5 (a five-minute window). Commit revisions land inside that window:

0  2026-07-27 17:47:31  by atmarcus  via cli
1  2026-07-27 17:46:44  by atmarcus  via cli

/config/config.boot was written at 2026-07-27 17:47. No rollback occurred, and the system ran the confirmed configuration continuously for the following twelve days. Had the confirm gone unanswered, the rollback would have fired long ago.

Thermal impact

Before kill:  Package id 0:  +90.0 °C   (high = +105.0, crit = +105.0)
After kill:   Package id 0:  +71.0 °C   (within 25 seconds)

Load average was exactly 1.00 / 1.00 / 1.00 throughout — one core of four. Because the router stayed fully functional and responsive, no reachability-based monitoring detected anything. The only symptom was heat.

Reproduction

Reproduced deliberately on the same system, 2026-08-09. Running commit-confirm from a non-interactive vbash session spawns the child, which immediately spins at ~99% CPU:

vbash -c "
source /opt/vyatta/etc/functions/script-template
configure
set interfaces ethernet eth3 description 'test'
commit-confirm 10
exit
"

Within seconds:

  PID  ELAPSED %CPU STAT COMMAND
19287    02:24 98.9 R    /usr/bin/python3 /usr/bin/config-mgmt commit_confirm -t=10
19286    02:24  0.0 S    sudo sg vyattacfg /usr/bin/config-mgmt commit_confirm -t=10
19188    02:24  0.0 S    vbash -c ...

In this reproduction the invoking vbash blocks indefinitely and the commit never applies — no new revision is created and the configuration is unchanged. Killing the child, its sudo parent and the vbash clears the condition; configuration, interfaces and routing are unaffected.

This differs from the original incident in one respect, and I want to be precise about it. In the original the commit did apply — revisions landed inside the window and config.boot was written — and only the child leaked. In this reproduction the commit does not apply at all. These may be two faces of the same non-blocking wait loop, or related but distinct defects. What both share is a commit_confirm child spinning at 100% CPU rather than sleeping.

A plain commit does not reproduce it, in either form. Tested twice on the same system through the identical harness — a normal set + commit + save completes correctly and leaves no lingering process. That points at the commit_confirm path specifically rather than at commit handling in general.

Other context:

  • The original commit-confirm was issued from an interactive CLI session over SSH; the change concerned interface addressing and NAT on a WAN-side interface, and the confirm was issued normally from the same session within the window.
  • The configuration is small — config.boot is 4188 bytes — so the known large-configuration slow-commit behaviour in T5388 does not apply.
  • T3556 (commit-archive causing 100% CPU on boot) is a different path; commit-archive is not configured on this system.
  • vyos-commitd and vyos-configd were active throughout, before and after.

Impact

  • One core pegged indefinitely, per occurrence.
  • On fanless hardware, sustained operation near thermal limits.
  • Entirely silent — the router routes, answers and commits normally throughout, so reachability and service monitoring do not fire.
  • Cumulative: each affected commit-confirm would leak another core. Workflows that use commit-confirm as standard practice for changes affecting remote access are the most exposed.

Workaround

kill the child process. Configuration, interfaces and routing are unaffected — verified: interface state and addressing were identical before and after, and the router did not drop a packet.

Possibly relevant

python/vyos/config_mgmt.py in vyos-1x implements this path:

https://github.com/vyos/vyos-1x/blob/current/python/vyos/config_mgmt.py

A spin rather than a sleep suggests the wait loop is not blocking, and that the confirm-received signal terminates the parent's wait without tearing down the child.

Details

Version
2026.06.22-0055-rolling
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Event Timeline

sageframe triaged this task as Normal priority.
sageframe created this object in space S1 VyOS Public.

Also reproduces on VyOS Stream 2026.03 (circinus), not only rolling.

Same hardware, migrated from 2026.06.22-0055-rolling to 2026.03, re-tested 2026-08-09:

PID  ELAPSED %CPU STAT COMMAND

3472 00:19 97.2 R /usr/bin/python3 /usr/bin/config-mgmt commit_confirm -t=10
3471 00:19 0.0 S sudo sg vyattacfg /usr/bin/config-mgmt commit_confirm -t=10
3372 00:19 0.7 S vbash -c ...

Same signature — child spinning within seconds, invoking vbash blocked, and the commit never applies. The test set a description on eth4; after killing the three processes the interface still carried its previous value, with no new commit revision.

A plain commit does not reproduce it on either train, tested repeatedly through the identical harness.

Since Stream is the quality gate for the upcoming LTS, this appears to be on the path into an LTS release. It may be worth adding the circinus tag — I don't seem to have edit permission on the task.

Root cause found: ask_yes_no() in python/vyos/utils/io.py:57 catches EOFError from input() but loops right back to the top of while True with no backoff or exit condition:

def ask_yes_no(question, default=False) -> bool:
    while True:
        try:
            ...
            c = input().lower()
            ...
        except EOFError:
            stdout.write("\nPlease respond with yes/y or no/n\n")
        except KeyboardInterrupt:
            return False
        
When stdin isn't a TTY (exactly the case for vbash -c "..." run non-interactively, as in the repro), input() raises EOFError immediately and repeatedly — there's nothing to wait on, so the loop spins as fast as the CPU allows, burning ~100% of a core forever.
        
commit_confirm() (config_mgmt.py:222) calls ask_yes_no(prompt_str, default=True) whenever no_prompt isn't set, which is exactly what the reported vbash one-liner hits. This also explains the "after successful confirm" framing: a later confirm call only stops the systemd timer and pkills the notify script — it never touches the original
commit_confirm process, which is stuck in the EOFError loop from the very first prompt and was never actually confirmed. It just keeps spinning, orphaned, while the system otherwise continues to work normally — hence unnoticed until something checks CPU/thermal load.

Suggested minimal fix — bail out to default immediately when stdin isn't a TTY, instead of ever entering the retry loop: 

def ask_yes_no(question, default=False) -> bool:
    from sys import stdin, stdout
    if not stdin.isatty():
        return default
    default_msg = "[Y/n]" if default else "[y/N]"
    while True:
        ...
    
This is behaviorally identical to a user pressing Enter (accepting the default) in an interactive session, so it changes nothing for normal interactive use. ask_yes_no() has ~40 call sites across the tree (image_installer, pki, format_disk, remote, etc.) that would all hang the same way if invoked non-interactively today — this fix resolves
the underlying issue for all of them, not just commit-confirm.

Happy to open a PR with this + a unit test (mocking stdin.isatty()) if useful, @jestabro — let me know if you'd rather take it from here since it's already assigned to you.

One thing not covered above — this isn't rolling-only. Same box migrated to Stream 2026.03 (circinus), reproduces identically:

PID  ELAPSED %CPU STAT COMMAND

3472 00:19 97.2 R /usr/bin/python3 /usr/bin/config-mgmt commit_confirm -t=10

Same signature, commit never applies. Given Stream feeds LTS it's probably worth the circinus tag alongside VyOS Rolling — I don't have edit permission on the task.

Also happy to test a patch on either train, I've got a box here that reproduces on both.

hi please try:

diff --git a/python/vyos/utils/io.py b/python/vyos/utils/io.py

index 77e30cbb0..13ef8709a 100644
--- a/python/vyos/utils/io.py
+++ b/python/vyos/utils/io.py
@@ -56,7 +56,10 @@ def ask_input(question, default='', numeric_only=False, valid_responses=[],

 def ask_yes_no(question, default=False) -> bool:
     """Ask a yes/no question via input() and return their answer."""
-    from sys import stdout
+    from sys import stdin, stdout
+
+    if not stdin.isatty():
+        return default
     default_msg = "[Y/n]" if default else "[y/N]"
     while True:
         try:
diff --git a/src/tests/test_utils_io.py b/src/tests/test_utils_io.py
new file mode 100644
index 000000000..7adb0dc61
--- /dev/null 
+++ b/src/tests/test_utils_io.py
@@ -0,0 +1,46 @@
+# Copyright VyOS maintainers and contributors <maintainers@vyos.io>
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 2 or later as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+
+import unittest
+from unittest.mock import patch
+
+from vyos.utils.io import ask_yes_no
+
+
+class TestVyOSUtilsIO(unittest.TestCase):
+    def test_ask_yes_no_non_interactive_returns_default_without_reading_stdin(self):
+        # T9185: on a non-TTY stdin (e.g. a non-interactive vbash session),
+        # input() raises EOFError immediately and forever; ask_yes_no() must
+        # return the default straight away instead of ever looping on input().
+        with patch('sys.stdin') as mock_stdin:
+            mock_stdin.isatty.return_value = False
+            with patch('builtins.input') as mock_input:
+                self.assertTrue(ask_yes_no('Proceed?', default=True))
+                self.assertFalse(ask_yes_no('Proceed?', default=False))
+                mock_input.assert_not_called()
+
+    def test_ask_yes_no_interactive_reads_input(self):
+        with patch('sys.stdin') as mock_stdin:
+            mock_stdin.isatty.return_value = True
+            with patch('builtins.input', return_value='y'):
+                self.assertTrue(ask_yes_no('Proceed?', default=False))
+            with patch('builtins.input', return_value='n'):
+                self.assertFalse(ask_yes_no('Proceed?', default=True))
+            with patch('builtins.input', return_value=''):
+                self.assertTrue(ask_yes_no('Proceed?', default=True))
+                self.assertFalse(ask_yes_no('Proceed?', default=False))
+
+
+if __name__ == '__main__':
+    unittest.main()

Dazu als Begleittext, damit klar ist, dass es schon einen PR gibt:

▎ Opened a PR with this fix + the test above: https://github.com/vyos/vyos-1x/pull/5390

Following @rherold analysis of root cause above, cf. https://github.com/vyos/vyatta-cfg/pull/153 --- so you can simply add the no-prompt option to the script in question.

Thanks, no-prompt does the job for my case, I'll use that.

One thing worth adding from the operational side. The failure mode is silent. A caller that forgets no-prompt doesn't get an error, or a hang you'd notice. It gets a process pegging a core while everything else looks completely healthy. Mine ran for 12 days 17 hours before I caught it, and only because it's a fanless box and the package temp was sitting at 90°C. The router routed fine the whole time, SSH answered, monitoring was green.

Also, correcting my own report: I said there was evidence the confirm had succeeded. Following the root cause above, that can't be right. It never got past the prompt, so nothing was committed, and those revisions were me going back and committing separately after the first attempt looked like it had hung. Which also means my original incident was a non-TTY invocation, not the interactive session I described.

Not pushing for any particular fix, just noting that "pass no-prompt" is easy to get right once you know about it, and gives you nothing if you don't.

At risk of sounding like a broken record, is it worth adding the circinus tag? It reproduces on Stream 2026.03 as well as rolling.

Thanks @sageframe for the update and the original report.

I agree that we want to eliminate the silent spinning, irrespective of the cause: that is in discussion in context of the PR here (cf. last few comments): https://github.com/vyos/vyos-1x/pull/5390
Regarding your correction, yes that was my guess; thanks for confirming.
Regarding a backport, I agree that we will likely want to backport both the no-prompt bash option and however we resolve the silent spinning; I will add an internal tag to the respective PRs/tasks.

Viacheslav subscribed.

We cannot remove images after this merge

Installed images: 2026.08.14-0025-rolling
2026.08.13-0024-rolling
Cannot prompt "Do you really want to delete the image 2026.08.13-0024-rolling?" ([Y/n] default=False): stdin is not a TTY. Pass an explicit non-interactive option (e.g. no_prompt/-y) instead of relying on ask_yes_no().
Error: Process completed with exit code 1.

Reopened