Page MenuHomeVyOS Platform

smoketest: DHCP client process checks race against daemon startup/shutdown — flaky interface tests in CI
Closed, ResolvedPublic

Description

CI interface smoketests (test_interfaces_cli job of the VyOS ISO Integration Test workflow) fail intermittently on DHCP-related assertions. Pass rate over the last 30 workflow runs: 42-55%, with independent failures near-daily — the least stable job in the workflow.

Three timing hazards in smoketest/scripts/cli/base_interfaces_test.py:

  1. PID discovery window too short. process_named_running(..., timeout=10) (8 call sites: dhclient + dhcp6c + wide-DHCPv6 option tests). On loaded CI runners the client daemon can take longer than 10 s to appear after commit; the poll helper returns as soon as the process shows up, so a longer grace only affects the failure path.
  1. Unguarded /proc/<pid>/cmdline read after PID discovery (4 sites, e.g. lines 284/318/339/1202 on current rolling). dhclient re-executes when it daemonizes — the PID captured right after commit can be gone by the time /proc is read. read_file() then returns the defaultonfailure=None, and the subsequent assertIn dies with a confusing TypeError instead of a clean assertion.
  1. Instant teardown leak asserts. tearDown() runs assertFalse(process_named_running(daemon)) immediately after cli_commit() removes the config — a daemon still in its shutdown path is reported as a leak.

Fix: raise the startup-poll window to 60 s via a module constant, re-resolve the PID once when the /proc read comes back empty, and give teardown a bounded grace period (existing wait_for_result shim helper) before declaring a daemon leak.

Details

Version
rolling
Issue type
Bug (incorrect behavior)

Event Timeline

syncer triaged this task as High priority.

Fix up as draft PR: https://github.com/vyos/vyos-1x/pull/5310

All three hazards addressed in smoketest/scripts/cli/base_interfaces_test.py: shared PROCESS_WAIT_TIMEOUT = 60 at the 8 dhclient/dhcp6c discovery sites, a get_process_cmdline() helper that re-resolves the PID once when the /proc read fails (note: read_file() re-raises unless defaultonfailure is set — the naive is None check never fires), and a bounded 60 s grace in tearDown() via the existing wait_for_result() shim before declaring a daemon leak. The interface smoketest jobs on the PR itself exercise the change.

Viacheslav changed the task status from Open to In progress.Jul 6 2026, 1:08 PM
Viacheslav assigned this task to syncer.

Merged in https://github.com/vyos/vyos-1x/pull/5310 (head 11cd6c125, merged by c-po 2026-07-07). First post-merge rolling run (28872988100): test_interfaces_cli passed — against a 42–55% pass-rate baseline over the prior 30 runs. Note for the record: the same run's test_smoketest_cli failure is the unrelated TestKernelModules.test_hypervisor_hyperv flake, not a DHCP/interface assertion.