Page MenuHomeVyOS Platform

http-api: commit returns before the API server is back up after a reload - race
In progress, NormalPublicBUG

Description

The test_service_https.py smoketest fails from time to time in test_api_auth. The test commits service https api graphql and then sends a GraphQL request, and that request gets back a body that isn't JSON, so r.json() raises JSONDecodeError.

When the API server is already running, a commit doesn't restart it. It only sends SIGHUP and returns at once. The server then shuts uvicorn down, waits up to one second, re-reads its config, loads the GraphQL routes and opens /run/api.sock again. Until then, nginx probably answers with an HTML 502 page, though I haven't captured that response yet. The conf-mode script covers this gap with a fixed sleep(1), which isn't enough on slower systems such as the QEMU smoketest guest. The failure depends on timing, which is why it's intermittent.

This isn't limited to the smoketest. Any API client that sends a request right after a commit can get errors.

Proposed fix: the server writes a readiness marker once uvicorn is listening. The conf-mode script deletes the marker before the reload and waits for it to come back, with a timeout, instead of sleeping for a fixed time.

Details

Version
rolling, 1.5.1
Is it a breaking change?
Perfectly compatible
Issue type
Bug (incorrect behavior)

Event Timeline

c-po changed the task status from Open to In progress.
c-po claimed this task.
c-po triaged this task as Normal priority.