Page MenuHomeVyOS Platform

vyos-hostsd reloads pdns-recursor frequently, even when no config change
Open, LowPublicBUG

Description

When debugging a pdns-recursor issue, I noticed that vyos-hostsd was frequently reloading it every time the DHCP lease renews on the WAN interface.

From the logs on hostsd we can see it is responding to a set of nameservers and search domains being removed and then replaced with identical ones. It then re-renders multiple files and calls pdns-recursor to reload its config.

I do not have the router configured to use the nameservers/search domains from DHCP, so none of this actually ends up being rendered into any files. The rendered files before and after the apply operation are the same, as is the vyos-hostsd state file.

Jul 29 05:28:49 vyos-hostsd[1111]: Request data: {"type": "search_domains", "op": "delete", "data": ["dhcp-eth0"]}
Jul 29 05:28:49 vyos-hostsd[1111]: Saving state to /run/vyos-hostsd/vyos-hostsd.state
Jul 29 05:28:49 vyos-hostsd[1111]: Sent response: {'data': None}
Jul 29 05:28:49 vyos-hostsd[1111]: Request data: {"type": "search_domains", "op": "add", "data": {"dhcp-eth0": ["attlocal.net"]}}
Jul 29 05:28:49 vyos-hostsd[1111]: Saving state to /run/vyos-hostsd/vyos-hostsd.state
Jul 29 05:28:49 vyos-hostsd[1111]: Sent response: {'data': None}
Jul 29 05:28:49 vyos-hostsd[1111]: Request data: {"type": "name_servers", "op": "delete", "data": ["dhcp-eth0"]}
Jul 29 05:28:49 vyos-hostsd[1111]: Saving state to /run/vyos-hostsd/vyos-hostsd.state
Jul 29 05:28:49 vyos-hostsd[1111]: Sent response: {'data': None}
Jul 29 05:28:49 vyos-hostsd[1111]: Request data: {"type": "name_servers", "op": "add", "data": {"dhcp-eth0": ["192.168.1.254"]}}
Jul 29 05:28:49 vyos-hostsd[1111]: Saving state to /run/vyos-hostsd/vyos-hostsd.state
Jul 29 05:28:49 vyos-hostsd[1111]: Sent response: {'data': None}
Jul 29 05:28:49 vyos-hostsd[1111]: Request data: {"op": "apply"}
Jul 29 05:28:49 vyos-hostsd[1111]: Applying 4 changes
Jul 29 05:28:49 vyos-hostsd[1111]: Writing /etc/resolv.conf
Jul 29 05:28:49 vyos-hostsd[1111]: Writing /etc/hosts
Jul 29 05:28:49 vyos-hostsd[1111]: Writing /run/powerdns/recursor.vyos-hostsd.conf.lua
Jul 29 05:28:49 vyos-hostsd[1111]: Writing /run/powerdns/recursor.forward-zones.conf
Jul 29 05:28:50 vyos-hostsd[1111]: Running "rec_control reload-lua-config"
Jul 29 05:28:50 vyos-hostsd[1111]: Running "rec_control reload-zones"
Jul 29 05:28:50 vyos-hostsd[1111]: Success
Jul 29 05:28:50 vyos-hostsd[1111]: Saving state to /run/vyos-hostsd/vyos-hostsd.state
Jul 29 05:28:50 vyos-hostsd[1111]: Sent response: {'data': {'message': 'Applied 4 changes'}}

The impact of this whole operation is just some CPU usage and log spam, both from hostsd and recursor. Ideally we should track when the before/after state is identical and skip re-render/reload in those cases.

Details

Version
dates all the way back to 1.3.5
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Performance optimization

Event Timeline

lucasec created this object in space S1 VyOS Public.
lucasec updated the task description. (Show Details)
lucasec changed Version from - to dates all the way back to 1.3.5.
syncer edited projects, added VyOS Rolling; removed VyOS 1.5 Circinus.
syncer changed the subtype of this task from "Task" to "Bug".
syncer moved this task from Need Triage to Backlog - Bug on the VyOS Rolling board.

Same bug, with a second cost beyond CPU/log noise, plus a tested workaround.

Starlink CGNAT renews between 300 and 600 seconds
PPPOE VLAN with a 2nd VLAN for IPTV DHCP renews ever 180 seconds

The reload is not just wasted work: reload-zones purges the recursor's entire cache. pdns-recursor logs, on every hostsd apply:

msg="Received rec_control command via control socket" command="reload-zones"
msg="Reloading zones, purging data from cache"

So any short-lease DHCP interface (even one whose DNS the router doesn't use) keeps the cache permanently cold.

Environment: VyOS 2025.06.06 rolling, two routers, both unmodified stock /etc/dhcp/dhclient-enter-hooks.d/04-vyos-resolvconf. Neither uses service dns forwarding dhcp <iface>; vyos-hostsd-client --get-name-server-tags-recursor returns only ['static'], so the DHCP-tagged servers are stored but never read.

Trigger (log line from the hook on a RENEW):

reason='RENEW' interface='eth0.36'
new_domain_name_servers='1.1.1.1 8.8.8.8'
old_domain_name_servers='1.1.1.1 8.8.8.8'

The hook checks only [ -n "$new_domain_name_servers" ], never whether it differs from old_*, so it delete+adds the same servers and runs vyos-hostsd-client --apply ("Applying 2 changes"), which runs reload-lua-config + reload-zones.

Measured:

  • Router A: two DHCP WANs, 300 s and 600 s leases, interleaved -> a purge every ~85 s (42/h), over 111 min of uptime.
  • Router B: one DHCP interface (IPTV VLAN), 180 s lease -> 45 purges/h, one per renew, hourly for the whole 15-day recursor uptime.
  • Router B, share of client questions answered without an upstream query (1 - delta all-outqueries / delta questions; not cache-hits/cache-misses, because the packet cache answers repeats without touching cache-hits): 22 % lifetime under churn -> 43 % over 5 min (144 questions) after the workaround. Router B's resolver is lightly used, so the gain is modest there. I did not record this metric on router A before patching; I only have its record-cache hits/misses (138/6353), which understates the packet cache.
  • Not claimed: a purge makes a lookup a cache miss, not a failure. We checked and this was not the cause of unrelated client DNS timeouts. The costs are latency on every lookup and no cache to ride through a short WAN blackout.

Workaround (running on both routers for weeks, no side effects seen). Guard the two branches so they fire only on a real change:

diff
-        if [ -n "$new_domain_name" ]; then
+        if [ -n "$new_domain_name" ] && \
+           [ "$new_domain_name" != "$old_domain_name" ]; then
...
-        if [ -n "$new_domain_name_servers" ]; then
+        if [ -n "$new_domain_name_servers" ] && \
+           [ "$new_domain_name_servers" != "$old_domain_name_servers" ]; then

A steady-state RENEW then sets no hostsd_changes and logs "No changes to apply via vyos-hostsd-client". After the patch: 0 purges and 0 hostsd applies over 2 h while the interface kept renewing (45/h). First BOUND (old empty) and a changed server list still apply.

Caveat for an upstream version of this fix. Comparing against the previous lease is not safe for users with service dns forwarding dhcp <iface>, where the recursor does read the dhcp-* tags. If vyos-hostsd loses its state while the lease values stay the same (for example hostsd restarts during a commit), the DHCP servers would not be re-pushed until the next real change. A more robust version would compare against what hostsd currently holds for the dhcp-<iface> tag, or do the no-op detection inside vyos-hostsd (the original suggestion in this task: skip re-render and reload when the state is unchanged). That would also cover the other hook callers (PPPoE ip-up, dhcpv6). We don't use that option, so I have not tested it.

Not verified: behaviour with forwarding dhcp <iface> set; hostsd restart/state-loss scenarios; the PPPoE and DHCPv6 callers of --apply; anything newer than 2025.06.06.