VyOS allows multiple NTP server listen addresses to be configured (see here: https://docs.vyos.io/en/latest/configuration/service/ntp.html#cfgcmd-set-service-ntp-listen-address-address).
However the chrony NTP server only allows one bindaddress to be configured.
From man chrony.conf:

For each of the IPv4 and IPv6 protocols, only one bindaddress directive can be specified.
Therefore, it is not useful on computers which should serve NTP on multiple network interfaces.

VyOS will emit one bindaddress config option per configured listen address, of which chrony will only use the last one.

Since I was configuring NTP to listen on all networks anyway, for me the fix was to delete all listen addresses.

Version info, since this is a custom build (I include some additional packages such as the prometheus-node-exporter, but no changes to any core packages are done):

Version:          VyOS 1.4-rolling-202306161708
Release train:    current

Built by:         <me>
Built on:         Fri 16 Jun 2023 17:08 UTC
Build UUID:       b9ca9e71-6ba9-425e-8e70-ae8e61fcc67c
Build commit ID:  f034ed9750f01a-dirty