The "deny all" option that is always set currently in the chrony config overrides any allow option set. This forces chrony to never listen to NTP requests.
This can be see by setting up the NTP server, commiting, and running ss -antpul | grep 123. I see nothing listing to port 123.
It looks like "allow all" or "deny all" options trounce any other allow or deny options: https://chrony.tuxfamily.org/doc/3.4/chrony.conf.html#_ntp_server
I was able to fix this temporarily by removing the "deny all" option in the generated config and restarting chrony. After that clients could use the vyos router for NTP.
I don't think "deny all" is needed at all, but I already have a PR to put the "deny all" in an else condition so it will only appear if the config if no allowed IPs are added.