I see infrequent SERVEFAIL responses from VyOS's dns forwarder because of authoritative DNS servers taking longer than 1500ms (the default timeout) to respond.

1500ms is a reasonable time to expect a response, but still I do see some failures, usually but not always from freedns.afraid.org. Anecdotally VyOS is the strictest forwarder I've used, I think the setting should be configurable.