As of this commit, every port is checked with an exec(ipset -T):

https://github.com/vyos/vyatta-cfg-firewall/commit/835304e5aaa252e8b0bcf4651629cd089e670147

If there is a large port range, this can take many many minutes. A simple example would be

conf
set firewall group port-group slowwwwww port '20000-65531'
commit

That would run ipset -T 45,531 times.

A better idea is to get the ipset result BEFORE the check, and then iterate over the result to see if anything is missing.