Python config node validators are slow.
$ time sudo /usr/libexec/vyos/validate-value.py --exec /usr/libexec/vyos/validators/ip-prefix --value '192.0.2.1/24' real 0m0.133s user 0m0.097s sys 0m0.037s $ time sudo sh -c ' for ((n=0;n<1000;n++)); do /usr/libexec/vyos/validate-value.py --exec /usr/libexec/vyos/validators/ip-prefix --value "192.0.2.1/24"; done' real 1m58.335s user 1m34.704s sys 0m24.341s
That's 118s for 1000 config lines (590s for 5000 lines) just for the validator, not counting other execution time. Also note this is a relatively powerful 2.5Ghz dual-core Pentium E5300.
$ cat /usr/libexec/vyos/validators/ip-prefix #!/bin/sh ipaddrcheck --is-any-net $1 $ time ipaddrcheck --is-any-net '192.0.2.1/24' real 0m0.003s user 0m0.003s sys 0m0.000s $ time sudo sh -c ' for ((n=0;n<1000;n++)); do ipaddrcheck --is-any-net "192.0.2.1/24"; done' real 0m1.985s user 0m1.235s sys 0m0.868s
Only 1.9s for 1000 invocations of the same validator directly, without going through the Python invocation.
It's due to Python startup time. Validation of every config node requires a full startup of Python. It would be better to if the validators were shell scripts or C programs, as they were before, the load time is very small. I think this is also a part of the significant difference in boot and commit times between 1.2 and 1.3.
I think this is a powerful enough argument to rewrite the validators to shell or C (ipaddrcheck is already C). validate-value.py doesn't do anything other than invoke the destination validator or do a simple regex, which can be a sh or C validator too.
This was originally discussed in T2425.
I have rewritten validate-value.py in bash, which is much faster already (12-14 times per validation in my testing):
$ time sudo bash -c ' for ((n=0;n<1000;n++)); do ./validate-value.sh --exec /usr/libexec/vyos/validators/ip-prefix --value "192.0.2.1/24"; done' real 0m9.606s user 0m5.576s sys 0m4.285s $ time sudo bash -c ' for ((n=0;n<1000;n++)); do /usr/libexec/vyos/validate-value.py --exec /usr/libexec/vyos/validators/ip-prefix --value "192.0.2.1/24"; done' real 1m58.335s user 1m34.704s sys 0m24.341s $ time sudo bash -c ' for ((n=0;n<1000;n++)); do ./validate-value.sh --regex "([0-9A-Fa-f]{1,2}[:])*([0-9A-Fa-f]{1,2})" --value "ff:ff:ff:ff:ff:ff"; done' real 0m7.455s user 0m4.264s sys 0m3.900s $ time sudo bash -c ' for ((n=0;n<1000;n++)); do /usr/libexec/vyos/validate-value.py --regex "([0-9A-Fa-f]{1,2}[:])*([0-9A-Fa-f]{1,2})" --value "ff:ff:ff:ff:ff:ff"; done' real 1m45.868s user 1m28.573s sys 0m17.378s
This seems to have reduced my boot time a bit from approx. 220-250s before to 194s after. I'm sure rewriting all the individual validators will bring similar orders of magnitude improvements.