Would love to see this resolved — a large (but reasonable) configuration doing IRR-based filtering from BGP peers took 9 hours to boot up.

I tried to add the prototype policy with XML.
https://github.com/sever-sever/vyos-1x/blob/policy-prefix/interface-definitions/policy-prefix-list.xml.in
Speed of validators "<validator name="ip-prefix"/>" very slow.

It's due to Python startup time. Validation of every config node requires a full startup of Python. It would be better to if the validators were shell scripts or C programs, as they were before, the load time is very small :) I think this is also a part of the significant difference in boot and commit times between 1.2 and 1.3.

Or if python were invoked only once to validate all the config nodes.

$ time sudo /usr/libexec/vyos/validate-value.py --exec /usr/libexec/vyos/validators/ip-prefix --value '192.0.2.1/24'

real	0m0.133s
user	0m0.097s
sys	0m0.037s

$ time sudo sh -c ' for ((n=0;n<1000;n++)); do /usr/libexec/vyos/validate-value.py --exec /usr/libexec/vyos/validators/ip-prefix --value "192.0.2.1/24"; done'

real	1m58.335s
user	1m34.704s
sys	0m24.341s

That's 118s for 1000 config lines (590s for 5000 lines) just for the validator, not counting other execution time. Also note this is a relatively powerful 2.5Ghz dual-core Pentium E5300.

$ cat /usr/libexec/vyos/validators/ip-prefix
#!/bin/sh

ipaddrcheck --is-any-net $1
$ time ipaddrcheck --is-any-net '192.0.2.1/24'

real	0m0.003s
user	0m0.003s
sys	0m0.000s
$ time sudo sh -c ' for ((n=0;n<1000;n++)); do ipaddrcheck --is-any-net "192.0.2.1/24"; done'

real	0m1.985s
user	0m1.235s
sys	0m0.868s

Only 1.9s for 1000 invocations of the same validator directly, without going through the Python invocation.

I think this is a powerful enough argument to rewrite the validators to shell or C (ipaddrcheck is already C). validate-value.py doesn't do anything other than invoke the destination validator or do a simple regex, which can be a sh or C validator too.

Using shell scripts would be a step back. The biggest part of the python script is the parsing of the vyos code. That's why I am suggesting that vyos command should be sent to a python daemon.
We can connect to it using IPC, and this can be done via a small C wrapper (or even directly in the caller's code to not even fork).

vyos@vyos# time bash -c ""

real	0m0.001s
user	0m0.001s
sys	0m0.000s
[edit]
vyos@vyos# time python3 -c ""

real	0m0.019s
user	0m0.011s
sys	0m0.007s
[edit]
vyos@vyos# time python3 -c "import vyos.ifconfig"

real	0m0.173s
user	0m0.104s
sys	0m0.031s

For example using the server at: https://pymotw.com/3/socket/uds.html, forking shell, nc, head and using echo:

vyos@vyos:~$ time bash -c 'echo "test" | nc -U uds_socket | head -1'
test

real	0m0.004s
user	0m0.003s
sys	0m0.000s

So it is possible to have all the code in a high language and still having good performance that way.

@thomas-mangin if you can demonstrate your idea implementing the actual validator code would be close performance-wise to the old shell/C validators, I would be on board with it. But personally I like shell and C better :)

Edit: since the old config system is C and calls an executable for each validated value, that executable would need to have a very fast load time.

@jjakob I was going to offer to do so this week if people agreed to give me a chance to show good it could be.

Personally I don't understand the insistence on keeping everything Python - in some cases, it is simply much better to use a more suitable language. A simple C program is IMO simpler and more KISS than a python daemon with a listening socket with clients connecting to it via UDP. You're invoking several layers of the OS (networking, process management, sockets) that are completely unnecessary in a simple "is this value OK?" program. I don't like convoluted solutions to solve a simple problem.

https://github.com/vyos/vyos-1x/pull/397

Also related to T2088

XML for policy access|access6|prefix|prefix6|community list

PR https://github.com/vyos/vyos-1x/pull/733

vyos@r2-roll# set npolicy 
Possible completions:
+> access-list  IP access-list filter
+> access-list6 IPv6 access-list filter
+> as-path-list Border Gateway Protocol (BGP) autonomous system path filter
+> community-list
                Border Gateway Protocol (BGP) autonomous system path filter
+> extcommunity-list
                Border Gateway Protocol (BGP) extended community-list filter
+> large-community-list
                Border Gateway Protocol (BGP) large-community-list filter
+> prefix-list  IP prefix-list filter
+> prefix-list6 IPv6 prefix-list filter
+> route-map    IP route-map