vyos@vyos:~$ wake-on-lan interface eth0.201 host a:b:c:d:e:f

Thank you for this long answer @jjakob. I want to demonstrate that a full python solution can provide the performance we need. I appreciate that changing the Vyatta code need to be done carefully with many consideration about backward compatibility. What I am doing is surely 1.4 material. However I do not believe this is as hard to achieve as everyone may think, and as working code is the best way to discuss code design, that is what I am doing.

The cli is mostly functional. I am able to validate the data as it is typed in conf mode (the CLI has both completion and validation working), as soon as delete and show (the current show is "show configuration commands") are implemented, it will be mostly usable. The code can already load a configuration from file, allow some "set" edit and then allow the use of save (the config format is a number of "set" commands, one per line), respect the initialisation order of the XML.

I used @runar code on my branch working on T2522 .. Configuring 100 dummy interface went from 42 to 8 seconds.

As the current "priority map" there aren't a loot of concurrent python blocks, but i think many of the remaining bash/perl scripts could be moved to new places. https://pastebin.com/z6ZvkJKB

I've created some proof of concept code that i think could help on this issue. https://github.com/runborg/vyos-1x/blob/main-cfg/src/conf_mode/main.py this is a conf-mode executor that handles multiple conf mode scripts. The reason i think this could seriously help on this issue is that as this is all running inside a single python tnterpreter, its able to load the config object once and pass it to all needed conf_mode scripts without a need for reinitialization.

I also couldn't immediately grasp how your suggestion would fit into the whole design concept of VyOS - now that I've had some time to think about it, it indeed makes sense. I was having problems seeing how having a complex daemon just for validators would make sense while still leaving everything else as is - it makes sense if everything is moved there. And only because of the current state of having to "marry" vyatta-cfg with Python and Python's slow initialization times. I guess in the future, if VyConf gets to a place where it can replace vyatta-cfg, it would still make sense to run everything in a single daemon.

@thomas-mangin I think there was a misunderstanding between us. The disagreement we had regarding the way to implement vyatta-cfg validators was because the validators are a integral part of vyatta-cfg operation. They are also simple and small as they only need to validate the types and constraints of config nodes. As they are tied to vyatta-cfg closely, which operates by executing a new process for each config node, that execution needs to be very fast. I was against your solution (a validator daemon in Python listening on a socket file and a companion client in a language that's faster to start up) just because it seemed needlessly complicated for what it needs to achieve. Node validation in vyatta-cfg is a case of simple constraints, not complex interdependencies that would require a higher level language. As we later do the complex validation in the configuration scripts that are written in python themselves, all the complexity can already be put there. Now you may be wondering why this validation is done in two places, it's because of the legacy of vyatta-cfg. In the old days of vyatta, many config nodes didn't have corresponding scripts at all, they were self contained and applied the config directly using system utilities and simple shell scripts that were part of the node definitions themselves. In that case, the config node validators were the only validation of a value that was done and each config node coould specify their own shell snippet or script to validate its own value. This made sense in that design concept.
It is also still an integral part of the shell environment: in config mode, a set command with a invalid value will return an error immediately as its validator returns an error. The configuration script can catch an error only when a commit is triggered.
Now that we are tacking a completely different design concept onto that, things become complex. If the new design says: "all new code must be python" but since we're marrying this new code with the old vyatta-cfg core (vyatta-cfg is still the heart and core of VyOS with Python being the "worker"), things will become very unoptimal and complex and bizarre in some places that wouldn't need to be that way and could be left simple. The above being an example of this complexity due to a design choice.

Possibly it's in cases where it's first called in get_config(), then also later in verify() or apply() - the object is function local scope and isn't saved in a global variable. It would be a simple fix if Config() were initialised in main() and passed as an argument to any function that needs it.

Why is it sometimes called twice?

Begging for a merge will not speed things up. You multiple times refused to adopt the patch to our guidelines and it is still unclear if this is the right path to go.

I just had a look at VyConf and it is excellent, I fear that no one but @dmbaturin can maintain participate to it.
Also, VyConf will still need to fork all the python code and unless we have a resolution to T2088 - I am not sure what the best path forward will be

I tried to get a flamegraph showing what I was wanting to say but .. do not look very clear :-(

Please merge the following PR, if there is a problem, please let me modify it.

Well I do no longer maintain it so it‘s defacto dead. It only served as poc, I rather use some bash aliases now to build my packages and rely on the deb mirror.

No

@c-po IMO the script should be kept but fixed so it builds all valid packages. Otherwise there's no way to build our own packages with one command. Sure we can build them one by one by manually cloning each repository, but that's automated by this script. There's a task I already opened for it.

If so, it's better to consider porting the 1.2 NPT implementation instead of using a new solution. Can they coexist? I'm just a suggestion.

I believe it was git pull —tags which fixed it for me ..

Many scripts also so do implement if name == “main”

I'm working on a larger set of patches for DNS, a fix for this will be included

This does not compile python scripts without a .py extension (there are several in src/services, src/utils that have #!/usr/bin/env python3)

The recent implementation here uses the python ifconfig module and walks it to detect interfaces marked as beidgable. I found such constructs are way too slow, simply listing ls -1 /sys/class/net (and do some filtering) is magnitudes faster.

I think I drop the script as it was considered as a PoC but its heavily unmaintained

That is what VyConf is for

It‘s implemented in 1.2 but not with the new nftables based NAT backend as the required commands could not be translated from ip6tables.

Validator now prevents this

In addition a show opmode command should be added to list all the USB serial stuff in a human friendly way

Completion helper: