I'm looking into it. (not sure how I could assign the task to myself?)

I looked into ti, the xml is fairly easy, the perl code pretty nasty, it also refers (hardcoded) to sysV initd.
In my opinion it's almost impossible to convert the perl code into python without getting eye cancer, I think the python code needs to be written from scratch (will be invisible to the enduser on the cli, options will be the same)
Let me know if you need more details, I'm more than happy to discuss my finding.
The complete rewrite won't too long either, but hard to estimate due to the fact that the original code used libs and has different scripts for config and op commands.

@hagbard If you are working on it, by all means, assign it to yourself. We do not have a formal assignment policy, it's more like "I'm working on it" flag for coordination.

In my opinion it's almost impossible to convert the perl code into python without getting eye cancer

Perl code should be taken with a huge grain of Peyrone's salt indeed. ;)
We often have to take the risk though because a lot of time it contains some hidden knowledge of underlying software quirks, or turns out to be referenced in unexpected places.
Otherwise, those rewrites are indeed more of clean room rewrites, word for word rewrites would be a bad idea because many scripts suffer from poor architecture irrespective of the language they are written in.
And the new architecture designed to allow transactional commits and automated tests requires strict separation of obtaining the config data, validating it, generating target configs, and restarting services.

Make sure to read http://blog.vyos.net/vyos-2-dot-0-development-digest-number-5-doing-1-dot-2-x-and-2-dot-0-development-in-parallel for the motivation and existing scripts for real life examples.

the original code used libs and has different scripts for config and op commands

If you see code that can be shared between op and conf mode, it's a good idea to move it to the vyos.* module hierarchy.

Also, in op mode scripts I also deliberately separate data retrieval and formatting so that we can add machine readable output support (or support for alternative formats) simply by using a serializer.

Thanks for your comment. I read the wiki page already, but wasn't sure how to proceed. How do I assign myself to a task? I didn't find an option in phabricator to do it. I think I don't have permissions for it unless I created the task, I suppose.
I did dive through the majority of the existing code and am about to do some cleanups and therefore rewriting major parts. As far as I understand, that's the long term plan anyway until 2.0 can be released.
Let me know if it doesn't make sens to do surgery on the open heart, since it's going to be remove at one point anyway. Otherwise, I would check and see how I can become most useful to the entire project in general.

@hagbard If rewrite can be done in a reasonable time (days to a couple of weeks), I think trying to clean up old code is more trouble than it's worth.
With big stuff that cannot be easily rewritten, it may be worthwhile.

Hi Guys,

I've got a question which popped up in my mind during the rewrite of conntrack-sync.
The legacy implementation parses from cache IP, ports and protocol and displays it. It does not show other, IMHO important information such as timers.
Conntrackd basically echos the nf_conntrack tables (/proc/net/nf_conntrack && /proc/net/nf_conntrack_expect), while the cli output filters and shows only src, dst and protocol.

legacy:
Source Destination Protocol

Main Table Entries:

what I think would be better (conntrack - ct):
unknown 112 src=192.168.0.100 dst=224.0.0.18 [UNREPLIED] src=224.0.0.18 dst=192.168.0.100 mark=0 [active since 3524s]
udp 17 src=192.168.0.100 dst=225.0.0.50 sport=36829 dport=3780 [UNREPLIED] src=225.0.0.50 dst=192.168.0.100 sport=3780 dport=36829 [active since 3523s]
udp 17 src=192.168.0.101 dst=225.0.0.50 sport=53899 dport=3780 [UNREPLIED] src=225.0.0.50 dst=192.168.0.101 sport=3780 dport=53899 mark=0 [active since 3524s]
tcp 6 ESTABLISHED src=10.1.1.1 dst=10.1.1.100 sport=41688 dport=22 src=10.1.1.100 dst=10.1.1.1 sport=22 dport=41688 [ASSURED] mark=0 [active since 3524s]
tcp 6 ESTABLISHED src=10.1.1.1 dst=10.1.1.100 sport=41838 dport=22 src=10.1.1.100 dst=10.1.1.1 sport=22 dport=41838 [ASSURED] mark=0 [active since 3524s]

That displays activation timers etc. basically all the conntrack table information. However , that information is also available via snmp, so displaying internal and external connections doesn't make much sense to me.
I can easily implement it, but doubt that it's worth the time, does anyone know if that is required/used or useful to anybody?
Is it worth implementing it? (not hard to do)

I think it's useful for troubleshooting

All right. op-mode should be ready to go, but I haven't request a pull yet. I'm going to do still some tests and then moving onto cfg-mode.

Making progress. There is a ton of options not implemented (e.g. multicast with more than 1 interface for HA etc.), if I implement it now I suppose I will need to write migration scripts?
According to https://manpages.debian.org/testing/conntrackd/conntrackd.conf.5.en.html, there are really nice features which I think become very handy once implemented, but I would also have to change the cli interface, probably putting sections in.
e.g.
set multicast default interface ...
set multicast ha1 interface ...
set multicast ha2 interface ...

That looks like a massive overhaul for the interface, shall I do that or just implement what the perl script does right now (it's faulty too, when you set more than 1 interface for the same multicast group).
Let me know if you want to have a glimpse into the script code.
I personally think, sooner or later it needs top be done anyway.

My interest is sync over multiple interfaces in unicast mode (but multicast also) among other things

Oh there is more stuff, like the different modes and IPv6 which I think need to be implemented as well. My concern is if I change the nodes and all that stuff for the cli, it requires migration scripts, otherwise it will break configs from people who update.
How do you usually deal with that stuff?

@hagbard Migration scripts are no longer that much of a problem (though they still need to be carefully written). See http://blog.vyos.net/writing-migration-scripts-and-manipulating-vyos-config-files-outside-vyos-just-got-easier

to give you an idea what I have in mind:

service conntrack-sync:

general options
sync mode FTFW || ALARM || NOTRACK
protocol MULTICAST || TCP || UDP

Each would require different nodes.
Also if you look into the example section, the first is basically just doing accounting/statistics which could be a separate node which has already always a default config and you could activate it via let's say 'set service conntrack-sync statistics' (like single device installations).

How should I deal with the existing nodes, do they need to be preserved? Or removed and a migration script handles existing configs?

I think new version should still be able to sync with the old versions, since people will be upgrading routers in their HA pairs one by one, and loss of functionality during upgrades isn't a very nice situation.

Sounds good, I experimented already with it. I can drop in certain functions into the same nodes. So the existing ones will still work. However the implementation of it will take a few weeks, primarily testing will be crucial.
The other option which I think might be a good one, is to implement everything which does currently exist and shift the new functions in for each update. That way rollbacks should be way easier and the overview might be better too.
I really like the functions I read about and played with them already in VMs, you can actually download a file and kill 1 firewall which doesn't affect your connection. So, multiple sync interfaces should be the way to go anyway.
I gotta think about it a bit more, op-mode is way easier :).

Is there any work being done? If not, I'll move it to VyOS 1.3.x target, since 1.2.0 is about to get a feature freeze.

I started with it but it's far away from being finished, since I jumped on wireguard first and am currently on pppoe (accel-ppp).

VyOS 1.3 is fine for me