Firewalld (which is a software that creates nftables rulesets similar to VyOS config-mode) have recently added support for acceleration through software fastpath based on flowtables in nftables.
So far the results looks promising (+58.5% increase in throughput for the usecase shown in the below blogpost (from 12.47Gbps to 19.77Gbps with 2048 concurrent connections)) however I am currently not aware of any particular drawbacks with this feature:
https://firewalld.org/2023/05/nftables-flowtable
From the looks of it this should be fairly easy to implement into VyOS.
A suggestion would be that this is by default disabled but can be enabled by adding the interfaces that should operate through flowtables either by software or hardware acceleration:
set interface ethernet ethX offload_firewall <value>
Where accepted values are "none" (default), "flowtable_software" or "flowtable_hardware").
In the Firewalld case they enable flowtable by adding one flowtable object and one rule to the nftables ruleset:
table inet firewalld { flowtable fastpath { hook ingress priority filter + 10 devices = { eth0, eth1 } } [..] chain filter_FORWARD { type filter hook forward priority filter + 10; policy accept; ct state { established, related } meta l4proto { tcp, udp } flow add @fastpath <--- new rule ct state { established, related } accept [..]
To enable hardware offloading "flags offload;" is added to the "flowtable fastpath" object.
For more information:
https://wiki.nftables.org/wiki-nftables/index.php/Flowtables
https://wiki.nftables.org/wiki-nftables/index.php/Netfilter_hooks