This is with Mellanox ConnectX-6 Dx. I'm using a vyos built with latest OFED driver, but in-kernel one will be the same. Based on what I've read the flowtable hardware offload was never tested due to lack of hardware, so I suspect other drivers will have the same problem.

The current nft and flowtable implementation creates seperate ip and ip6 tables, and defines the flowtable in both. From what I can tell (documentation is sparse), this seems not possible with hardware offload as one device can only be used in a single flowtable.

david@vyos01# set flowtable flowtablehw interface en1f0pf1sf88
[edit firewall]
david@vyos01# set flowtable flowtablehw offload hardware 
[edit firewall]
david@vyos01# commit
[ firewall ]
Unknown firewall error detected: /run/nftables.conf:384:15-40: Error:
Could not process rule: Device or resource busy     flowtable
VYOS_FLOWTABLE_flowtablehw {               ^^^^^^^^^^^^^^^^^^^^^^^^^^
[[firewall]] failed
Commit failed

Not that this is the flowtable definition in the ip6 chain, the one in the ip chain has already been loaded successfully.

I have verified this by adding nft tables myself. Adding ip or ip6 table with HW offload flowtable works, but adding both tables results in Device or resource busy error:

david@vyos01# sudo nft add 'table ip test  { flowtable fttest { hook ingress priority 0; devices =  { en1f0pf1sf88 }; flags offload; }; }'
[edit firewall]
david@vyos01# sudo nft add 'table ip6 test  { flowtable fttest { hook ingress priority 0; devices =  { en1f0pf1sf88 }; flags offload; }; }'
Error: Could not process rule: Device or resource busy
add table ip6 test  { flowtable fttest { hook ingress priority 0; devices =  { en1f0pf1sf88 }; flags offload; }; }

The (very few) documentation snippets I've seen all use table inet (combined v4/v6), so I guess that's really the only way this can work.

If someone wants to test this, here is what's needed to test HW flowtables on mlx5. Using TC offload with the base PF does *not* work - we need to create a subfunction (representor) device and then an ethernet device from that. The representor device is added to the flowtable, and the .

sudo devlink port add pci/0000:01:00.0 flavour pcisf pfnum 1 sfnum 88
sudo devlink port function set pci/0000:01:00.0/98304  hw_addr 56:04:da:67:25:12
sudo devlink port function set pci/0000:01:00.0/98304  state active
sudo devlink dev param set  auxiliary/mlx5_core.sf.1 name enable_eth value true cmode driverinit
sudo devlink dev reload auxiliary/mlx5_core.sf.1

With this added to /config/scripts/vyos-preconfig-bootup.script, I am able to create a flowtable with the representor device en1f0pf1sf88. But not in VyOS config because of the above.