From kernel 5.13, hardware offload is supported (if nic supports it).
Info: https://www.kernel.org/doc/html/v5.13/networking/nf_flowtable.html#hardware-offload

If possible, it would be great to be able to compare performance:
No flowtable vs soft flowtables offload vs hw flowtable offload

Required version for offload hardware flag nftables 0.9.9
The current version we use 0.9.8-3.1

on nightly build nftables v1.0.5 and kernel 5.15.76

we can start thinking about how to integrate the parameters into the config structure
list of required parameters are:

• devices in flowtable
• offloaded ip protocol in forwarding chain
• hardware offload flag as n.fort suggests

i see it like that

nat {
  flowtable NAME {
    hw-offload
    member {
        interface eth1
        interface eth2
      }
    protocols {
      tcp
      udp
      }
    }
}

any suggestions? @Viacheslav @n.fort

or maybe better add this subsection in firewall section?

I opened PR https://github.com/vyos/vyos-1x/pull/2062 for this.

Hi @vfreex . The idea is to add this feature, and several more, once we get new firewall cli structure: https://github.com/vyos/vyos-1x/pull/2016

As mentioned in https://vyos.dev/T5419 the offloading should not only apply for NAT.

Also it would be prefered if there were option to select between flowtable_software and flowtable_hardware per inferface (or whatever proper naming there might be).

Suggestion is that this setting should be placed in the "set interface ethernet ethX" section instead such as:

set interface ethernet ethX offload_firewall <value>

Where accepted values are "none" (default), "flowtable_software" or "flowtable_hardware").

Note that PR2062 is broken.

Please see:

https://github.com/vyos/vyos-1x/pull/2062#issuecomment-1720105493

and

https://github.com/vyos/vyos-1x/pull/2062#issuecomment-1720377092

for more information.

Some extra lines were mistakenly included during rebase:

{{ group_tmpl.groups(group, True) }}
}

I saw @c-po had removed the extra } in 45cfd56. I created https://github.com/vyos/vyos-1x/pull/2272 to remove the extra {{ group_tmpl.groups(group, True) }}

Tried to enable both software and hardware flowtable with VyOS 1.5-rolling-202309151051:

flow-offload {
    hardware {
        interface eth0
        interface eth1
        interface eth2
        interface eth3
    }
    software {
        interface eth0
        interface eth1
        interface eth2
        interface eth3
    }
}

Got this when trying to commit:

vyos@vyos# commit
[ firewall ]
Failed to apply firewall: /run/nftables.conf:271:11-33: Error: Could not
process rule: Operation not supported flowtable VYOS_FLOWTABLE_hardware
{           ^^^^^^^^^^^^^^^^^^^^^^^ /run/nftables.conf:279:69-101:
Error: Could not process rule: No such file or directory         ct
state { established, related } meta l4proto { tcp, udp } flow add
@VYOS_FLOWTABLE_hardware
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

[[firewall]] failed
Commit failed

Content of /run/nftables.conf (regarding flowtable):

table inet vyos_offload                                                                              
delete table inet vyos_offload                                                                       
table inet vyos_offload {                                                                            
flowtable VYOS_FLOWTABLE_hardware {                                                                  
    hook ingress priority filter - 2; devices = { eth0, eth1, eth2, eth3 };                          
    flags offload;                                                                                   
    counter                                                                                          
}                                                                                                    
                                                                                                     
    chain VYOS_OFFLOAD_hardware {                                                                    
        type filter hook forward priority filter - 2; policy accept;                                 
        ct state { established, related } meta l4proto { tcp, udp } flow add @VYOS_FLOWTABLE_hardware
    }                                                                                                
flowtable VYOS_FLOWTABLE_software {                                                                  
    hook ingress priority filter - 1; devices = { eth0, eth1, eth2, eth3 };                          
    counter                                                                                          
}                                                                                                    
                                                                                                     
    chain VYOS_OFFLOAD_software {                                                                                                                                                  
        type filter hook forward priority filter - 1; policy accept;                                 
        ct state { established, related } meta l4proto { tcp, udp } flow add @VYOS_FLOWTABLE_software
    }                                                                                                
}

Im looking into if one is supposed to be able to mix software and hardware flowtable but also how nft is supposed to react if the nic doesnt support hardware flowtable.

But in the meantime...

Why is there an empty table inet vyos_offload before that delete line?

table inet vyos_offload                                                                              
delete table inet vyos_offload                                                                       
table inet vyos_offload {

Looks like a typo at line 276?

https://github.com/vyos/vyos-1x/blob/current/data/templates/firewall/nftables.j2#L276

1. Using hardware flowtable (flags offload;) on an interface which doesnt support it returns an error:

The 2nd line of error (trying to reference @VYOS_FLOWTABLE_hardware) is because "flowtable VYOS_FLOWTABLE_hardware" failed to be defined due to lack of hw offload support by the nic.

./nftables.conf:284:11-33: Error: Could not process rule: Operation not supported
flowtable VYOS_FLOWTABLE_hardware {
          ^^^^^^^^^^^^^^^^^^^^^^^
./nftables.conf:292:65-97: Error: Could not process rule: No such file or directory
    ct state { established, related } meta l4proto { tcp, udp } flow add @VYOS_FLOWTABLE_hardware
                                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

2. It seems like an interface can either be software OR hardware offloaded.

If you try to add it twice (as in "try to hardware offload, if that fails then try to software offload") you will end up with following error:

Again 2nd line of error is due to that "flowtable VYOS_FLOWTABLE_software" failed to complete and therefor doesnt exist when @VYOS_FLOWTABLE_software is being referenced.

./nftables.conf:295:11-33: Error: Could not process rule: File exists
flowtable VYOS_FLOWTABLE_software {
          ^^^^^^^^^^^^^^^^^^^^^^^
./nftables.conf:302:65-97: Error: Could not process rule: No such file or directory
    ct state { established, related } meta l4proto { tcp, udp } flow add @VYOS_FLOWTABLE_software
                                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Im trying to verify with #netfilter if the above observations are correct (or if the errors are due to something else).

@Apachez The empty table inet vyos_offload is to ensure the table exists before deleting its content. Regarding hardware offload, I don't have a hardware supporting that. The implementation is totally based on documentation and I don't add any checks before applying the nftables config.

Some feedback from the #netfilter channel over at libera.chat:

1. Software flowtable is supported by all nics (simply because this is a kernel thingy).

2. Hardware flowtable is only supported by some nics (aka needs hardware and driver support to be able to utilize hardware flowtable).

According to https://wiki.nftables.org/wiki-nftables/index.php/Flowtables the number of nics that supports hardware flowtable are limited.

A talk from 2019 lists these nics as candidates:

https://youtu.be/dIr88TIi9JM?t=3820

Not the same slides but from the same year:

https://lpc.events/event/4/contributions/463/attachments/286/485/2019-plumbers-lisboa.pdf

– Mellanox: mlx5, mlxsw (flower)
– Broadcom: bnxt (flower), bcm_sf2 (ethtool)
– Chelsio: cxgdb4 (flower)
– Intel: i40eia, iavf, igb (flower)
– Qlogic: qede (ethtool + flower)
– Mscc: ocelot (flower)
– Netronome: nfp (flower)

Also good reading:

https://www.kernel.org/doc/html/latest/networking/nf_flowtable.html

https://thermalcircle.de/doku.php?id=blog:linux:flowtables_1_a_netfilter_nftables_fastpath

3. For nics that supports hardware flowtable you can mix the way VyOS does it with priority -2 for hardware and priority -1 for software.

Note that when hardware flowtable kicks in only the first initial packets will be seen by tcpdump, the rest of the packets for this flow wont show up because the CPU doesnt "see" them when they gets "offloaded".

4. Trying to apply a hardware flowtable for a nic that doesnt support this will raise EOPNOTSUPP and the apply through nft will fail. There are however ongoing discussions if this fail should become a warning instead. That is that the nic would silently fallback to software flowtable instead of bringing an error when trying to apply the ruleset.

https://lore.kernel.org/netdev/[email protected]/T/

...

Given the above information the question is if software flowtable shouldnt be default that is always enabled and then let the user decide if set firewall flow-offload hardware interface ethX should be set and for which interface(s)?

This way perhaps the vyos-config could trap the error from nft and display better information to the user why the setting might not want to be enabled?

That is unless there is some flag that can be fetched through ethtool or so regarding if the device supports hardware flowtables or not (and if so then only list those interfaces)?

Perhaps a possible way to detect if the nic supports hardware flowtables or not.

Try to set sudo ethtool -K eth0 hw-tc-offload on.

If the result becomes:

Actual changes:
hw-tc-offload: off [requested on]
Could not change any device features

Then it doesnt support hardware flowtables.

Could also verify by reading the capability like so:

$ ethtool -k eth0 | grep hw-tc-offload
hw-tc-offload: off [fixed]

In T4502#160404, @Apachez wrote:

Perhaps a possible way to detect if the nic supports hardware flowtables or not.

Try to set sudo ethtool -K eth0 hw-tc-offload on.

If the result becomes:

Actual changes:
hw-tc-offload: off [requested on]
Could not change any device features

Then it doesnt support hardware flowtables.

Could also verify by reading the capability like so:

$ ethtool -k eth0 | grep hw-tc-offload
hw-tc-offload: off [fixed]

This has been implemented here: https://github.com/vyos/vyos-1x/commit/bbe3274#diff-22639aad95616c8d057fd5f8e672c148e7d29e26077361e9175e089971c0f267R295