Consider implementing (NAT/other) flow table offload
Closed, ResolvedPublicFEATURE REQUEST
Actions

Assigned To

Authored By

	Viacheslav
	Jul 1 2022, 2:27 PM

Description

SNAT, Use alternative forwarding path via nftables flowtable (fastpath bypass)

                                       userspace process
                                        ^              |
                                        |              |
                                   _____|____     ____\/___
                                  /          \   /         \
                                  |   input   |  |  output  |
                                  \__________/   \_________/
                                       ^               |
                                       |               |
    _________      __________      ---------     _____\/_____
   /         \    /          \     |Routing |   /            \
-->  ingress  ---> prerouting ---> |decision|   | postrouting |--> neigh_xmit
   \_________/    \__________/     ----------   \____________/          ^
     |      ^                          |               ^                |
 flowtable  |                     ____\/___            |                |
     |      |                    /         \           |                |
  __\/___   |                    | forward |------------                |
  |-----|   |                    \_________/                            |
  |-----|   |                 'flow offload' rule                       |
  |-----|   |                   adds entry to                           |
  |_____|   |                     flowtable                             |
     |      |                                                           |
    / \     |                                                           |
   /hit\_no_|                                                           |
   \ ? /                                                                |
    \ /                                                                 |
     |__yes_________________fastpath bypass ____________________________|

Example configuration:

table inet x {
        flowtable f {
                hook ingress priority 0; devices = { eth0, eth1 };
        }
        chain y {
                type filter hook forward priority 0; policy accept;
                ip protocol tcp flow offload @f
                counter packets 0 bytes 0
        }
}

Example NAT

vyos@r14:~$ cat nat.nft 
flush ruleset

table ip filter {
	flowtable fastnat {
		hook ingress priority filter
		devices = { eth0, eth1 }
	}

	chain forward {
		type filter hook forward priority filter; policy accept;
		ip protocol { tcp, udp } flow add @fastnat
	}
}
table ip nat {
	chain POSTROUTING {
		type nat hook postrouting priority srcnat; policy accept;
		ip saddr 192.0.2.0/24 oif "eth0" snat to 192.168.122.14 persistent
	}

	chain PREROUTING {
		type nat hook prerouting priority dstnat; policy accept;
	}
}

https://www.kernel.org/doc/html/v5.10/networking/nf_flowtable.html
https://lwn.net/Articles/738214/
https://blog.fearcat.in/a?ID=01600-b829de09-5dee-4117-afb5-28b005e630a2

Details

Difficulty level: Unknown (require assessment)
Version: -
Why the issue appeared?: Will be filled on close
Is it a breaking change?: Unspecified (possibly destroys the router)
Issue type: Unspecified (please specify)

Related Objects

Mentioned In: T5419: Software/Hardware fastpath with nftables flowtable
Mentioned Here: T5419: Software/Hardware fastpath with nftables flowtable

Event Timeline

Viacheslav created this task.Jul 1 2022, 2:27 PM

Viacheslav renamed this task from Consider implementing NAT flow table offload to Consider implementing (NAT/other) flow table offload.Jul 2 2022, 1:23 PM

pasik added a subscriber: pasik.Jul 3 2022, 8:34 AM

hard awarded a token.Aug 10 2022, 8:33 PM

hard added a subscriber: hard.

Viacheslav updated the task description. (Show Details)Aug 30 2022, 7:36 AM

Viacheslav updated the task description. (Show Details)Aug 30 2022, 11:46 AM

From kernel 5.13, hardware offload is supported (if nic supports it).
Info: https://www.kernel.org/doc/html/v5.13/networking/nf_flowtable.html#hardware-offload

If possible, it would be great to be able to compare performance:
No flowtable vs soft flowtables offload vs hw flowtable offload

Required version for offload hardware flag nftables 0.9.9
The current version we use 0.9.8-3.1

on nightly build nftables v1.0.5 and kernel 5.15.76

we can start thinking about how to integrate the parameters into the config structure
list of required parameters are:

devices in flowtable
offloaded ip protocol in forwarding chain
hardware offload flag as n.fort suggests

mpueschel added a subscriber: mpueschel.Nov 2 2022, 9:20 AM

i see it like that

nat {
  flowtable NAME {
    hw-offload
    member {
        interface eth1
        interface eth2
      }
    protocols {
      tcp
      udp
      }
    }
}

any suggestions? @Viacheslav @n.fort

or maybe better add this subsection in firewall section?

Viacheslav added a subscriber: sarthurdev.Nov 13 2022, 4:37 PM

ServerForge added a subscriber: ServerForge.Feb 12 2023, 4:00 AM

I opened PR https://github.com/vyos/vyos-1x/pull/2062 for this.

Hi @vfreex . The idea is to add this feature, and several more, once we get new firewall cli structure: https://github.com/vyos/vyos-1x/pull/2016

eronlloyd added a subscriber: eronlloyd.Jul 11 2023, 1:41 AM

Viacheslav mentioned this in T5419: Software/Hardware fastpath with nftables flowtable.Jul 30 2023, 7:16 AM

As mentioned in https://vyos.dev/T5419 the offloading should not only apply for NAT.

Also it would be prefered if there were option to select between flowtable_software and flowtable_hardware per inferface (or whatever proper naming there might be).

Suggestion is that this setting should be placed in the "set interface ethernet ethX" section instead such as:

set interface ethernet ethX offload_firewall <value>

Where accepted values are "none" (default), "flowtable_software" or "flowtable_hardware").

Note that PR2062 is broken.

Please see:

https://github.com/vyos/vyos-1x/pull/2062#issuecomment-1720105493

and

https://github.com/vyos/vyos-1x/pull/2062#issuecomment-1720377092

for more information.

Some extra lines were mistakenly included during rebase:

{{ group_tmpl.groups(group, True) }}
}

I saw @c-po had removed the extra } in 45cfd56. I created https://github.com/vyos/vyos-1x/pull/2272 to remove the extra {{ group_tmpl.groups(group, True) }}

Tried to enable both software and hardware flowtable with VyOS 1.5-rolling-202309151051:

flow-offload {
    hardware {
        interface eth0
        interface eth1
        interface eth2
        interface eth3
    }
    software {
        interface eth0
        interface eth1
        interface eth2
        interface eth3
    }
}

Got this when trying to commit:

vyos@vyos# commit
[ firewall ]
Failed to apply firewall: /run/nftables.conf:271:11-33: Error: Could not
process rule: Operation not supported flowtable VYOS_FLOWTABLE_hardware
{           ^^^^^^^^^^^^^^^^^^^^^^^ /run/nftables.conf:279:69-101:
Error: Could not process rule: No such file or directory         ct
state { established, related } meta l4proto { tcp, udp } flow add
@VYOS_FLOWTABLE_hardware
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

[[firewall]] failed
Commit failed

Content of /run/nftables.conf (regarding flowtable):

table inet vyos_offload                                                                              
delete table inet vyos_offload                                                                       
table inet vyos_offload {                                                                            
flowtable VYOS_FLOWTABLE_hardware {                                                                  
    hook ingress priority filter - 2; devices = { eth0, eth1, eth2, eth3 };                          
    flags offload;                                                                                   
    counter                                                                                          
}                                                                                                    
                                                                                                     
    chain VYOS_OFFLOAD_hardware {                                                                    
        type filter hook forward priority filter - 2; policy accept;                                 
        ct state { established, related } meta l4proto { tcp, udp } flow add @VYOS_FLOWTABLE_hardware
    }                                                                                                
flowtable VYOS_FLOWTABLE_software {                                                                  
    hook ingress priority filter - 1; devices = { eth0, eth1, eth2, eth3 };                          
    counter                                                                                          
}                                                                                                    
                                                                                                     
    chain VYOS_OFFLOAD_software {                                                                                                                                                  
        type filter hook forward priority filter - 1; policy accept;                                 
        ct state { established, related } meta l4proto { tcp, udp } flow add @VYOS_FLOWTABLE_software
    }                                                                                                
}

Im looking into if one is supposed to be able to mix software and hardware flowtable but also how nft is supposed to react if the nic doesnt support hardware flowtable.

But in the meantime...

Why is there an empty table inet vyos_offload before that delete line?

table inet vyos_offload                                                                              
delete table inet vyos_offload                                                                       
table inet vyos_offload {

Looks like a typo at line 276?

https://github.com/vyos/vyos-1x/blob/current/data/templates/firewall/nftables.j2#L276

Using hardware flowtable (flags offload;) on an interface which doesnt support it returns an error:

The 2nd line of error (trying to reference @VYOS_FLOWTABLE_hardware) is because "flowtable VYOS_FLOWTABLE_hardware" failed to be defined due to lack of hw offload support by the nic.

./nftables.conf:284:11-33: Error: Could not process rule: Operation not supported
flowtable VYOS_FLOWTABLE_hardware {
          ^^^^^^^^^^^^^^^^^^^^^^^
./nftables.conf:292:65-97: Error: Could not process rule: No such file or directory
    ct state { established, related } meta l4proto { tcp, udp } flow add @VYOS_FLOWTABLE_hardware
                                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

It seems like an interface can either be software OR hardware offloaded.

If you try to add it twice (as in "try to hardware offload, if that fails then try to software offload") you will end up with following error:

Again 2nd line of error is due to that "flowtable VYOS_FLOWTABLE_software" failed to complete and therefor doesnt exist when @VYOS_FLOWTABLE_software is being referenced.

./nftables.conf:295:11-33: Error: Could not process rule: File exists
flowtable VYOS_FLOWTABLE_software {
          ^^^^^^^^^^^^^^^^^^^^^^^
./nftables.conf:302:65-97: Error: Could not process rule: No such file or directory
    ct state { established, related } meta l4proto { tcp, udp } flow add @VYOS_FLOWTABLE_software
                                                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Im trying to verify with #netfilter if the above observations are correct (or if the errors are due to something else).

@Apachez The empty table inet vyos_offload is to ensure the table exists before deleting its content. Regarding hardware offload, I don't have a hardware supporting that. The implementation is totally based on documentation and I don't add any checks before applying the nftables config.

Some feedback from the #netfilter channel over at libera.chat:

Software flowtable is supported by all nics (simply because this is a kernel thingy).

Hardware flowtable is only supported by some nics (aka needs hardware and driver support to be able to utilize hardware flowtable).

According to https://wiki.nftables.org/wiki-nftables/index.php/Flowtables the number of nics that supports hardware flowtable are limited.

A talk from 2019 lists these nics as candidates:

https://youtu.be/dIr88TIi9JM?t=3820

Not the same slides but from the same year:

https://lpc.events/event/4/contributions/463/attachments/286/485/2019-plumbers-lisboa.pdf

– Mellanox: mlx5, mlxsw (flower)
– Broadcom: bnxt (flower), bcm_sf2 (ethtool)
– Chelsio: cxgdb4 (flower)
– Intel: i40eia, iavf, igb (flower)
– Qlogic: qede (ethtool + flower)
– Mscc: ocelot (flower)
– Netronome: nfp (flower)

Also good reading:

https://www.kernel.org/doc/html/latest/networking/nf_flowtable.html

https://thermalcircle.de/doku.php?id=blog:linux:flowtables_1_a_netfilter_nftables_fastpath

For nics that supports hardware flowtable you can mix the way VyOS does it with priority -2 for hardware and priority -1 for software.

Note that when hardware flowtable kicks in only the first initial packets will be seen by tcpdump, the rest of the packets for this flow wont show up because the CPU doesnt "see" them when they gets "offloaded".

Trying to apply a hardware flowtable for a nic that doesnt support this will raise EOPNOTSUPP and the apply through nft will fail. There are however ongoing discussions if this fail should become a warning instead. That is that the nic would silently fallback to software flowtable instead of bringing an error when trying to apply the ruleset.

https://lore.kernel.org/netdev/[email protected]/T/

...

Given the above information the question is if software flowtable shouldnt be default that is always enabled and then let the user decide if set firewall flow-offload hardware interface ethX should be set and for which interface(s)?

This way perhaps the vyos-config could trap the error from nft and display better information to the user why the setting might not want to be enabled?

That is unless there is some flag that can be fetched through ethtool or so regarding if the device supports hardware flowtables or not (and if so then only list those interfaces)?

Perhaps a possible way to detect if the nic supports hardware flowtables or not.

Try to set sudo ethtool -K eth0 hw-tc-offload on.

If the result becomes:

Actual changes:
hw-tc-offload: off [requested on]
Could not change any device features

Then it doesnt support hardware flowtables.

Could also verify by reading the capability like so:

$ ethtool -k eth0 | grep hw-tc-offload
hw-tc-offload: off [fixed]

In T4502#160404, @Apachez wrote:
Perhaps a possible way to detect if the nic supports hardware flowtables or not.

Try to set sudo ethtool -K eth0 hw-tc-offload on.

If the result becomes:
Actual changes:
hw-tc-offload: off [requested on]
Could not change any device features
Then it doesnt support hardware flowtables.

Could also verify by reading the capability like so:
$ ethtool -k eth0 | grep hw-tc-offload
hw-tc-offload: off [fixed]

This has been implemented here: https://github.com/vyos/vyos-1x/commit/bbe3274#diff-22639aad95616c8d057fd5f8e672c148e7d29e26077361e9175e089971c0f267R295

Viacheslav changed the task status from Open to Needs testing.Sep 20 2023, 6:43 AM

The blog over at claims:

https://blog.vyos.io/vyos-project-october-2023-update

set firewall flowtable <name> interface <name>
set firewall flowtable <name> offload [software|hardware]
set system conntrack flow-accounting
set firewall [ipv4|ipv6] forward filter rule <n> action offload
set firewall [ipv4|ipv6] forward filter rule <n> offload-target <name>
# For hardware offload:
set interfaces ethernet <name> offload hw-tc-offload

But there is zero information about this in the manual!?

Is for example set system conntrack flow-accounting really needed in order to enable software/hardware flowtable?

Do I need to enable this per rule, I cant enable this globally?

If I need to enable it as rule aswell, should the num be low or high?

Per discussion on forums (https://forum.vyos.io/t/how-to-use-flowtables-in-vyos-1-4-with-zone-based-firewall/13004/4), custom chains do not currently have the ability to set offload as an action. The action is only able to be applied directly within the forward filter.

[edit]
vy1@vyos# edit firewall ipv4 name CONN_FILTER rule 10
[edit firewall ipv4 name CONN_FILTER rule 10]
vy1@vyos# set action
Possible completions:
   accept               Accept matching entries
   continue             Continue parsing next rule
   jump                 Jump to another chain
   reject               Reject matching entries
   return               Return from the current chain and continue at the next rule of the last chain
   drop                 Drop matching entries
   queue                Enqueue packet to userspace
   synproxy             Synproxy connections



[edit firewall ipv4 name CONN_FILTER rule 10]
vy1@vyos# set action offload



  Invalid value
  Value validation failed
  Set failed

GurliGebis added a subscriber: GurliGebis.Dec 6 2023, 3:42 PM

I have implemented this - PR: https://github.com/vyos/vyos-1x/pull/2638

set interfaces ethernet eth0 address dhcp
set interfaces ethernet eth1 address 10.10.10.1/24
set firewall flowtable TRAFFIC interface eth0
set firewall flowtable TRAFFIC interface eth1
set firewall flowtable TRAFFIC offload software
set system conntrack flow-accounting
set firewall ipv4 name TEST rule 10 action offload
set firewall ipv4 name TEST rule 10 offload-target TRAFFIC

Results in this:

vyos@vyos:~$ sudo nft -s list table ip vyos_filter
table ip vyos_filter {
        flowtable VYOS_FLOWTABLE_TRAFFIC {
                hook ingress priority filter
                devices = { eth0, eth1 }
                counter
        }

        chain VYOS_FORWARD_filter {
                type filter hook forward priority filter; policy accept;
                counter accept comment "FWD-filter default-action accept"
        }

        chain VYOS_INPUT_filter {
                type filter hook input priority filter; policy accept;
                counter accept comment "INP-filter default-action accept"
        }

        chain VYOS_OUTPUT_filter {
                type filter hook output priority filter; policy accept;
                counter accept comment "OUT-filter default-action accept"
        }

        chain VYOS_FRAG_MARK {
                type filter hook prerouting priority -450; policy accept;
                ip frag-off & 16383 != 0 meta mark set 0x000ffff1 return
        }

        chain NAME_TEST {
                counter flow add @VYOS_FLOWTABLE_TRAFFIC comment "ipv4-NAM-TEST-10"
                counter drop comment "TEST default-action drop"
        }
}

Pull requests for 1.4 backport: https://github.com/vyos/vyos-1x/pull/2641

danhusan added a subscriber: danhusan.Dec 18 2023, 8:03 PM

dmbaturin closed this task as Resolved.Jan 10 2024, 10:20 PM

dmbaturin claimed this task.

Consider implementing (NAT/other) flow table offloadClosed, ResolvedPublicFEATURE REQUESTActions

Description

Details

Related Objects

Event Timeline

Consider implementing (NAT/other) flow table offload
Closed, ResolvedPublicFEATURE REQUEST
Actions