Page MenuHomeVyOS Platform

hardware flowtables not working on supposedly supported hardware
Open, NormalPublicBUG

Description

danhusan@myv.c1:~$ ethtool -k eth0 | grep hw-tc-offload
hw-tc-offload: on
danhusan@myv.c1:~$ ethtool -k eth1 | grep hw-tc-offload
hw-tc-offload: on
danhusan@myv.c1:~$ ethtool -k eth2 | grep hw-tc-offload
hw-tc-offload: on
danhusan@myv.c1:~$ ethtool -k eth3 | grep hw-tc-offload
hw-tc-offload: on

Software working ok:

danhusan@myv.c1# set firewall flowtable test interface eth0
[edit]
danhusan@myv.c1# commit
danhusan@myv.c1#

Hardware not so much:

danhusan@myv.c1# set firewall flowtable test offload hardware 
[edit]
danhusan@myv.c1# commit

Failed to apply firewall: /run/nftables.conf:21:15-33: Error: Could not
process rule: Operation not supported     flowtable VYOS_FLOWTABLE_test
{               ^^^^^^^^^^^^^^^^^^^ /run/nftables.conf:108:15-33: Error:
Could not process rule: Operation not supported     flowtable
VYOS_FLOWTABLE_test {               ^^^^^^^^^^^^^^^^^^^

[[firewall]] failed
Commit failed
danhusan@myv.c1:~$ sudo dmesg | grep eth
[    2.347477] igb 0000:05:00.0: added PHC on eth0
[    2.353678] igb 0000:05:00.0: eth0: (PCIe:5.0Gb/s:Width x4) 00:25:90:5c:fd:42
[    2.358573] igb 0000:05:00.0: eth0: PBA No: 010A00-000
[    2.413335] igb 0000:05:00.1: added PHC on eth1
[    2.430705] igb 0000:05:00.1: eth1: (PCIe:5.0Gb/s:Width x4) 00:25:90:5c:fd:43
[    2.430790] igb 0000:05:00.1: eth1: PBA No: 010A00-000
[    2.505209] igb 0000:05:00.1 e3: renamed from eth1
[    2.586324] igb 0000:05:00.0 e2: renamed from eth0
[    3.523298] ixgbe 0000:04:00.0 eth0: MAC: 2, PHY: 1, PBA No: FFFFFF-0FF
[    3.534853] ixgbe 0000:04:00.0 eth0: Enabled Features: RxQ: 16 TxQ: 16 FdirHash
[    3.543850] ixgbe 0000:04:00.0 eth0: Intel(R) 10 Gigabit Network Connection
[    4.835303] ixgbe 0000:04:00.1 eth1: MAC: 2, PHY: 1, PBA No: FFFFFF-0FF
[    4.846821] ixgbe 0000:04:00.1 eth1: Enabled Features: RxQ: 16 TxQ: 16 FdirHash
[    4.856344] ixgbe 0000:04:00.1 eth1: Intel(R) 10 Gigabit Network Connection
[    4.866587] ixgbe 0000:04:00.1 e5: renamed from eth1
[    4.891498] ixgbe 0000:04:00.0 e4: renamed from eth0
[   23.121269] igb 0000:05:00.0 eth0: renamed from e2
[   23.139342] ixgbe 0000:04:00.1 eth3: renamed from e5
[   23.151000] igb 0000:05:00.1 eth1: renamed from e3
[   23.169424] ixgbe 0000:04:00.0 eth2: renamed from e4
[   47.263170] ixgbe 0000:04:00.1: registered PHC device on eth3
[   48.413555] ixgbe 0000:04:00.0: registered PHC device on eth2

Any ideas on how to troubleshoot further?

Details

Version
1.4.0
Is it a breaking change?
Perfectly compatible
Issue type
Bug (incorrect behavior)
Forum thread
https://forum.vyos.io/t/how-to-use-flowtables-in-vyos-1-4-with-zone-based-firewall/13004/15

Event Timeline

syncer triaged this task as Normal priority.Jun 27 2024, 11:27 PM

Try native nft commands for offload and check what it says.

# cat /tmp/offload.nft

 table inet x {
      flowtable f {
               hook ingress priority 10 devices = { eth0, eth1, eth2, eth3 }
	       flags offload
      }
      chain y {
               type filter hook forward priority 0; policy accept;
               ip protocol tcp flow offload @f
      }
 }

load

sudo nft -f /tmp/offload.nft

If it fails, it needs to open a bug report to Netfilter.

There is no information if it is a correct flag for hardware nft offload, or there should be some another check.

danhusan@myv.c1:~$ ethtool -k eth0 | grep hw-tc-offload
hw-tc-offload: on

Using native nft tooling does not seem to help, I will keep investigating - seems to be very little information online on the topic, mostly people having the same issue as me.

I have similar problem, testing setup:
VyOS 1.5-rolling-202407010024 as VM in proxmox 8.2
Intel x710 card in pci-e passthrow to the VM

eth1 is XGSPON SFP+ with VLAN and PPPoE inside
eth2 is DAC to the MikroTik switch

set firewall flowtable forward-table description 'Flow Table for the forward chain'
set firewall flowtable forward-table interface eth1
set firewall flowtable forward-table interface eth2
set firewall flowtable forward-table offload software
vyos@vyos:~$ ethtool -k eth1 | grep hw-tc-offload
hw-tc-offload: on
vyos@vyos:~$ ethtool -k eth2 | grep hw-tc-offload
hw-tc-offload: on

If I enable hw-tc-offload via

set interfaces ethernet eth1 offload hw-tc-offload

I get the message:

WARNING: could not change "eth1" flow control setting!

But using ethtool directly works

Enabling HW offload for flowables result in:

vyos@vyos# set firewall flowtable forward-table offload hardware
[edit]
vyos@vyos# commit

Failed to apply firewall: /run/nftables.conf:21:15-42: Error: Could not
process rule: Operation not supported     flowtable
VYOS_FLOWTABLE_forward-table {
^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /run/nftables.conf:30:48-85: Error: Could
not process rule: No such file or directory         ct state
{established,related} counter flow add @VYOS_FLOWTABLE_forward-table
comment "ipv4-FWD-filter-5"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /run/nftables.conf:104:15-42:
Error: Could not process rule: Operation not supported     flowtable
VYOS_FLOWTABLE_forward-table {
^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /run/nftables.conf:113:48-85: Error: Could
not process rule: No such file or directory         ct state
{established,related} counter flow add @VYOS_FLOWTABLE_forward-table
comment "ipv6-FWD-filter-5"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

[[firewall]] failed
Commit failed

Some info

To make hardware offloading works, the NIC should be programmable to load forwarding rules like flowtable and this is what switchdev doing

ref. https://forums.servethehome.com/index.php?threads/nftables-flowtables-hardware-offload-with-connectx-5.43714/#post-431605

After some research, I am not even sure that flowtables offload works on intel cards

Are the in-tree or out-of-tree drivers for your Intel NICs being used?

That is output of these commands would be nice:

modinfo igb | grep -i intree
modinfo ixgbe | grep -i intree

Also since you run in a VM, make sure that your passthrough is properly setup since hardware flowtable for obvious reasons needs direct access to the hardware itself without some virtualization in between. Software flowtable should work no matter what.

In my case, one interface is direct passthrough of physical interface via proxmox, other is a passthrough of VF (interface is splitter into several and shared across VMs).
I tried to enable hardware offloading for each of them separately without any success.

vyos@vyos:~$ modinfo igb | grep -i intree
intree:         Y
vyos@vyos:~$ modinfo ixgbe | grep -i intree
vyos@vyos:~$ modinfo i40e | grep -i intree
intree:         Y
vyos@vyos:~$ lspci | grep -E -i --color 'network|ethernet'
01:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)
02:00.0 Ethernet controller: Intel Corporation Ethernet Virtual Function 700 Series (rev 02)
dmbaturin changed Is it a breaking change? from Unspecified (possibly destroys the router) to Perfectly compatible.
dmbaturin changed Issue type from Unspecified (please specify) to Bug (incorrect behavior).

As mentioned above, I'm not sure the Intel drivers actually support hardware flow table offload. I came across a thread on the netfilter development mailing list where the maintainer mentions that the driver needs to implement TC_SETUP_FT and that hw-tc-offload is not sufficient. The thread was discussing the igb driver for the Intel i350, but a cursory search of driver sources I only saw reference to TC_SETUP_FT in Mellanox and Mediatek drivers. It is entirely possible I'm missing something though.

https://lore.kernel.org/all/890f23df-cdd6-4dab-9979-d5700d8b914b@dev-mail.net/T/