Page MenuHomeVyOS Platform

hardware flowtables not working on supposedly supported hardware
Open, NormalPublicBUG

Description

[email protected]:~$ ethtool -k eth0 | grep hw-tc-offload
hw-tc-offload: on
[email protected]:~$ ethtool -k eth1 | grep hw-tc-offload
hw-tc-offload: on
[email protected]:~$ ethtool -k eth2 | grep hw-tc-offload
hw-tc-offload: on
[email protected]:~$ ethtool -k eth3 | grep hw-tc-offload
hw-tc-offload: on

Software working ok:

[email protected]# set firewall flowtable test interface eth0
[edit]
[email protected]# commit
[email protected]#

Hardware not so much:

[email protected]# set firewall flowtable test offload hardware 
[edit]
[email protected]# commit

Failed to apply firewall: /run/nftables.conf:21:15-33: Error: Could not
process rule: Operation not supported     flowtable VYOS_FLOWTABLE_test
{               ^^^^^^^^^^^^^^^^^^^ /run/nftables.conf:108:15-33: Error:
Could not process rule: Operation not supported     flowtable
VYOS_FLOWTABLE_test {               ^^^^^^^^^^^^^^^^^^^

[[firewall]] failed
Commit failed
[email protected]:~$ sudo dmesg | grep eth
[    2.347477] igb 0000:05:00.0: added PHC on eth0
[    2.353678] igb 0000:05:00.0: eth0: (PCIe:5.0Gb/s:Width x4) 00:25:90:5c:fd:42
[    2.358573] igb 0000:05:00.0: eth0: PBA No: 010A00-000
[    2.413335] igb 0000:05:00.1: added PHC on eth1
[    2.430705] igb 0000:05:00.1: eth1: (PCIe:5.0Gb/s:Width x4) 00:25:90:5c:fd:43
[    2.430790] igb 0000:05:00.1: eth1: PBA No: 010A00-000
[    2.505209] igb 0000:05:00.1 e3: renamed from eth1
[    2.586324] igb 0000:05:00.0 e2: renamed from eth0
[    3.523298] ixgbe 0000:04:00.0 eth0: MAC: 2, PHY: 1, PBA No: FFFFFF-0FF
[    3.534853] ixgbe 0000:04:00.0 eth0: Enabled Features: RxQ: 16 TxQ: 16 FdirHash
[    3.543850] ixgbe 0000:04:00.0 eth0: Intel(R) 10 Gigabit Network Connection
[    4.835303] ixgbe 0000:04:00.1 eth1: MAC: 2, PHY: 1, PBA No: FFFFFF-0FF
[    4.846821] ixgbe 0000:04:00.1 eth1: Enabled Features: RxQ: 16 TxQ: 16 FdirHash
[    4.856344] ixgbe 0000:04:00.1 eth1: Intel(R) 10 Gigabit Network Connection
[    4.866587] ixgbe 0000:04:00.1 e5: renamed from eth1
[    4.891498] ixgbe 0000:04:00.0 e4: renamed from eth0
[   23.121269] igb 0000:05:00.0 eth0: renamed from e2
[   23.139342] ixgbe 0000:04:00.1 eth3: renamed from e5
[   23.151000] igb 0000:05:00.1 eth1: renamed from e3
[   23.169424] ixgbe 0000:04:00.0 eth2: renamed from e4
[   47.263170] ixgbe 0000:04:00.1: registered PHC device on eth3
[   48.413555] ixgbe 0000:04:00.0: registered PHC device on eth2

Any ideas on how to troubleshoot further?

Details

Difficulty level
Unknown (require assessment)
Version
1.4.0
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Unspecified (please specify)
Forum thread
https://forum.vyos.io/t/how-to-use-flowtables-in-vyos-1-4-with-zone-based-firewall/13004/15

Event Timeline

syncer triaged this task as Normal priority.Thu, Jun 27, 11:27 PM

Try native nft commands for offload and check what it says.

# cat /tmp/offload.nft

 table inet x {
      flowtable f {
               hook ingress priority 10 devices = { eth0, eth1, eth2, eth3 }
	       flags offload
      }
      chain y {
               type filter hook forward priority 0; policy accept;
               ip protocol tcp flow offload @f
      }
 }

load

sudo nft -f /tmp/offload.nft

If it fails, it needs to open a bug report to Netfilter.

There is no information if it is a correct flag for hardware nft offload, or there should be some another check.

[email protected]:~$ ethtool -k eth0 | grep hw-tc-offload
hw-tc-offload: on

Using native nft tooling does not seem to help, I will keep investigating - seems to be very little information online on the topic, mostly people having the same issue as me.

I have similar problem, testing setup:
VyOS 1.5-rolling-202407010024 as VM in proxmox 8.2
Intel x710 card in pci-e passthrow to the VM

eth1 is XGSPON SFP+ with VLAN and PPPoE inside
eth2 is DAC to the MikroTik switch

set firewall flowtable forward-table description 'Flow Table for the forward chain'
set firewall flowtable forward-table interface eth1
set firewall flowtable forward-table interface eth2
set firewall flowtable forward-table offload software
vyos@vyos:~$ ethtool -k eth1 | grep hw-tc-offload
hw-tc-offload: on
vyos@vyos:~$ ethtool -k eth2 | grep hw-tc-offload
hw-tc-offload: on

If I enable hw-tc-offload via

set interfaces ethernet eth1 offload hw-tc-offload

I get the message:

WARNING: could not change "eth1" flow control setting!

But using ethtool directly works

Enabling HW offload for flowables result in:

vyos@vyos# set firewall flowtable forward-table offload hardware
[edit]
vyos@vyos# commit

Failed to apply firewall: /run/nftables.conf:21:15-42: Error: Could not
process rule: Operation not supported     flowtable
VYOS_FLOWTABLE_forward-table {
^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /run/nftables.conf:30:48-85: Error: Could
not process rule: No such file or directory         ct state
{established,related} counter flow add @VYOS_FLOWTABLE_forward-table
comment "ipv4-FWD-filter-5"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /run/nftables.conf:104:15-42:
Error: Could not process rule: Operation not supported     flowtable
VYOS_FLOWTABLE_forward-table {
^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /run/nftables.conf:113:48-85: Error: Could
not process rule: No such file or directory         ct state
{established,related} counter flow add @VYOS_FLOWTABLE_forward-table
comment "ipv6-FWD-filter-5"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

[[firewall]] failed
Commit failed

Some info

To make hardware offloading works, the NIC should be programmable to load forwarding rules like flowtable and this is what switchdev doing

ref. https://forums.servethehome.com/index.php?threads/nftables-flowtables-hardware-offload-with-connectx-5.43714/#post-431605