Page MenuHomeVyOS Platform
Create Task
Maniphest T6526

hardware flowtables not working on supposedly supported hardware
Open, NormalPublicBUG
Actions

Description

[email protected]:~$ ethtool -k eth0 | grep hw-tc-offload
hw-tc-offload: on
[email protected]:~$ ethtool -k eth1 | grep hw-tc-offload
hw-tc-offload: on
[email protected]:~$ ethtool -k eth2 | grep hw-tc-offload
hw-tc-offload: on
[email protected]:~$ ethtool -k eth3 | grep hw-tc-offload
hw-tc-offload: on

Software working ok:

[email protected]# set firewall flowtable test interface eth0
[edit]
[email protected]# commit
[email protected]#

Hardware not so much:

[email protected]# set firewall flowtable test offload hardware 
[edit]
[email protected]# commit

Failed to apply firewall: /run/nftables.conf:21:15-33: Error: Could not
process rule: Operation not supported     flowtable VYOS_FLOWTABLE_test
{               ^^^^^^^^^^^^^^^^^^^ /run/nftables.conf:108:15-33: Error:
Could not process rule: Operation not supported     flowtable
VYOS_FLOWTABLE_test {               ^^^^^^^^^^^^^^^^^^^

[[firewall]] failed
Commit failed
[email protected]:~$ sudo dmesg | grep eth
[    2.347477] igb 0000:05:00.0: added PHC on eth0
[    2.353678] igb 0000:05:00.0: eth0: (PCIe:5.0Gb/s:Width x4) 00:25:90:5c:fd:42
[    2.358573] igb 0000:05:00.0: eth0: PBA No: 010A00-000
[    2.413335] igb 0000:05:00.1: added PHC on eth1
[    2.430705] igb 0000:05:00.1: eth1: (PCIe:5.0Gb/s:Width x4) 00:25:90:5c:fd:43
[    2.430790] igb 0000:05:00.1: eth1: PBA No: 010A00-000
[    2.505209] igb 0000:05:00.1 e3: renamed from eth1
[    2.586324] igb 0000:05:00.0 e2: renamed from eth0
[    3.523298] ixgbe 0000:04:00.0 eth0: MAC: 2, PHY: 1, PBA No: FFFFFF-0FF
[    3.534853] ixgbe 0000:04:00.0 eth0: Enabled Features: RxQ: 16 TxQ: 16 FdirHash
[    3.543850] ixgbe 0000:04:00.0 eth0: Intel(R) 10 Gigabit Network Connection
[    4.835303] ixgbe 0000:04:00.1 eth1: MAC: 2, PHY: 1, PBA No: FFFFFF-0FF
[    4.846821] ixgbe 0000:04:00.1 eth1: Enabled Features: RxQ: 16 TxQ: 16 FdirHash
[    4.856344] ixgbe 0000:04:00.1 eth1: Intel(R) 10 Gigabit Network Connection
[    4.866587] ixgbe 0000:04:00.1 e5: renamed from eth1
[    4.891498] ixgbe 0000:04:00.0 e4: renamed from eth0
[   23.121269] igb 0000:05:00.0 eth0: renamed from e2
[   23.139342] ixgbe 0000:04:00.1 eth3: renamed from e5
[   23.151000] igb 0000:05:00.1 eth1: renamed from e3
[   23.169424] ixgbe 0000:04:00.0 eth2: renamed from e4
[   47.263170] ixgbe 0000:04:00.1: registered PHC device on eth3
[   48.413555] ixgbe 0000:04:00.0: registered PHC device on eth2

Any ideas on how to troubleshoot further?

Details

Difficulty level
Unknown (require assessment)
Version
1.4.0
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Unspecified (please specify)
Forum thread
https://forum.vyos.io/t/how-to-use-flowtables-in-vyos-1-4-with-zone-based-firewall/13004/15

Event Timeline

danhusan created this task.Jun 27 2024, 7:57 PM
pasik subscribed.Jun 27 2024, 8:01 PM
syncer triaged this task as Normal priority.Jun 27 2024, 11:27 PM
Viacheslav subscribed.Jun 28 2024, 7:19 AM

Try native nft commands for offload and check what it says.

# cat /tmp/offload.nft

 table inet x {
      flowtable f {
               hook ingress priority 10 devices = { eth0, eth1, eth2, eth3 }
	       flags offload
      }
      chain y {
               type filter hook forward priority 0; policy accept;
               ip protocol tcp flow offload @f
      }
 }

load

sudo nft -f /tmp/offload.nft

If it fails, it needs to open a bug report to Netfilter.

There is no information if it is a correct flag for hardware nft offload, or there should be some another check.

[email protected]:~$ ethtool -k eth0 | grep hw-tc-offload
hw-tc-offload: on
danhusan added a comment.Jun 28 2024, 7:42 PM

Using native nft tooling does not seem to help, I will keep investigating - seems to be very little information online on the topic, mostly people having the same issue as me.

Aljutor subscribed.Jul 2 2024, 2:10 PM

I have similar problem, testing setup:
VyOS 1.5-rolling-202407010024 as VM in proxmox 8.2
Intel x710 card in pci-e passthrow to the VM

eth1 is XGSPON SFP+ with VLAN and PPPoE inside
eth2 is DAC to the MikroTik switch

set firewall flowtable forward-table description 'Flow Table for the forward chain'
set firewall flowtable forward-table interface eth1
set firewall flowtable forward-table interface eth2
set firewall flowtable forward-table offload software
vyos@vyos:~$ ethtool -k eth1 | grep hw-tc-offload
hw-tc-offload: on
vyos@vyos:~$ ethtool -k eth2 | grep hw-tc-offload
hw-tc-offload: on

If I enable hw-tc-offload via

set interfaces ethernet eth1 offload hw-tc-offload

I get the message:

WARNING: could not change "eth1" flow control setting!

But using ethtool directly works

Enabling HW offload for flowables result in:

vyos@vyos# set firewall flowtable forward-table offload hardware
[edit]
vyos@vyos# commit

Failed to apply firewall: /run/nftables.conf:21:15-42: Error: Could not
process rule: Operation not supported     flowtable
VYOS_FLOWTABLE_forward-table {
^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /run/nftables.conf:30:48-85: Error: Could
not process rule: No such file or directory         ct state
{established,related} counter flow add @VYOS_FLOWTABLE_forward-table
comment "ipv4-FWD-filter-5"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /run/nftables.conf:104:15-42:
Error: Could not process rule: Operation not supported     flowtable
VYOS_FLOWTABLE_forward-table {
^^^^^^^^^^^^^^^^^^^^^^^^^^^^ /run/nftables.conf:113:48-85: Error: Could
not process rule: No such file or directory         ct state
{established,related} counter flow add @VYOS_FLOWTABLE_forward-table
comment "ipv6-FWD-filter-5"
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

[[firewall]] failed
Commit failed
Viacheslav added a comment.Jul 2 2024, 3:55 PM

Some info

To make hardware offloading works, the NIC should be programmable to load forwarding rules like flowtable and this is what switchdev doing

ref. https://forums.servethehome.com/index.php?threads/nftables-flowtables-hardware-offload-with-connectx-5.43714/#post-431605

Aljutor added a comment.Sat, Aug 10, 5:53 PM

After some research, I am not even sure that flowtables offload works on intel cards

Apachez subscribed.Sat, Aug 10, 10:06 PM

Are the in-tree or out-of-tree drivers for your Intel NICs being used?

That is output of these commands would be nice:

modinfo igb | grep -i intree
modinfo ixgbe | grep -i intree

Also since you run in a VM, make sure that your passthrough is properly setup since hardware flowtable for obvious reasons needs direct access to the hardware itself without some virtualization in between. Software flowtable should work no matter what.

Aljutor added a comment.Mon, Aug 12, 11:51 AM

In my case, one interface is direct passthrough of physical interface via proxmox, other is a passthrough of VF (interface is splitter into several and shared across VMs).
I tried to enable hardware offloading for each of them separately without any success.

vyos@vyos:~$ modinfo igb | grep -i intree
intree:         Y
vyos@vyos:~$ modinfo ixgbe | grep -i intree
vyos@vyos:~$ modinfo i40e | grep -i intree
intree:         Y
vyos@vyos:~$ lspci | grep -E -i --color 'network|ethernet'
01:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GbE SFP+ (rev 02)
02:00.0 Ethernet controller: Intel Corporation Ethernet Virtual Function 700 Series (rev 02)
Viacheslav added a comment.EditedTue, Aug 13, 7:22 AM

Needs to add a bug report to netfilter https://bugzilla.netfilter.org/