Page MenuHomeVyOS Platform

VPP: upstream PPPoE traffic is processed by a single worker (NIC RSS cannot hash PPPoE)
In progress, NormalPublicBUG

Description

Summary

With the VPP dataplane and the PPPoE server (accel-ppp, vpp-cp=true), all upstream (subscriber → network) PPPoE traffic is received on NIC queue 0. It is therefore processed by one VPP worker, however many workers and RX queues are configured. Downstream traffic (plain IP) is spread across all workers as expected.

The cause is NIC RSS: the NIC cannot parse inside PPPoE, so it cannot compute a hash and puts every PPPoE frame on one queue. This was observed on:

  • Intel XL710
  • Intel E810-C (validated NIC).

Upstream PPPoE capacity per router is therefore limited to what one CPU core can forward.

Use case

ISP BNG deployments of 10–100 Gbps with many PPPoE subscribers on one VLAN (N:1).

Steps to reproduce

  1. Configure VPP on an E810-C (or X710) interface with multiple workers and RX queues.
  2. Configure service pppoe-server on a VLAN sub-interface of that port.
  3. Establish many PPPoE sessions with distinct client MACs and IPs, and send upstream traffic.
  4. Clear counters, wait 60 s, and check vppctl show runtime: look at pppoe-input per worker.

My result (60 s windows)

Upstream offeredvpp_wk_0 pppoe-input vectorsvectors/callOther 12 workerseth8 rx-miss
25 Gbps (2.23 Mpps)133,962,1633.350 data packets0
50 Gbps (4.46 Mpps)267,924,0819.700 data packets0
90 Gbps (8.04 Mpps)462,382,463255.99 (saturated)0 data packets19,963,357
  • Ceiling: at 90 Gbps, vpp_wk_0 shows vector rates in 7.7045e6. One worker caps upstream PPPoE at about 7.7 Mpps (about 86 Gbps at 1400 B, about 32 Gbps at 512 B).
  • Downstream: in a two-way 45 + 45 Gbps test, downstream (plain IP) was spread over all 13 workers, while upstream PPPoE stayed on vpp_wk_0.

Expected result

Upstream PPPoE is distributed across all configured workers, as plain IP traffic is.

Details

Version
1.5.1
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Feature (new functionality)

Event Timeline

I looked at the task and at the VPP source. What I can say without an E810 here:

What the code does today. vnet flow has no PPPoE flow type. The flow types in src/vnet/flow/flow.h are ethernet, IPv4/IPv6 n-tuple, GTP, VXLAN, L2TPv3, ESP/AH and the IP-in-IP types, plus generic. src/plugins/dpdk/device/flow.c has nothing about PPPoE, and the generic type passes one RAW pattern item to rte_flow. So a command like flow add dev <if> proto pppoe rss ... does not exist in VPP.

On the suggestions in the discussion. The idea that DDP lets the E810/XL710 parse PPPoE and do RSS on the inner IP header matches the DPDK ice test plans, but I did not verify it here. I did not find the devargs proto_dissect=1,rss=pppoe string in the ice documentation I know and I would not use it without a source. The AF_XDP route would change the dataplane for these NICs, I would keep it as a last option.

Directions that look possible.

  1. A PPPoE flow type in the DPDK flow code: eth / [vlan] / pppoes / ipv4|ipv6 / [udp|tcp] with an RSS action, for NICs whose PMD accepts that pattern with the comms DDP package loaded. This is an upstream VPP change and needs E810 hardware to test, I have none.
  2. A software handoff in the PPPoE plugin: the worker that gets the frame on queue 0 only parses the PPPoE session and hands the packet to a worker chosen by session id or the inner address. It is NIC independent, but worker 0 still sees every upstream packet, so the gain is smaller than with a working RSS.

Questions for the reporter (please post text, no core files):

  • vppctl show version, vppctl show dpdk version
  • dmesg | grep -i -e ddp -e "ice " (is the comms package loaded, which one)
  • vppctl show hardware-interfaces <if> (RSS fields and queue count)
  • Does testpmd with a rule eth / pppoes / ipv4 and an RSS action spread the traffic on the same NIC? That tells whether the hardware side works before VPP is involved.

I can prototype option 2 on a multi-worker lab VM, but I cannot measure the 100 Gbit case.

Viacheslav triaged this task as Normal priority.Tue, Oct 27, 3:34 PM

Hi Ruben,

Versions

vppctl show version:       v25.10.0-55~vyos20260409073441.fe7767a18
vppctl show dpdk version:  DPDK 25.07.0

DDP package (stock VyOS, OS Default, no Comms)

dmesg: ice 0000:6f:00.0: The DDP package was successfully loaded: ICE OS Default Package version 1.3.43.0
vppctl show log: ICE_INIT: ice_load_pkg_type(): Active package is: 1.3.43.0, ICE OS Default Package (double VLAN mode)
/lib/firmware/intel/ice/ddp/: ice-1.3.43.0.pkg, ice.pkg -> ice-1.3.43.0.pkg   (no other package present)

vppctl show hardware-interfaces detail eth8 (no traffic)

Link speed: 100 Gbps
Intel E810 Family
rx: queues 13 (max 256), desc 4096 (min 64 max 8160 align 32)
tx: queues 14 (max 256), desc 4096 (min 64 max 8160 align 32)
pci: device 8086:1592 subsystem 8086:0014 address 0000:6f:00.00 numa 0
vlan offload: strip off filter off qinq off
rx offload active: ipv4-cksum scatter
rss avail:  ipv4-frag ipv4-tcp ipv4-udp ipv4-sctp ipv4-other ipv4
            ipv6-frag ipv6-tcp ipv6-udp ipv6-sctp ipv6-other ipv6 l2-payload
rss active: ipv4-tcp ipv4-udp ipv4 ipv6-tcp ipv6-udp ipv6
rx burst mode: Offload Vector AVX512 Scattered

13 RX queues map one-to-one to vpp_wk_0–vpp_wk_12.

Traffic shape (for a flow pattern or a handoff hash)

  • Upstream frames: eth / vlan (525) / pppoes / ipv4 / udp, 1426 bytes on the wire (1400-byte IP packet).
  • One VLAN for everyone (N:1): 1000 sessions, each with its own client MAC, PPPoE session ID, and source IPv4 address.
  • Same destination IP and UDP ports on every flow: so the variation is only in session ID, client MAC, and inner source IP.

testpmd with eth / pppoes / ipv4 + RSS: not tested.
VyOS doesn’t ship dpdk-testpmd, and it would need VPP stopped and a separate DPDK build. That puts it outside the stock setup. With the OS Default package loaded, I’d expect the rule to be refused or to have no effect.

Also, reproducing doesn’t require 100G, as even at a few Gbps, you'll see every upstream PPPoE packet land on vpp_wk_0. A simple VM should be enough to develop and test a handoff prototype, as long as its virtual NIC forwards plain IP packets across queues.

Lastly, on the topic of directions. I considered two approaches:

  • Either ship or let the user select the Intel E810 Comms DDP package, so the hardware can parse PPPoE and hash the inner IP header.
  • Or by software, a worker handoff keyed on session ID or inner source IP. This should improve performance, but the core issue will persist.

Thanks for the details. The loaded package is the OS Default package, and rss active lists only the IP and TCP/UDP types, so nothing on the NIC hashes on the PPPoE header. That fits what you see: all upstream PPPoE ends up on queue 0, and the plain IP downstream is spread.

The one test that tells us whether the NIC can do it at all is the testpmd rule. Could you run it on the E810, without VPP bound to the port:

flow create 0 ingress pattern eth / vlan / pppoes / ipv4 / end actions rss types ipv4 end queues 0 1 2 3 end / end

and send the 1000-session traffic. If the rule is refused, the OS Default package cannot do it and a Comms package would be needed. If it is accepted and the traffic spreads, we know the hardware side works and what is missing is the way VPP sets up the flow. The same test on the XL710 would show whether it is the same there.

As for the software handoff: it would work on any NIC, but the first worker still receives and reads every packet before it hands it over, so it moves the load and does not remove it. I would only look at it after the testpmd result. I can try it in a lab VM to show that the sessions spread over the workers, but I cannot measure it at 100 Gbit.

correction to my last comment, after reading the DPDK test plan for the ice driver (ice_advanced_rss_pppoe).

The rule I gave has a vlan item in the pattern. The feature file of DPDK 25.07 lists vlan for ice only as partial, and the test plan uses patterns without a VLAN. So if the rule is refused, that does not yet show that the OS Default package is the cause. The test plan also says that only the Comms DDP package is expected for these tests.

Please try these rules in testpmd (taken from the test plan, with queues added), on the E810 without VPP bound to the port:

flow create 0 ingress pattern eth / pppoes / end actions rss types pppoe end key_len 0 queues 0 1 2 3 end / end
flow create 0 ingress pattern eth / pppoes / ipv4 / end actions rss types ipv4 end key_len 0 queues 0 1 2 3 end / end
flow create 0 ingress pattern eth / vlan / pppoes / ipv4 / end actions rss types ipv4 end key_len 0 queues 0 1 2 3 end / end

The first two with traffic that has no VLAN tag, the third with your tagged traffic. A note on the result of each (accepted, refused, or accepted but not spread) tells us whether the package, the VLAN or the NIC is the limit. Separately, VPP does not ask for PPPoE hashing today: its list of RSS types for the dpdk startup option has no PPPoE entry and vnet flow has no PPPoE flow type, so a working rule in testpmd would still need VPP changes.

As mentioned, VyOS does not ship with testpmd.
This falls outside the scope of what VyOS distributes; hence, I'd prefer VyOS developers' input on what they'd like to do before proceeding.
I'd also like to iterate that the issue is not NIC-specific; VPP never asks the NIC for PPP hashing, hence the issue. I tested with a ConnectX-6 Dx and various other NICs and saw the same thing.

Thanks for the extra data, and the ConnectX-6 Dx result matters: in the DPDK 25.07 feature files mlx5 lists no PPPoE flow item, only the Intel ice driver does. So there is no common NIC feature to rely on, and a solution that spreads the upstream PPPoE traffic in software is the general answer. NIC hashing (E810 with the Comms package) would be an optional addition for one vendor, and VPP would need changes for it as well (no PPPoE entry in its RSS type list, no PPPoE flow type in vnet flow).

So testpmd is no longer the deciding test, and it is not part of the VyOS image. If the E810 is in a lab where you can run it anyway, the rules from my last comment would still tell us whether hashing on the PPPoE session ID with the Comms package is worth adding later, but it is optional. I am not a VyOS maintainer and cannot decide the direction. I will bring the question to the maintainers: software distribution in or before pppoe-input (keyed on the session ID or the client MAC) as the general path, NIC hashing as a later option.

A measurement that would help whichever way this goes: the number of VPP workers and RX queues you ran with, and the rate at which vpp_wk_0 saturates with a handful of sessions (you wrote that a few Gbps in a VM reproduce it). That would let us test a software distribution in a lab without the 100G hardware.

Some data from a VPP PPPoE BNG lab that may help here (VPP 26.10 with the VyOS PPPoE patches, accel-ppp-ng, BNG Blaster and TRex as testers, 10k-64k sessions, single- and double-tagged).

1. E810 with the comms DDP package: the NIC spreads PPPoE itself. With the OS default package the E810 does not parse PPPoE, as you found. With ice_comms-1.3.63.0.pkg loaded, the ice PMD hashes PPPoE session frames by the inner IPv4 header whenever ipv4 RSS is requested, which is VPP's default, so upstream PPPoE is spread over all RX queues with no worker handoff and no VPP change. To load it for one card only, use the device-specific file name and reset the card (or reboot), so the new package replaces the one already on it:

cp ice_comms-1.3.63.0.pkg /lib/firmware/updates/intel/ice/ddp/
ln -s ice_comms-1.3.63.0.pkg /lib/firmware/updates/intel/ice/ddp/ice-<serial>.pkg   # serial: dmesg | grep ice-

VPP has to see the kernel interface down, or it skips the device. Subscribers on one E810 port (10k QinQ sessions, 46-byte IP upstream, no handoff, 12 workers, RX/TX rings of 4096):

UpstreamLoss
6 Mpps0.003%
10 Mpps0.001%
14 Mpps0.019%
18 Mpps0.055%
5 down / 4 up Mbit/s per session, 1400 B (93 Gbps)0.0% / 0.0%

At 18 Mpps each worker takes ~1.5 Mpps. Rings of 1024 lose ~0.1%, so use 4096 descriptors. We tested this with the DPDK of VPP 26.10; with VyOS' DPDK 25.07 it is worth a try, the ice PMD has had PPPoE RSS for a long time, but we have not run it there.

2. NICs without PPPoE RSS (ConnectX, XL710 without a PPPoE DDP profile): VPP's worker handoff plus a small hash change. VPP already has the software distribution that was suggested above: set interface handoff <if> workers <list>. It does not help PPPoE as it is, because handoff-eth only builds a key for IPv4, IPv6 and MPLS; every PPPoE frame gets the ethertype as key and all subscribers land on the same worker again. Hashing PPPoE session and discovery frames by the client MAC fixes that, and keeps each subscriber on one worker. With two ConnectX-5 ports (all PPPoE on RX queue 0 of each port), handoff on both ports and each queue 0 on its own worker, 46-byte upstream reached 12 Mpps with 0.14% loss, pppoe-input running on all 12 workers. The receiving worker still does RX and the handoff of all PPPoE of its port, so this caps a port at roughly 6-8 Mpps; it is a fallback, the NIC hash goes further.

On a bond, worker-handoff runs on the members but looks up its config by the RX interface, which bond-input has already changed to the bond; so it needs the config on the bond (enable it on the bond first, then on the members). With the members alone VPP crashed (SIGSEGV at PC 0, a NULL hash function). 25.10 has the same code.

Both changes apply to stable/2510 with the VyOS patch series and were checked on a 25.10 build (packet generator: all PPPoE on one worker before, spread by client MAC after; the bond-member crash reproduced before and gone after). PR: https://github.com/vyos/vyos-vpp-patches/pull/62

Thanks @najieb_haswell for the measurements, they answer the question about the NIC side.

Summary for the E810 in this task: with the OS Default package the NIC does not parse PPPoE, as DarkDreamSan saw. With the Comms package (ice_comms-1.3.63.0.pkg) the ice PMD hashes the PPPoE session frames by the inner IPv4 header whenever ipv4 RSS is requested, and that is the VPP default. Upstream PPPoE then spreads over all RX queues without a handoff and without a VPP change. najieb_haswell measured 18 Mpps with 0.055 % loss on 12 workers and 93 Gbit/s without loss, with 4096 descriptors (1024 lost about 0.1 %).

One point is open: this was tested with the DPDK of VPP 26.10. VyOS rolling and 1.5 use DPDK 25.07. @DarkDreamSan, you have the E810 and the 1000-session traffic: could you try it on your VyOS 1.5.1 box?

  1. Put the Comms package in place for that card only, with the device specific file name, and reset the card or reboot (the steps are in the comment above). VPP needs the kernel interface down, otherwise it skips the device.
  2. Keep the RSS types at the VPP default and set 4096 RX and TX descriptors.
  3. Report vppctl show hardware-interfaces detail <if> (the rss active line) and vppctl show runtime after 60 s with traffic: is pppoe-input on all workers, or still on vpp_wk_0?

For NICs that cannot hash PPPoE (ConnectX, XL710 without a PPPoE profile) the software handoff in vyos-vpp-patches PR #62 is the way. Whether VyOS ships or offers the Comms package is for the maintainers, I cannot say anything about the licence or the image.

Viacheslav changed the task status from Open to In progress.Thu, Nov 19, 4:15 PM
Viacheslav assigned this task to najieb_haswell.

Thanks for assigning it. The fix is https://github.com/vyos/vyos-vpp-patches/pull/62 (now patches 0038 and 0039 on top of the current rolling): handoff-eth keys PPPoE frames by client MAC, so set interface handoff spreads PPPoE subscribers over the workers on NICs whose RSS cannot hash PPPoE, and handoff on bond members without a config on the bond no longer crashes VPP. I updated it for the review: with symmetrical handoff the key is now the same for both directions.

Tested on VyOS rolling with VyOS' own package build, without and with the two patches, 1000 PPPoE sessions through accel-ppp: on x86 (E810, whose RSS does not hash PPPoE in this setup, 12 workers) all pppoe-input was on one worker without handoff and with the unpatched handoff, and 7-10 % on each of the 12 workers with the patches; on arm64 (OCTEON 10, 5 workers) the unpatched handoff put everything on one worker, the patched one 18-22 % on each. Plain and symmetrical handoff behave the same. Details in the PR description.