Page MenuHomeVyOS Platform
Create Task
Maniphest T9520

VPP: upstream PPPoE traffic is processed by a single worker (NIC RSS cannot hash PPPoE)
Open, NormalPublicBUG
Actions

Description

Summary

With the VPP dataplane and the PPPoE server (accel-ppp, vpp-cp=true), all upstream (subscriber → network) PPPoE traffic is received on NIC queue 0. It is therefore processed by one VPP worker, however many workers and RX queues are configured. Downstream traffic (plain IP) is spread across all workers as expected.

The cause is NIC RSS: the NIC cannot parse inside PPPoE, so it cannot compute a hash and puts every PPPoE frame on one queue. This was observed on:

  • Intel XL710
  • Intel E810-C (validated NIC).

Upstream PPPoE capacity per router is therefore limited to what one CPU core can forward.

Use case

ISP BNG deployments of 10–100 Gbps with many PPPoE subscribers on one VLAN (N:1).

Steps to reproduce

  1. Configure VPP on an E810-C (or X710) interface with multiple workers and RX queues.
  2. Configure service pppoe-server on a VLAN sub-interface of that port.
  3. Establish many PPPoE sessions with distinct client MACs and IPs, and send upstream traffic.
  4. Clear counters, wait 60 s, and check vppctl show runtime: look at pppoe-input per worker.

My result (60 s windows)

Upstream offeredvpp_wk_0 pppoe-input vectorsvectors/callOther 12 workerseth8 rx-miss
25 Gbps (2.23 Mpps)133,962,1633.350 data packets0
50 Gbps (4.46 Mpps)267,924,0819.700 data packets0
90 Gbps (8.04 Mpps)462,382,463255.99 (saturated)0 data packets19,963,357
  • Ceiling: at 90 Gbps, vpp_wk_0 shows vector rates in 7.7045e6. One worker caps upstream PPPoE at about 7.7 Mpps (about 86 Gbps at 1400 B, about 32 Gbps at 512 B).
  • Downstream: in a two-way 45 + 45 Gbps test, downstream (plain IP) was spread over all 13 workers, while upstream PPPoE stayed on vpp_wk_0.

Expected result

Upstream PPPoE is distributed across all configured workers, as plain IP traffic is.

Details

Version
1.5.1
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Feature (new functionality)

Event Timeline

DarkDreamSan created this task.Tue, Oct 27, 1:38 PM
rherold subscribed.Tue, Oct 27, 3:25 PM

I looked at the task and at the VPP source. What I can say without an E810 here:

What the code does today. vnet flow has no PPPoE flow type. The flow types in src/vnet/flow/flow.h are ethernet, IPv4/IPv6 n-tuple, GTP, VXLAN, L2TPv3, ESP/AH and the IP-in-IP types, plus generic. src/plugins/dpdk/device/flow.c has nothing about PPPoE, and the generic type passes one RAW pattern item to rte_flow. So a command like flow add dev <if> proto pppoe rss ... does not exist in VPP.

On the suggestions in the discussion. The idea that DDP lets the E810/XL710 parse PPPoE and do RSS on the inner IP header matches the DPDK ice test plans, but I did not verify it here. I did not find the devargs proto_dissect=1,rss=pppoe string in the ice documentation I know and I would not use it without a source. The AF_XDP route would change the dataplane for these NICs, I would keep it as a last option.

Directions that look possible.

  1. A PPPoE flow type in the DPDK flow code: eth / [vlan] / pppoes / ipv4|ipv6 / [udp|tcp] with an RSS action, for NICs whose PMD accepts that pattern with the comms DDP package loaded. This is an upstream VPP change and needs E810 hardware to test, I have none.
  2. A software handoff in the PPPoE plugin: the worker that gets the frame on queue 0 only parses the PPPoE session and hands the packet to a worker chosen by session id or the inner address. It is NIC independent, but worker 0 still sees every upstream packet, so the gain is smaller than with a working RSS.

Questions for the reporter (please post text, no core files):

  • vppctl show version, vppctl show dpdk version
  • dmesg | grep -i -e ddp -e "ice " (is the comms package loaded, which one)
  • vppctl show hardware-interfaces <if> (RSS fields and queue count)
  • Does testpmd with a rule eth / pppoes / ipv4 and an RSS action spread the traffic on the same NIC? That tells whether the hardware side works before VPP is involved.

I can prototype option 2 on a multi-worker lab VM, but I cannot measure the 100 Gbit case.

Viacheslav triaged this task as Normal priority.Tue, Oct 27, 3:34 PM
pasik subscribed.Tue, Oct 27, 4:33 PM