The only requirement I have for this change is that the MTU versus performance conflict should be resolved completely on our side and be transparent to users. Maybe by tuning buffers data-size so they fit Jumbo frames when needed, if this solves the problem.

It seems that this change occurs behind the scenes as in not a configurable option. Being dealt with by adjusting buffers so jumboframes are still properly supported (even if DPDK documentation claims this is NOT compatible with jumbo frames!?).

The numbers in the original post looks great (increase from 84MPPS to 140MPPS) - but are there no drawbacks with this?

How come its not on by default to begin with in DPDK/VPP?

I found for example this (perhaps below is just local to the netgate implementation of DPDK for their product?):

https://docs.netgate.com/tnsr/en/latest/advanced/dataplane-dpdk.html

dataplane dpdk no-multi-seg

Disables multi-segment buffers for network devices. Can improve performance, but disables jumbo MTU capability.

Warning:

This option is not currently compatible with Intel X552 10G network interfaces.
When enabled on incompatible hardware this option can lead to instability such as dataplane crashes while under load.

I am not a huge expert in VPP, nor DPDK, but I think the answer is simple enough to be done without deep expertize.

no-multi-seg is not a binary “faster/slower” switch that can be turned on/off and hardcoded.

Example:

• Not all NICs work with multi-seg: https://doc.dpdk.org/guides/nics/bnx2x.html, https://doc.dpdk.org/guides/nics/vmxnet3.html
• Number of segments has hardware limitation counterparts: https://doc.dpdk.org/guides/nics/i40e.html, https://doc.dpdk.org/guides/nics/ice.html, https://doc.dpdk.org/guides/nics/intel_vf.html
• Some operations are limited to one segment per operation (performance impact): https://doc.dpdk.org/guides/cryptodevs/aesni_gcm.html

From a VPP perspective, if multi-segment usage is disabled, the actual consumable packet size depends on (what is less):

• buffer size
• hardware maximum frame size support

Check: https://github.com/FDio/vpp/blob/a8fe31c6577381fa95714e3f485661e8004b6d50/src/plugins/dpdk/device/common.c#L225

This value is controlled via the configuration file, with a default value of 2048:

• https://github.com/FDio/vpp/blob/a8fe31c6577381fa95714e3f485661e8004b6d50/src/vlib/buffer.h#L20
• https://github.com/FDio/vpp/blob/a8fe31c6577381fa95714e3f485661e8004b6d50/src/vlib/buffer.c#L805
• https://github.com/FDio/vpp/blob/a8fe31c6577381fa95714e3f485661e8004b6d50/src/vlib/buffer.c#L1012

Therefore, you can restore Jumbo frames even without multi-seg by setting big enough:

vpp settings resource-allocation buffers data-size

However, this would have a significant memory consumption penalty - basically multiplying RAM needed for buffers by the Jumbo-capable-size/2048 value. That is one of the biggest reasons why no one did this as the default setting.

Therefore, we must:

• not hardcode no-multi-seg (risk of losing Jumbo frames)
• not hardcode buffers default data-size (too big memory penalty)
• not allow MTU > 2026 (I hope I calculated it properly, taking into account overheads - need to recheck for the actual code) if no-multi-seg is set without buffers default data-size tuning accordingly

As you see, there are too many places to make a mistake that can very easily lead to incapability to reinitialize the NIC (lack of buffers), memory-related crashes, and traffic-blocking configurations. And that is the main reason why I do not want the feature to be user-controllable - they will definitely make mistakes or get stuck with a lowish-performance mode, where they could get a “free” speed bonus.

What I want is to consider during the configuration process:

• vpp settings resource-allocation buffers data-size
• interfaces ethernet ethX mtu

Then:

• If MTU is more than buffers data-size - do not set no-multi-seg. If MTU is lower or equal to buffers data-size - set no-multi-seg
• If MTU on VPP-enabled interfaces is updated, check buffers data-size. If it cannot fit the new MTU - reconfigure VPP without no-multi-seg (same memory usage, no/low risk for a crash)
• If buffers data-size is updated, check MTU on VPP-enabled interfaces. If any of them has it too big, remove no-multi-seg. If not - set no-multi-seg.

To me, it sounds safe enough, possible to be implemented via dependencies, and compatible with LTS (no CLI items added, no incompatible configs possible).

But, as I mentioned, I’m not an expert. So there may be something else I'm missing.

https://github.com/vyos/vyos-1x/pull/5376