Page MenuHomeVyOS Platform
Create Task
Maniphest T9484

VPP netlink: receive buffer is silently capped at 4 MiB by 'net.core.rmem_max'; the configured 256 MiB is not applied
Closed, ResolvedPublicBUG
Actions

Description

VyOS asks VPP for a 256 MiB netlink receive buffer, but its own sysctl default caps it, so VPP gets 4 MiB. With a full table routed over a VPP interface, a bond rebuild overflows that buffer and linux-cp falls back to a resync, which can leave VLANs admin-down without their IPv6 link-local (T9481).

Details

  • data/templates/vpp/startup.conf.j2 hardcodes nl-rx-buffer-size 268435456 for linux-nl (and linux-xfrm-nl).
  • src/etc/sysctl.d/30-vyos-router.conf sets net.core.rmem_max=2097152. The kernel caps SO_RCVBUF at it, so VPP's socket gets 4194304 (ss -f netlink -m -p). No warning.
  • interface-definitions/include/vpp/netlink.xml.i defines netlink { rx-buffer-size, batch-size, batch-delay-ms }, but nothing includes it: no CLI, and the template does not read it.
  • set system sysctl parameter net.core.rmem_max is applied after VPP opens its socket at boot, so VPP keeps 4 MiB. Only /config/scripts/vyos-preconfig-bootup.script worked (VPP then got 536870912).

Measured (rolling, full table over a bond VLAN, bond rebuilt by a hash-policy change)

  • rmem_max default: one overflow per trial, a VIF stuck in 3 of 10 trials.
  • rmem_max 268435456 (preconfig script): no overflow, no stuck VIF in 5 trials.

Suggested change

  • Raise net.core.rmem_max in 30-vyos-router.conf to at least 268435456.
  • Wire netlink.xml.i into the VPP settings and template (checked against rmem_max)
  • Warn when VPP's effective netlink buffer is smaller than requested.

Details

Version
1.5.1, 1.5-rolling-20261018
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Related Objects
Search...

Event Timeline

a.kudientsov created this task.Sun, Oct 18, 4:23 PM
a.kudientsov updated the task description. (Show Details)Sun, Oct 18, 4:25 PM
a.kudientsov added a parent task: T7070: VPP related bugs the root task.Sun, Oct 18, 4:31 PM
pasik subscribed.Sun, Oct 18, 5:13 PM
rherold subscribed.Mon, Oct 19, 8:44 AM

I checked this against current rolling and the description matches what I see in the sources:

  • data/templates/vpp/startup.conf.j2 sets netlink_rx_buffers to 268435456 and uses it for linux-nl and linux-xfrm-nl.
  • src/etc/sysctl.d/30-vyos-router.conf sets net.core.rmem_max=2097152, so the 256 MiB request is capped to 4 MiB without a message.
  • interface-definitions/include/vpp/netlink.xml.i is not included anywhere, so the options in it have no effect.

I did not reproduce the overflow myself.

Before I send a patch, which direction do the maintainers prefer? The options I see:

  1. Raise the default net.core.rmem_max in 30-vyos-router.conf to at least 268435456. It is only a cap, a socket allocates the memory only if it asks for it, but it applies system-wide. It is also in place before VPP starts at boot, unlike set system sysctl.
  2. Raise it only when VPP is enabled, from the VPP conf-mode script before VPP is started. The change is limited to VPP routers, but the script has to run before VPP opens its netlink socket.
  3. Add a warning when the effective netlink buffer is smaller than the requested one, whichever of the above is used.
  4. Wire netlink.xml.i (rx-buffer-size, batch-size, batch-delay-ms) into the VPP CLI and template and check it against rmem_max. I would keep that as a separate task.

    My suggestion is 1 or 2 plus 3. I am happy to implement and test whichever is chosen.
rherold mentioned this in T9481: VPP linux-cp: a VLAN that comes up during the netlink overflow resync is pushed back to admin-down.Mon, Oct 19, 8:50 AM
a.kudientsov added a comment.Mon, Oct 19, 5:27 PM

https://github.com/vyos/vyos-1x/pull/5583

Viacheslav closed this task as Resolved.Thu, Oct 22, 4:35 PM
Viacheslav claimed this task.
Viacheslav reassigned this task from Viacheslav to a.kudientsov.
Viacheslav moved this task from Need Triage to Completed on the VyOS Rolling board.
Viacheslav subscribed.