Page MenuHomeVyOS Platform

VPP PPPoE with RADIUS Idle-Timeout: sessions drop every 60 s, accounting reports 0 bytes used
In progress, NormalPublicBUG

Description

With the PPPoE server on a VPP-owned interface and RADIUS returning an Idle-Timeout attribute, every subscriber session is torn down 60 s after it comes up and immediately redials. RADIUS records the disconnect as Acct-Terminate-Cause NAS-Error. Session traffic counters never move: "show pppoe-server sessions" shows 0 B / 0 B and every accounting record reports 0 bytes used, even while traffic is forwarding normally. Both symptoms have the same cause. PPPoE on the kernel dataplane is unaffected.

Steps to reproduce:

1. set vpp settings interface eth1             
2. set service pppoe-server interface eth1      (+ client-ip-pool, gateway-address)
3. set service pppoe-server authentication mode radius
   set service pppoe-server authentication radius server <ip> key <secret>
4. Have RADIUS return Idle-Timeout = 600 in the Access-Accept
5. Connect a subscriber and watch the session

Result:
The session dies at 63 s and redials, indefinitely:

<Acct-Status-Type Stop> <Acct-Session-Time 63> <Acct-Input-Octets 0> <Acct-Output-Octets 0>
<Acct-Terminate-Cause NAS-Error>

No "idle timeout" is logged, the teardown is a silent failure path. Without VPP the session runs past 180 s, no NAS-Error stops.

Root cause:
A VPP-terminated session has no kernel interface, so ses->ifindex keeps its init value -1 (session.c:80): vpphooks.c sets vpp_sw_if_index but never ses->ifindex, and ifcfg.c:103 skips the ifcfg routine for hook-owned sessions.

  • ap_session_read_stats() returns -1 at session.c:420 ("if (ses->ifindex == -1) return -1;") silently, with no log line, so counters never update and radius/acct.c:59 uses the same call, hence 0 bytes in accounting.
  • ap_session_timer() treats that failure as fatal (session.c:145-147): if (ap_session_read_stats(ses, NULL)) ap_session_terminate(ses, TERM_NAS_ERROR, 0); RADIUS Idle-Timeout arms this timer, which ticks every 60 s (timer.period = 60000).

Details

Version
2026.08.13-0024-rolling
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Event Timeline

Viacheslav changed the task status from Open to In progress.Sep 10 2026, 1:43 PM
Viacheslav assigned this task to rherold.
Viacheslav triaged this task as Normal priority.

I was testing vyos as BNG for my network especially with vpp. But I faced same issue. I hope issue will be resolved soon

@saadinnocent909 can you test my PR in your environment?

can you share me patched iso or lates rolling release includes the PR ?

@saadinnocent909 there is no ISO or rolling image with the fix, the PR (https://github.com/accel-ppp/accel-ppp-ng/pull/42) is still in review. Here is a test build of the three changed files that you can put on an existing rolling system:

https://cloud.insecure.pw/s/GwgfJ2YWdDT3N7a
sha256: 480ac868d80b7e1dcb0cca9eb3c50a66d3bbd050627fd8f43ebd6965eabe1798

What it is: accel-ppp-ng at the commit that rolling uses (dd07b23) plus the four commits of the PR. It contains accel-pppd, libradius.so and libvyos_vpp.so.

Requirements: a rolling image with VPP 25.10.0 (it links against these libraries). I built and loaded it on 2026.10.16-0024-rolling. A rolling image that uses another VPP version or another accel-ppp-ng commit can fail to load it. Check with show version and dpkg -l | grep libvppinfra.

Install, as root:

tar xzf accel-ppp-ng-T9211.tar.gz
cd accel-ppp-ng-T9211
./install.sh

The script keeps the original files in /root/accel-ppp-ng.orig and restarts accel-ppp@pppoe. Established PPPoE sessions are dropped by the restart. ./restore.sh puts the originals back. A reboot or an image update also replaces the files again.

What to check: with the RADIUS Idle-Timeout, sessions should no longer drop every 60 seconds and the accounting should report the real byte counters. Please post what you see, with the Idle-Timeout value and the output of show pppoe-server sessions.

What I tested: the build loads, accel-ppp@pppoe starts with vyos_vpp and the install and restore scripts work. I did not run a PPPoE client session over VPP with Idle-Timeout, so your test is the first real check of the fix.

Tomorrow I will give it a try and update you.

I did some testing with patch. Counters started working and I started to
get usage in freeradius interim accounting updates. But I also noticed
another issue if I kick a user using reset command all users are
disconnected. Tomorrow I will share stats as well and will do some more
testing. Idle timeout and session timout I will check these as well
tomorrow

@saadinnocent909 thank you for testing, good to hear that the counters work now and that the usage shows up in the interim updates.

For the reset: I need a few details to tell if the build causes it, because reset pppoe-server ends sessions in three different ways, all through accel-cmd:

  • reset pppoe-server username <name> ends the sessions of that user
  • reset pppoe-server interface <interface> ends every session on that interface, which is all users if they share one
  • reset pppoe-server all ends every session

Which of the three did you run, with which name or interface? Please also post:

  • the output of show pppoe-server sessions before and after
  • the accel-ppp log of the moment (journalctl -u accel-ppp@pppoe --since "5 minutes ago" right after the reset)
  • if you can, the same reset on the original files: run ./restore.sh from the test build, open the same sessions again, run the same reset. Then we know if it is the same without the change.

The test build only touches how the byte counters of a VPP session are read and the session interface lookup. I do not see how that ends other users, but I did not test a reset with several users. I will try to reproduce it.

For Idle-Timeout and Session-Timeout, the RADIUS value and the accounting packets (Acct-Session-Time, the octet counters and Acct-Terminate-Cause) are the most useful. Thanks!

I do understand that. I will update you today, after thorough testing Insha
Allah

@saadinnocent909 I could reproduce the reset problem, and it is not caused by the test build. It is a crash that is already in the current rolling image.

What happens: ending a VPP PPPoE session makes accel-pppd abort, systemd restarts it, and every other session is gone with it. I reproduced it with three sessions, both with reset pppoe-server username and with a disconnect on the client side, on the unchanged files and with the test build. Both abort with status=6/ABRT. An idle or session timeout ends a session through the same code, I did not run those two. strace shows why: the session socket is closed first and then removed from the epoll set (epoll_ctl(DEL) returns EBADF), which makes the daemon call abort(). The cause is an upstream change from July (T9238) that closes the descriptor too early for sessions of the VPP dataplane.

This most likely affects the tests you plan as well: an Idle-Timeout or Session-Timeout that ends one session should drop all users, whatever the byte counters say.

I have a fix (https://vyos.dev/T9499, accel-ppp-ng PR https://github.com/accel-ppp/accel-ppp-ng/pull/44) and a second test build with it (the first test build plus the fix):

https://cloud.insecure.pw/s/dnAoaHZMqswMNzW
sha256: 61ea1a2cad61bcabcc724baba4caa13793a2f2909562ca2dd252fada92965bfa

Install and restore are the same as before (./install.sh, ./restore.sh after unpacking, originals in /root/accel-ppp-ng.orig, same requirements: rolling with VPP 25.10.0, established sessions drop at the restart). If you still have the first build installed, run ./restore.sh of that package first, or just run ./install.sh of the new one, it keeps the first originals.

With the build, three sessions on my VM: terminate username ends only that user, terminate all ends all and the daemon stays up, a client side disconnect leaves the others up. Please repeat the reset and the timeout tests with it. The accounting packets (Acct-Session-Time, octets, Acct-Terminate-Cause) are the most useful, and whether the other users stay connected.

Ok I will test these today. And will share all my findings with you

Thanks @ruben-herold, I tested the package accel-ppp-ng-T9211-v2 (#42 + #44).
▎
▎ Setup: fresh VyOS 2026.10.16-0024-rolling VM, VPP 25.10.0 (PPPoE on eth0, NAT44 removed for this test), FreeRADIUS 3.2.5 + PostgreSQL, MikroTik CHR clients: 2 real CPEs + up to ~1,700 PPPoE clients on macvlans. Idle-Timeout = 120 in radgroupreply for the group of the 2 CPEs only. No interim interval configured.
▎
▎ 1. Install: the binaries have no RUNPATH, accel-pppd exits with libtriton.so: cannot open shared object file (libluasupp.so also not found for libradius.so). The stock binary has RUNPATH /usr/lib64/accel-ppp. With a drop-in Environment=LD_LIBRARY_PATH=/usr/lib64/accel-ppp it starts.
▎
▎ 2. #44 works. reset pppoe-server username usman with the other sessions up: same accel-pppd PID before and after, no EBADF/ABRT in the journal. Also reset pppoe-server all with ~1,500 sessions: the daemon stayed up (NRestarts=0). On stock the first teardown aborts the daemon (backtrace triton_md_disable_handler <- destablish_ppp, same as your strace).
▎
▎ 3. Idle-Timeout (#42) — not fixed yet in my test. radacct for the CPE user usman:
▎ - stock: 7 of 9 sessions ended after exactly 60 s with Acct-Terminate-Cause = NAS-Error, all with 0 bytes.
▎ - with the package: rx bytes are counted now (e.g. 477 B), but sessions still end with NAS-Error: 4 sessions after 60 s or 120 s while VPP was up (two more ended around the VPP restart in point 5). None ended with Idle-Timeout.
▎
▎ 4. Counters drop to 0 under load. When ~1,000 sessions connected within a minute (19:37-19:38 local), the counters of the two CPE sessions went from 1.4 KiB / 477 B to 0 B, and usman ended with NAS-Error at the next timer tick (19:38:48). I think this is the heartbeat check: /sys/heartbeat in the stats segment changes about every 10 s, so the 3 s / 10 s "VPP instance lost" limits trigger while VPP is busy, the stats read returns an error and the session is ended. In my own build of dd07b23 + #42 + #44 I raised those limits to 35 s; with that, two CPE sessions stayed up 19 h with ~2,000 sessions online and the counters stayed correct.
▎
▎ 5. VPP worker crashes (not caused by #44, but #44 makes this path run).
▎ - 13:44:52 UTC, ~1,500 sessions, sessions being torn down: received signal SIGBUS in policer_input_node_fn_hsw, vpp-failure-handler restarted VPP, all sessions dropped.
▎ - 14:41:22 UTC VPP restarted again with ~1,700 sessions and no config change: [VPP 14:41].
▎
▎ My guess for the policer crash: vpppolicer_remove_limiter() first calls policer_input(apply=0) — VPP sets policer_index_by_sw_if_index to ~0 and only disables the device-input arc — then deletes the policer, and only after that disables the policer-input features on the ip4/ip6 unicast/multicast arcs (added by vyos-vpp-patches 0018). A packet in that window reads pm->policers[~0]. Disabling those four features before vpppolicer_set_input(0, ...) should close it. CoA reaches the same code through install_limiter(). Before #44 this path was never reached, because the daemon aborted first.

▎ 6. Minor: I saw two active sessions shown with the same ifname (pppoe_session162 for saad and usman at the same time, later pppoe_session158 reused). I did not look into it
▎ further.
▎
▎ I can test a new package any time; happy to run the same matrix again.

@saadinnocent909 thank you for the detailed test, this is very useful. Going through your points:

  • Reset (T9499 / accel-ppp-ng #44): good to know it holds with 1,500 sessions, one user ends alone and reset all keeps the daemon up.
  • libtriton.so not found: the binaries of the first two builds had no library path of their own and relied on an ld.so entry of /usr/lib64/accel-ppp. The new build sets the path in the binary, I checked it with the ld.so entry removed. The installer also keeps the original files in /config/accel-ppp-ng.orig now, /root was gone after a reboot.
  • Counters and NAS-Error under load: your suspicion is right and it is a mistake of my change. The stats code decided that VPP had been restarted when the heartbeat did not move for 3 s (10 s for old sessions), or when the dump failed for 10 s. A VPP that is busy creating interfaces can do both without any restart, and then every session of the "old instance" lost its counters and was ended with NAS-Error at the next 60 s tick. I could reproduce it without load: with VPP frozen for 15 s (kill -STOP and -CONT, no restart) the old build logged "stats segment lost" and ended all three sessions with Acct-Terminate-Cause 9 (NAS-Error). The same test with the new build keeps the sessions and logs nothing. A restart is now told by the stats socket (VPP creates a new one), and the heartbeat and the unreachable time only count after 60 s as a fallback.
  • A session that really ends by Idle-Timeout shows as Acct-Terminate-Cause 5. On my VM, sessions without traffic of the second build ended that way after the configured time (RADIUS Idle-Timeout 120 s), and with the new build rx-bytes follows ping traffic. The change does not touch the idle logic.

Third test build (the second one with this change, the path fix and the new installer):

https://cloud.insecure.pw/s/FcX5Zsf5EE9M3py
sha256: 5a2a602b06f6d3620b0bf7ef381f5f40e66e170fed48e24d00aa3a033c857968

Same requirements as before (rolling with VPP 25.10.0), ./install.sh after unpacking, ./restore.sh to go back.

  • The VPP crash in policer_input_node_fn_hsw: I read the code, vpppolicer_remove_limiter() unbinds and deletes the input policer first and takes the policer-input node out of the ip4 and ip6 feature arcs afterwards, which is the reverse of the install order. Your proposal (features first) is the same change I would make. I prepared it as a separate patch, but I cannot reproduce the crash with a few sessions, so it is untested. It is not in this build. If you can try it with your churn test, it is these lines in vpppolicer.c (the four vpppoe_set_feature(..., 0, "policer-input", ...) calls move in front of vpppolicer_set_input(0, ...)). I will open a separate task for it.
  • Same ifname for two active sessions (pppoe_session162): I have seen that once on my VM too, I did not look into it yet and will check it separately.

Still open from your side, if you can: the Session-Timeout test and the accounting values (Acct-Session-Time, octets, Acct-Terminate-Cause) of the sessions that ended. Thanks!

I will do the test and let you know

@saadinnocent909 a fourth test build, for the policer crash (https://vyos.dev/T9502). It is the third build plus one change in libvyos_vpp.so, the other two files are the same:

https://cloud.insecure.pw/s/fbPszz4pAyAs4sY
sha256: 53fbbf719b2facb6fb899fa47be0d11a0f8a8a0c9c2488d5e9190fad0b1fcd50

The change: when a session ends, the four policer-input features are removed from the ip4 and ip6 feature arcs first, then the input policer is unbound and deleted. Before, the policer was unbound and deleted first, and packets of the interface could still reach policer_input_node_fn in between. It is the order you suggested.

What I could and could not check:

  • On my VM, three sessions with a RADIUS rate limit (Mikrotik-Rate-Limit 2M/2M, so VPP policers are in use) and a ping flood on each of them, then accel-cmd terminate all and a reconnect, 15 times: VPP and accel-pppd kept their PID, vppctl show policer was empty after every round, no crash. This was the same with the build that has the old order (third build), also 15 rounds without a crash. So I could not reproduce the crash with three sessions, and the test does not show that the new order prevents it. It shows that the new order removes the policers cleanly and does not break anything.
  • With your load (many sessions ending at once while traffic flows) the window is much wider, so you are the only one who can tell. If you can run your churn test with this build and see no SIGBUS, that is the proof we need for a pull request. If VPP still crashes, please post the backtrace (coredumpctl or the crash log), then the order was not the only problem.

Same requirements and the same ./install.sh and ./restore.sh as before. The third build is the one without this change, so you can compare the two.

@saadinnocent909 a question for your crash tests, so that the VPP crashes are not mixed up. There are at least two different ones now:

  • T9502: SIGBUS in policer_input_node_fn_hsw (13:44:52 UTC in your report). The change to test is in the fourth build above.
  • T9501: the crash you saw with the NAT44 output-feature workaround, in fib_path_list_get_resolving_interface, called from nat_pre_in2out_output, while one user was disconnected. This is NAT44, not the policer.

You also wrote about a second VPP restart (14:41:22 UTC) without a configuration change. I do not know which of the two it was. For every VPP crash in your next tests it would help to have:

  • the backtrace (coredumpctl info vpp, or the crash log of VPP)
  • whether the output-feature workaround of T9501 was active
  • whether a policer was on the sessions (RADIUS rate limit)

Both crashes happen when a session ends while packets of its interface are still in the graph, so they are similar, but I do not claim that they have the same cause. If the fourth build crashes again in the policer, the order of the removal was not the only problem, and the backtrace will show it. If it crashes in NAT44, it belongs to T9501.

@saadinnocent909 there is a new test build, v5: https://cloud.insecure.pw/s/695B3YeXyZWpHn6

sha256 of the archive: bc2355f8b0cb56cbd947ff006806d1834b3ab403134e37721b37c93bf3948046

It is v3 (accel-ppp-ng #42 plus the fix from #44) with the changes that came out of the maintainer review of #42. It does not contain the policer change that was in v4, because that one is not confirmed yet. Install and restore work as before: ./install.sh as root (the originals are kept in /config/accel-ppp-ng.orig) and ./restore.sh to go back.

What changed compared to v3:

  • The counter baseline of a session is taken right after its VPP interface is created, before anything is forwarded. If no baseline can be taken (about one second of retries), the session is refused instead of starting without one. Earlier the first read took it, which could be after traffic had already passed.
  • A new session no longer throws away the cached counters of all other sessions. Sessions that come up together share one read of the stats segment. This should matter at your session count.
  • A counter that goes backwards (a reset such as vppctl clear interfaces, or an index reused by another interface) is no longer clamped to zero. The session ends, a warning is logged (the counters went backwards), and the accounting stop reports the last good values.
  • While the counters are temporarily unavailable, the idle timeout is not evaluated. The session timeout still is.

What would help to know, if you can run it with the same load as before:

  1. Do you still see sessions ending with NAS-Error or sessions disappearing without an obvious reason? If so, please add the log lines around it (journalctl -u accel-ppp@pppoe).
  2. Are sessions refused while many come up at once? A log line no counter baseline for the session interface would show it.
  3. Do you see the warning the counters went backwards without having cleared any counters?
  4. Do the Acct-Input/Output values of the accounting stop match what VPP shows (show interface) for a session that you end by hand?
  5. If VPP or accel-pppd crashes, a backtrace of each crash, with and without the NAT44 output-feature workaround, as discussed in T9501.

I tested this on a VM with VPP 25.10 and three sessions only (index reuse after terminate and reconnect, accounting stop against VPP, and a counter reset), so everything that depends on a large number of sessions is untested on my side.

@saadinnocent909 sorry for the confusion with the last build. I described v5 as "v3 plus the review changes", but it was built on a newer accel-ppp-ng base than v3, and I left out the policer change from v4, which is the one you are testing for the SIGBUS. Please use v6 and ignore v5.

v6: https://cloud.insecure.pw/s/PmyDkZFoyKS3brg

sha256 of the archive: 047910eb65d4dfcccf38b536501a4334e28b5c86b687bb4e20f78bb2075e94a5

What v6 is, exactly:

  • the accel-ppp-ng base is the current one (dd07b23), four upstream commits newer than the base of v3 and v4
  • accel-ppp-ng #42 with its latest commits, including the changes from the v5 text (baseline right after the interface is created, shared reads when sessions come up together, no clamping of a counter that goes backwards, no idle check while the counters are unavailable)
  • the fix from #44 (the abort when a VPP session ends)
  • the policer change from v4 (the four policer-input features are removed before the input policer is unbound and deleted, T9502)

Compared with v5 only libvyos_vpp.so differs, by the policer change. accel-pppd and libradius.so are the same files. Install and restore are as before: ./install.sh as root (originals in /config/accel-ppp-ng.orig), ./restore.sh to go back.

What I checked, on a VM with VPP 25.10:

  • v6 starts, sessions come up, and the counters match vppctl show int. Five rounds of accel-cmd terminate all with a reconnect keep the PIDs of VPP and accel-pppd.
  • Policers: with a RADIUS rate limit of 2M/2M (Mikrotik-Rate-Limit) and a ping flood on every session, each session has two VPP policers (up and down). I ran 60 sessions at the same time and ended them all eight times (terminate all, the clients reconnect). Every round started with 120 policers, afterwards the number matched the sessions that were up, nothing was left over, and there was no crash or SIGBUS. I ran the same test with v5, which has the old removal order, and got the same result, also no crash. So the test does not show that the new order prevents the SIGBUS, I could not reproduce it with 60 sessions at my small traffic. It shows that v6 removes the policers cleanly under this load and does not break anything. Your churn test, with the load you have, is the one that can show the difference.

A note in case your rate limit does not show up in VPP: the shaper looks the vendor name up in the RADIUS dictionary with a case-sensitive match, and the name there is Mikrotik. With set service pppoe-server authentication radius rate-limit vendor mikrotik (lower case) the vendor is not found, no limit is applied and there is no error message, so no policer is created. With vendor Mikrotik it works. If you use that attribute, please check that the rate-limit column of accel-cmd show sessions is filled and that vppctl show policer lists policers, otherwise the policer test does not test anything.

The questions from the v5 text still stand: NAS-Error or disappearing sessions, sessions refused while many come up at once (log line no counter baseline for the session interface), the warning the counters went backwards without a counter reset, the Acct-Input/Output values of the accounting stop against show interface, and a backtrace of each VPP or accel-pppd crash, with and without the NAT44 output-feature workaround (T9501).

I working on it. I will update you once testing is done

Thanks @ruben-herold. I tested v6 (sha256 047910eb…4a94a5 OK) for a full day. Setup: rolling 2026.10.16-0024, VPP 25.10.0, PPPoE on VPP eth0, FreeRADIUS on a separate VM, rate-limit via Filter-Id (no vendor), 2 MikroTik CPEs + ~2,000 PPPoE load sessions. Installed 11:33 UTC. I removed my LD_LIBRARY_PATH drop-in and it starts fine, so the RUNPATH fix works. accel-pppd ran the whole test with one PID and NRestarts=0, so the EBADF abort (#44) did not come back.

Q1 – NAS-Error / unexplained ends
In normal operation there was no NAS-Error at all (v2 ended every session at 60/120 s). Idle-Timeout 120 and Session-Timeout 180 both fire at the exact second. The NAS-Error stops in the journal:

  • ~2,000 at 13:35:39-41: VPP restart caused by my NAT44 config commit. Expected.
  • ~517 at 14:13: reconnect storm after reset pppoe-server all with 2,000 sessions. The journal shows radius: no available servers right before them. My FreeRADIUS could not 2,429 NAS-Request (the reset) + 517 NAS-Error.
  • ~218 at 13:11: the earlier reset-all storm, probably the same cause.

One small bug: when the idle timeout fires, Acct-Terminate-Cause is Session-Timeout, not Idle-Timeout. No Session-Timeout attribute was sent, and the journal says idle timeout. In dd07b23 accel-pppd/session.c:150-152:
if (ses->idle_timeout && ts.tv_sec - ses->idle_time >= ses->idle_timeout) {

log_ppp_msg("idle timeout\n");
ap_session_terminate(ses, TERM_SESSION_TIMEOUT, 0);

TERM_IDLE_TIMEOUT exists and maps to Idle-Timeout in radius/acct.c. This is old upstream code, not something from your PR, but it matters now that idle timeout works on VPP.

Q2 – sessions refused under mass connect
Yes, 2 of ~2,000 during the reset-all storm:
13:11:48 pppoe_session3878:lt0728: vpp: no counter baseline for the session interface (-11)
13:15:29 pppoe_session3878:lt0625: vpp: no counter baseline for the session interface (-11)
Both times the reused ifname was the same (pppoe_session3878). The CPE redialed 3-4 s later and succeeded. In the same storm there were bursts of failed to get interface statistics, also for sessions that never got an interface (auth failed / ppp_chan_send: short write -1 during teardown). These look like log noise only.

Q3 – "counters went backwards"
None during the whole run.

Q4 – Accounting-Stop vs VPP
Exact match. show interface pppoe_session3786: rx 492 pkts / 492,000 B, tx 492 / 502,824 B. radacct stop: in 492,000 / out 502,824. Note: rx is counted at IP level (1000 B/pkt), tx includes 22 B L2 (Eth 14 + PPPoE 6 + PPP 2) per packet, so downstream is ~2 % higher than upstream for the same traffic. I'm not sure if that is intended.

Q5 – crashes
No VPP crash with v6, no coredump. VPP's only restart was the SIGTERM from my config commit.

  • Policers, no NAT44 (T9502): 2,000 sessions, user traffic up to ~1 Gbit/s, 13x reset of that user, 6x CoA Filter-Id 20480 ↔ 1000000 under traffic (all ACK, rate changes), Disconnect, then reset pppoe-server all. VPP up since 11:27:30 the whole time.
  • NAT44 + output-feature workaround (T9501): 6 rounds of CoA Filter-Id 20480/20480 followed by Disconnect (the 10-21 sequence). 3 rounds with ICMP, 3 with TCP downloads through NAT (up to 1.3 GiB, 6 TCP + 15 UDP NAT sessions). Then reset pppoe-server all once. VPP up since 13:35:43 the whole time.

So for me the policer fix in v6 solves T9502. T9501 did not reproduce with v6 either. The 10-21 crash was on my own build without your #44 changes, so I can't say if v6 fixed it or I just did not hit it.

Other observations

  • After reset all, some old sessions still showed as active with frozen counters for ~2 min before they were removed. Once, one CPE got a new session with an empty rate-limit column and redialed by itself.
  • Full recovery of 2,000 sessions after reset all took ~5-7 min (mostly my RADIUS).

@saadinnocent909 thank you, this is the most useful test so far: a full day with about 2,000 sessions, the exact comparison of the accounting stop with show interface, and the details on the resets.

Going through your points:

  • Idle timeout and Acct-Terminate-Cause: you are right. In session.c the idle check ends the session with TERM_SESSION_TIMEOUT, TERM_IDLE_TIMEOUT exists and radius/acct.c maps it to Idle-Timeout. I checked it in the code. It is old upstream code and independent of this change, so I will handle it as a separate small fix and not put it into #42. I also have to correct something I wrote earlier in this task: I said that a session ended by Idle-Timeout shows Acct-Terminate-Cause 5. 5 is Session-Timeout, Idle-Timeout is 4. That was my mistake, and this bug is the reason why the two looked the same.
  • NAS-Error: thanks for sorting the three groups out. Nothing in normal operation, the VPP restart from your config commit, and the storms after reset pppoe-server all with radius: no available servers in front of them, that matches what the code does. A session ends with NAS-Error when its counters cannot be read for good, or when RADIUS cannot be reached while the session needs it.
  • Sessions refused (no counter baseline ... (-11)): this one is mine. -11 is -EAGAIN: the interface did not show up in the stats segment during the one second the create hook waits for it. Two cases out of about 2,000 is little, and the CPE got a session on the next try, but I do not know yet why both cases were the same interface (pppoe_session3878, 13:11:48 and 13:15:29). I will look at how the stats vectors grow with the interface index. If you still have the journal around those two times: the lines of vpp: and stats just before them, and whether vppctl show interface knew that interface at that time, would help.
  • failed to get interface statistics for sessions without an interface: I read this as noise too. The read fails because the session has no VPP interface yet. I will check that the message is not printed for sessions that never got one.
  • Rx at IP level, Tx with 22 bytes more per packet: the accounting reports what VPP counts for the interface, I do not change the values. So rx 492,000 and tx 502,824 for 492 packets are VPP's numbers, with Tx counted on the interface including the Ethernet, PPPoE and PPP header. I do not know whether the difference is intended on the VPP side, I can ask there if you want.
  • After reset all: I have not looked at the sessions that stayed active for about two minutes with frozen counters, and at the session with the empty rate-limit column. If you have the accel-cmd show sessions output of one of them, or the log of its start (journalctl -u accel-ppp@pppoe for the ifname), I can check whether the interface was gone at that time.

On the crashes: thank you for the exact sequences. With them, the policer order change (T9502) held in your test with 13 resets, 6 CoA changes under traffic, Disconnect and reset pppoe-server all at up to about 1 Gbit/s. I would like to open a pull request for it with your result, since it is public here. Please tell me if you do not want me to quote it. T9501 did not reproduce either, and as you say, your earlier crash was on a build without #44, so I do not claim that v6 fixed it.

Sure, feel free to quote my v6 results in the T9502 PR. Thanks for picking
up the idle-timeout cause as a separate fix.

On the -11 refusals: both affected users (lt0728, lt0625) got the same
interface name pppoe_session3878 on the refused attempt, during the `reset
pppoe-server all` storm with ~2000 sessions while old sessions were still
being torn down. Both redialed 3-4 s later and came up fine. Happy to rerun
a reset-all storm with extra debug logging if that helps.

saadinnocent909 thank you, and thanks for the permission: the T9502 pull request (https://github.com/accel-ppp/accel-ppp-ng/pull/46) quotes your v6 results and your comment on T9502 about the second crash.

About the two refused dial attempts (no counter baseline ... (-11)): that both got pppoe_session3878 fits what I would expect while many sessions are torn down at once. As far as I can tell VPP gives out a freed interface index again first, and during such a storm the stats segment is changing all the time, so the one second that the create hook waits for the baseline can be too short. That is my reading of the code and I have not shown it. With 2 of about 2,000 attempts and a redial 3 to 4 seconds later that succeeded, I would leave it as it is for now.

If you do run another reset-all storm, this would tell me what VPP was doing when a dial attempt was refused. A higher accel-ppp log level does not help for this, the message is already logged at error level and there are no debug lines for it yet. Please capture:

  1. The journal with precise timestamps, around every refusal: journalctl -u accel-ppp@pppoe -o short-precise --since "<start of the storm>" > /tmp/accel-storm.txt. The lines before a refusal matter too (stats segment lost, failed to get interface statistics).
  2. The state of VPP during the storm, one sample every two seconds, started just before reset pppoe-server all and stopped when all sessions are back: ` while true; do date +%T.%N; sudo vppctl show memory stats-segment | sed -n 2,3p; sudo vppctl show api ring-stats | head -8; sudo vppctl show runtime | sed -n 1,2p; sleep 2; done > /tmp/vpp-storm.txt ` show memory stats-segment shows how full the 128 MB stats segment is (it holds the counters of every interface), show api ring-stats shows whether the API rings run out of room during the storm, and show runtime shows how busy the main thread is.
  3. The number of interfaces before and after: sudo vppctl show interface | wc -l.

With the timestamps of a refusal and the VPP samples around it, I can tell whether the stats segment, the API rings or the main thread is the bottleneck. If that does not show enough, I will build a variant that logs the reason of the refusal (index, length of the snapshot, number of tries).