Page MenuHomeVyOS Platform

IS-IS Memory Leak in isisd Leading to OOM Crash
Open, NormalPublicBUG

Description

Summary
A deployment of 400 VyOS routers (4 sites × 100 per IS-IS domain) on 2025.03.30.0020-rolling triggers unbounded growth in isisd memory usage. After ~30 minutes of full-mesh adjacency, isisd peaks above 2 GB and is OOM-killed, leading to FRR restarts and route flaps.
Affected Versions
VyOS build: 2025.03.30.0020-rolling
Topology & Scale
4 sites, each a single IS-IS domain over one shared L2 segment
100 routers per segment, full-mesh IS-IS adjacency
Analysis
The logs point to a leak in the IS-IS daemon’s SPF vertex-adjacency code path. During large-scale LSP processing, millions of “vertex adjacency” entries are built up in the fragment cache and never freed, causing isisd to consume over 2 GB of RAM (evidenced by ~1.6 GB of transparent huge-page allocations) before being killed by the OOM killer.
Key Logs & Metrics

Apr 17 11:45:15 … watchfrr: Thread Starvation – wakeup_send_echo() >4 s late  
Apr 17 11:52:27 … kernel: Out of memory: Killed process 1665 (isisd) total-vm:1405008kB, anon-rss:1395804kB  
Apr 17 11:52:29 … systemd: frr.service: Failed with result 'oom-kill'  
Apr 17 11:52:34 … watchfrr: all daemons up, restarting FRR 10.2.2  
# Mem-Info: Node 0 anon_thp=1605632kB for isisd

The suspected reason for this issue is:

--- qmem isisd ---
Type                          : Current#   Size       Total     Max#  MaxBytes
ISIS SPF Vertex Adjacency     : 36419888     64   2625814848 36419911 2625817304

Details

Version
-
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Unspecified (please specify)

Event Timeline

n.sambajon triaged this task as Normal priority.
n.sambajon created this object in space S1 VyOS Public.
n.sambajon changed Issue type from Bug (incorrect behavior) to Unspecified (please specify).May 13 2025, 12:11 AM

I did a configuration, basic in this case, I'm not able replicate this bug:
example:

set interfaces ethernet eth1 address '100.111.111.120/23'
set interfaces loopback lo address '100.111.20.220/32'
set policy prefix-list pl-export-lo-isis rule 10 action 'permit'
set policy prefix-list pl-export-lo-isis rule 10 prefix '100.111.20.220/32'
set policy route-map rm-export-lo-isis rule 10 action 'permit'
set policy route-map rm-export-lo-isis rule 10 match ip address prefix-list 'pl-export-lo-isis'
set protocols isis interface eth1 circuit-type 'level-1'
set protocols isis interface lo
set protocols isis log-adjacency-changes
set protocols isis metric-style 'wide'
set protocols isis net '49.0001.1101.1101.0220.00'
set protocols isis redistribute ipv4 connected level-1 route-map 'rm-export-lo-isis'

basic log:S

Sep 05 20:41:42 rt-01 isisd[1359]: [XVMQ9-2GBBQ] ISIS-SPF: No LSP found from root to L1 1101.1101.0222.00-00
Sep 05 20:41:42 rt-01 isisd[1359]: [XQ4FP-5NHW3] ISIS-SPF: No LSP found for 1101.1101.0222.00
Sep 05 20:41:42 rt-01 isisd[1359]: [XVMQ9-2GBBQ] ISIS-SPF: No LSP found from root to L1 1101.1101.0222.00-00
Sep 05 20:41:42 rt-01 isisd[1359]: [XQ4FP-5NHW3] ISIS-SPF: No LSP found for 1101.1101.0222.00
Sep 05 20:41:42 rt-01 isisd[1359]: [TMR1C-1X1AY] isis_redist_add: New route 100.111.110.0/23 from connected: distance 0.
Sep 05 20:41:42 rt-01 isisd[1359]: [TMR1C-1X1AY] isis_redist_add: New route 100.111.110.0/23 from connected: distance 0.
Sep 05 20:41:43 rt-01 isisd[1359]: [XVMQ9-2GBBQ] ISIS-SPF: No LSP found from root to L1 1101.1101.0221.03-00
Sep 05 20:41:43 rt-01 isisd[1359]: [XVMQ9-2GBBQ] ISIS-SPF: No LSP found from root to L1 1101.1101.0221.03-00
Sep 05 20:41:50 rt-01 isisd[1359]: [XVMQ9-2GBBQ] ISIS-SPF: No LSP found from root to L1 1101.1101.0221.03-00
Sep 05 20:41:50 rt-01 isisd[1359]: [XVMQ9-2GBBQ] ISIS-SPF: No LSP found from root to L1 1101.1101.0221.03-00
Sep 05 20:42:00 rt-01 isisd[1359]: [XVMQ9-2GBBQ] ISIS-SPF: No LSP found from root to L1 1101.1101.0221.03-00
Sep 05 20:42:00 rt-01 isisd[1359]: [XVMQ9-2GBBQ] ISIS-SPF: No LSP found from root to L1 1101.1101.0221.03-00
Sep 05 20:42:00 rt-01 isisd[1359]: [G2HNQ-ZA85D] ISIS-Snp (VyOS): Rcvd L1 PSNP from 5000.0003.0001 on eth1, skipping: we are not the DIS
Sep 05 20:42:11 rt-01 isisd[1359]: [XVMQ9-2GBBQ] ISIS-SPF: No LSP found from root to L1 1101.1101.0221.03-00
Sep 05 20:42:11 rt-01 isisd[1359]: [XVMQ9-2GBBQ] ISIS-SPF: No LSP found from root to L1 1101.1101.0221.03-00
Sep 05 20:42:12 rt-01 isisd[1359]: [TMR1C-1X1AY] isis_redist_add: New route 100.111.110.0/23 from connected: distance 0.
Sep 05 20:42:13 rt-01 isisd[1359]: [TMR1C-1X1AY] isis_redist_add: New route 100.111.110.0/23 from connected: distance 0.
Sep 05 20:42:41 rt-01 isisd[1359]: [X12R5-D6A2B] ISIS-Upd (VyOS): L1 LSP 1101.1101.0221.03-00 seq 0x00000002 aged out
Sep 05 20:42:41 rt-01 isisd[1359]: [TMR1C-1X1AY] isis_redist_add: New route 100.111.110.0/23 from connected: distance 0.
Sep 05 20:53:53 rt-01 rsyslogd[1869]: -- MARK --
Sep 05 20:56:07 rt-01 isisd[1359]: [TMR1C-1X1AY] isis_redist_add: New route 100.111.110.0/23 from connected: distance 0.
Sep 05 20:56:28 rt-01 isisd[1359]: [TMR1C-1X1AY] isis_redist_add: New route 100.111.110.0/23 from connected: distance 0.
Sep 05 20:56:32 rt-01 isisd[1359]: [TMR1C-1X1AY] isis_redist_add: New route 100.111.110.0/23 from connected: distance 0.
Sep 05 20:56:45 rt-01 isisd[1359]: [TMR1C-1X1AY] isis_redist_add: New route 100.111.110.0/23 from connected: distance 0.
Sep 05 20:56:51 rt-01 isisd[1359]: [TMR1C-1X1AY] isis_redist_add: New route 100.111.110.0/23 from connected: distance 0.
Sep 05 21:00:06 rt-01 systemd[1]: Starting Rotate log files...
Sep 05 21:00:06 rt-01 systemd[1]: logrotate.service: Deactivated successfully.
Sep 05 21:00:06 rt-01 systemd[1]: Finished Rotate log files.
Sep 05 21:01:20 rt-01 isisd[1359]: [Q7SVW-YVKRH] %ADJCHANGE: Adjacency to 1101.1101.0224 (eth1) for level-1 changed from Unknown to Initializing, unspecified
Sep 05 21:01:20 rt-01 isisd[1359]: [Q7SVW-YVKRH] %ADJCHANGE: Adjacency to 1101.1101.0224 (eth1) for level-1 changed from Initializing to Up, own SNPA found in LAN Neighbours TLV
Sep 05 21:01:20 rt-01 isisd[1359]: [TMR1C-1X1AY] isis_redist_add: New route 100.111.110.0/23 from connected: distance 0.
Sep 05 21:01:21 rt-01 isisd[1359]: [TMR1C-1X1AY] isis_redist_add: New route 100.111.110.0/23 from connected: distance 0.
Sep 05 21:01:21 rt-01 isisd[1359]: [G2HNQ-ZA85D] ISIS-Snp (VyOS): Rcvd L1 PSNP from 5000.0005.0001 on eth1, skipping: we are not the DIS
Sep 05 21:01:51 rt-01 isisd[1359]: [TMR1C-1X1AY] isis_redist_add: New route 100.111.110.0/23 from connected: distance 0.
Sep 05 21:01:52 rt-01 isisd[1359]: [TMR1C-1X1AY] isis_redist_add: New route 100.111.110.0/23 from connected: distance 0.
Sep 05 21:02:20 rt-01 isisd[1359]: [X12R5-D6A2B] ISIS-Upd (VyOS): L1 LSP 1101.1101.0222.03-00 seq 0x00000002 aged out
Sep 05 21:02:20 rt-01 isisd[1359]: [TMR1C-1X1AY] isis_redist_add: New route 100.111.110.0/23 from connected: distance 0.

It appears that when we redistribute connected in ISIS using route-map, it continues to add these routes or continuously learns them. I believe this may be the root cause of the problem.

Sep 11 09:18:11 rt-04 isisd[1321]: [TMR1C-1X1AY] isis_redist_add: New route 100.111.110.0/23 from connected: distance 0.
Sep 11 09:20:16 rt-04 isisd[1321]: [TMR1C-1X1AY] isis_redist_add: New route 100.111.110.0/23 from connected: distance 0.
Sep 11 09:22:26 rt-04 isisd[1321]: [TMR1C-1X1AY] isis_redist_add: New route 100.111.110.0/23 from connected: distance 0.
Sep 11 09:24:34 rt-04 isisd[1321]: [TMR1C-1X1AY] isis_redist_add: New route 100.111.110.0/23 from connected: distance 0.
Sep 11 09:27:41 rt-04 isisd[1321]: [TMR1C-1X1AY] isis_redist_add: New route 100.111.110.0/23 from connected: distance 0.
Sep 11 09:29:19 rt-04 rsyslogd[1838]: -- MARK --
Sep 11 09:29:48 rt-04 isisd[1321]: [TMR1C-1X1AY] isis_redist_add: New route 100.111.110.0/23 from connected: distance 0.
Sep 11 09:32:37 rt-04 isisd[1321]: [TMR1C-1X1AY] isis_redist_add: New route 100.111.110.0/23 from connected: distance 0.
Sep 11 09:34:58 rt-04 isisd[1321]: [TMR1C-1X1AY] isis_redist_add: New route 100.111.110.0/23 from connected: distance 0.
Sep 11 09:36:49 rt-04 isisd[1321]: [TMR1C-1X1AY] isis_redist_add: New route 100.111.110.0/23 from connected: distance 0.
Sep 11 09:39:30 rt-04 isisd[1321]: [TMR1C-1X1AY] isis_redist_add: New route 100.111.110.0/23 from connected: distance 0.
Sep 11 09:42:05 rt-04 isisd[1321]: [TMR1C-1X1AY] isis_redist_add: New route 100.111.110.0/23 from connected: distance 0.
Sep 11 09:44:21 rt-04 isisd[1321]: [TMR1C-1X1AY] isis_redist_add: New route 100.111.110.0/23 from connected: distance 0.
Sep 11 09:47:16 rt-04 isisd[1321]: [TMR1C-1X1AY] isis_redist_add: New route 100.111.110.0/23 from connected: distance 0.

We must conduct an investigation to ascertain the current practices for redistribution.

Viacheslav changed the subtype of this task from "Task" to "Bug".Sep 15 2025, 11:49 PM