Page MenuHomeVyOS Platform

System freezes after attempting commit with insufficient memory
Closed, InvalidPublicBUG

Description

System freezes.

set system login user vyos authentication public-keys sever@sever key AAAAB3N....
set system login user vyos authentication public-keys sever@sever type ssh-rsa
set service ssh disable-host-validation
vyos@r1-roll# commit

The process is hanging.

vyos@r1-roll:~$ ps ax | grep commit
 4354 pts/1    S+     0:00 /opt/vyatta/sbin/my_commit
 4400 pts/0    S+     0:00 grep commit
vyos@r1-roll:~$

Version

vyos@r1-roll:~$ show version 

Version:          VyOS 1.3-rolling-202008311805
Release Train:    equuleus

Built by:         [email protected]
Built on:         Mon 31 Aug 2020 18:05 UTC
Build UUID:       cd929a4c-ed2c-42b7-b898-f5537537bce6
Build Commit ID:  2078f5ceffa291

Architecture:     x86_64
Boot via:         installed image
System type:      KVM guest

Hardware vendor:  QEMU
Hardware model:   Standard PC (Q35 + ICH9, 2009)
Hardware S/N:     Unknown
Hardware UUID:    Unknown

Copyright:        VyOS maintainers and contributors

It seems I need more resources (RAM).

[Tue Sep  1 06:46:46 2020] Mem-Info:
[Tue Sep  1 06:46:46 2020] active_anon:42194 inactive_anon:339 isolated_anon:0
                            active_file:3 inactive_file:99 isolated_file:0
                            unevictable:2272 dirty:0 writeback:0
                            slab_reclaimable:2752 slab_unreclaimable:5902
                            mapped:1319 shmem:490 pagetables:610 bounce:0
                            free:1601 free_pcp:0 free_cma:0
[Tue Sep  1 06:46:46 2020] Node 0 active_anon:168776kB inactive_anon:1356kB active_file:12kB inactive_file:396kB unevictable:9088kB isolated(anon):0kB isolated(file):0kB mapped:5276kB dirty:0kB writeback:0kB shmem:1960kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB all_unreclaimable? no
[Tue Sep  1 06:46:46 2020] DMA free:2548kB min:2184kB low:2216kB high:2248kB reserved_highatomic:2048KB active_anon:12144kB inactive_anon:0kB active_file:8kB inactive_file:4kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB kernel_stack:16kB pagetables:12kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[Tue Sep  1 06:46:46 2020] lowmem_reserve[]: 0 192 192 192
[Tue Sep  1 06:46:46 2020] DMA32 free:3856kB min:5532kB low:5956kB high:6380kB reserved_highatomic:0KB active_anon:156632kB inactive_anon:1356kB active_file:12kB inactive_file:388kB unevictable:9088kB writepending:0kB present:245616kB managed:222552kB mlocked:9088kB kernel_stack:2288kB pagetables:2428kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
[Tue Sep  1 06:46:46 2020] lowmem_reserve[]: 0 0 0 0
[Tue Sep  1 06:46:46 2020] DMA: 33*4kB (MEH) 40*8kB (MEH) 25*16kB (MEH) 13*32kB (MH) 2*64kB (MH) 1*128kB (H) 0*256kB 0*512kB 1*1024kB (M) 0*2048kB 0*4096kB = 2548kB
[Tue Sep  1 06:46:46 2020] DMA32: 23*4kB (UME) 236*8kB (UME) 90*16kB (UE) 14*32kB (UE) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 3868kB
[Tue Sep  1 06:46:46 2020] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[Tue Sep  1 06:46:46 2020] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[Tue Sep  1 06:46:46 2020] 1665 total pagecache pages
[Tue Sep  1 06:46:46 2020] 0 pages in swap cache
[Tue Sep  1 06:46:46 2020] Swap cache stats: add 0, delete 0, find 0/0
[Tue Sep  1 06:46:46 2020] Free swap  = 0kB
[Tue Sep  1 06:46:46 2020] Total swap = 0kB
[Tue Sep  1 06:46:46 2020] 65402 pages RAM
[Tue Sep  1 06:46:46 2020] 0 pages HighMem/MovableOnly
[Tue Sep  1 06:46:46 2020] 5787 pages reserved
[Tue Sep  1 06:46:46 2020] 0 pages hwpoisoned
[Tue Sep  1 06:46:46 2020] Tasks state (memory values in pages):
[Tue Sep  1 06:46:46 2020] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
[Tue Sep  1 06:46:46 2020] [    584]     0   584     7708     1091   106496        0             0 systemd-journal
[Tue Sep  1 06:46:46 2020] [    590]     0   590    30500     3886   131072        0             0 python3
[Tue Sep  1 06:46:46 2020] [    592]     0   592    77195    10060   221184        0             0 python3
[Tue Sep  1 06:46:46 2020] [    598]     0   598     5244      716    69632        0         -1000 systemd-udevd
[Tue Sep  1 06:46:46 2020] [    604]     0   604     2022     1153    53248        0             0 haveged
[Tue Sep  1 06:46:46 2020] [    722]     0   722      572       17    40960        0             0 atopacctd
[Tue Sep  1 06:46:46 2020] [    726]     0   726      582      176    40960        0             0 acpid
[Tue Sep  1 06:46:46 2020] [    727]   106   727     2180      367    53248        0          -900 dbus-daemon
[Tue Sep  1 06:46:46 2020] [    734]     0   734     2127      447    57344        0             0 cron
[Tue Sep  1 06:46:46 2020] [    744]     0   744     4828      436    73728        0             0 systemd-logind
[Tue Sep  1 06:46:46 2020] [    745]     0   745     1609      491    49152        0             0 qemu-ga
[Tue Sep  1 06:46:46 2020] [    750]     0   750     1373      403    49152        0             0 atd
[Tue Sep  1 06:46:46 2020] [    761]     0   761     2935     2270    69632        0             0 atop
[Tue Sep  1 06:46:46 2020] [    818]     0   818      575       21    36864        0             0 netplugd
[Tue Sep  1 06:46:46 2020] [    846]     0   846     2493      503    57344        0             0 watchfrr
[Tue Sep  1 06:46:46 2020] [    883]   110   883   169041     1338   200704        0             0 zebra
[Tue Sep  1 06:46:46 2020] [    890]   110   890    49301     1840   135168        0             0 bgpd
[Tue Sep  1 06:46:46 2020] [    903]   110   903     8525      763   102400        0             0 ripd
[Tue Sep  1 06:46:46 2020] [    907]   110   907     2680      451    61440        0             0 ripngd
[Tue Sep  1 06:46:46 2020] [    911]   110   911     8848      943   102400        0             0 ospfd
[Tue Sep  1 06:46:46 2020] [    915]   110   915     8690      860   102400        0             0 ospf6d
[Tue Sep  1 06:46:46 2020] [    919]   110   919     2916      812    69632        0             0 pimd
[Tue Sep  1 06:46:46 2020] [    926]   110   926     2576      546    61440        0             0 ldpd
[Tue Sep  1 06:46:46 2020] [    927]   110   927     2576      629    53248        0             0 ldpd
[Tue Sep  1 06:46:46 2020] [    928]   110   928     2860      520    61440        0             0 ldpd
[Tue Sep  1 06:46:46 2020] [    933]   110   933     2601      403    61440        0             0 staticd
[Tue Sep  1 06:46:46 2020] [    937]   110   937     2615      399    61440        0             0 bfdd
[Tue Sep  1 06:46:46 2020] [   1275]   109  1275    19118      413    61440        0             0 ntpd
[Tue Sep  1 06:46:46 2020] [   1354]     0  1354     1347      399    49152        0             0 agetty
[Tue Sep  1 06:46:46 2020] [   1357]     0  1357     1642      561    49152        0             0 login
[Tue Sep  1 06:46:46 2020] [   1362]  1003  1362     2040      914    57344        0             0 vbash
[Tue Sep  1 06:46:46 2020] [   2066]  1003  2066     2200     1098    49152        0             0 vbash
[Tue Sep  1 06:46:46 2020] [   2256]     0  2256     3964      572    65536        0         -1000 sshd
[Tue Sep  1 06:46:46 2020] [   2272]  1003  2272    56142       84    73728        0             0 unionfs-fuse
[Tue Sep  1 06:46:46 2020] [   2290]     0  2290     4130      762    69632        0             0 sshd
[Tue Sep  1 06:46:46 2020] [   2296]  1003  2296     4130      671    69632        0             0 sshd
[Tue Sep  1 06:46:46 2020] [   2297]  1003  2297     2049      927    49152        0             0 vbash
[Tue Sep  1 06:46:46 2020] [   3014]     0  3014    56457      385    86016        0             0 rsyslogd
[Tue Sep  1 06:46:46 2020] [   3163]  1003  3163     2199     1090    49152        0             0 vbash
[Tue Sep  1 06:46:46 2020] [   3182]  1003  3182    37675      111    61440        0             0 unionfs-fuse
[Tue Sep  1 06:46:46 2020] [   3325]  1003  3325     6040      606    81920        0             0 my_commit
[Tue Sep  1 06:46:46 2020] [   3328]     0  3328     2410      512    57344        0             0 sudo
[Tue Sep  1 06:46:46 2020] [   3329]     0  3329    23049      433    65536        0             0 vyshim
[Tue Sep  1 06:46:46 2020] [   3332]     0  3332    22025    14646   212992        0             0 python3
[Tue Sep  1 06:46:46 2020] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/system.slice/ssh.service,task=python3,pid=3332,uid=0
[Tue Sep  1 06:46:46 2020] Out of memory: Killed process 3332 (python3) total-vm:88100kB, anon-rss:55936kB, file-rss:2648kB, shmem-rss:0kB, UID:0 pgtables:208kB oom_score_adj:0
[Tue Sep  1 06:46:46 2020] oom_reaper: reaped process 3332 (python3), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB

Details

Difficulty level
Unknown (require assessment)
Version
VyOS 1.3-rolling-202008311805
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Event Timeline

Viacheslav created this task.
Viacheslav raised the priority of this task from High to Requires assessment.Sep 1 2020, 7:48 AM
Viacheslav updated the task description. (Show Details)
Viacheslav renamed this task from commit doesn't work to The commit does not work if there is insufficient memory..Sep 1 2020, 12:48 PM

@Viacheslav et. al. note that the succinct Mem-Info message shown above was added after v4.19 (commit ef8444ea), so was only seen during the brief window when we moved to 5.8 before reverting the kernel version, due to QAT issues and a bug affecting parted.

erkin renamed this task from The commit does not work if there is insufficient memory. to System freezes after attempting commit with insufficient memory.Aug 29 2021, 1:14 PM
erkin set Issue type to Bug (incorrect behavior).
erkin removed a subscriber: Active contributors.
jestabro edited projects, added VyOS 1.3 Equuleus; removed VyOS 1.3 Equuleus (1.3.3).

This has not been seen subsequently. Close unless recurrence.