To add to this, this link pointed me at the correct solution:

https://forums.centos.org/viewtopic.php?t=62558

It seems like this is poor systemd behavior and the only way around it currently is one of the two hacks mentioned there.

I've also confirmed this only occurs when logging in via IPv6. Adding -4 to the ssh command makes the ssh connections only take up a few megabytes of memory

Oh, in a system like vyos, SSH memory leak appears to be relatively serious. If there is a solution, it should be handled first

This also has a measurable impact on CPU. You can tell exactly when I applied the nsswitch.conf fix.

I have checked with a v4/v6 full table router and VyOS 1.2.5 - each SSH session will consume 7MiB which semms okay for me.

The same goes for 1.3-rolling-202006150117 so it's a really odd situation but according to the posts I've read this could be a real issue even when it does not happen to me.

@kroy could it be to existing/non existing IPv6 PTR records?

further information: https://manpages.debian.org/testing/libnss-myhostname/nss-myhostname.8.en.html

This one is also reported via forums: https://forum.vyos.io/t/vyos-rolling-full-bgp-ssh-ram-usage/5528

I found that I had disable-host-validation configured and as soon as I removed it it happened to me, too. Changing task priority.

Hi,

for me it looks like a name lookup error. I have read the forum entry mentioned above. And they fixed it by disabling name lookup.

Can it be that it only happened if a hostname (local or client) can't be probably resolved? This could also be cause not all people have the problem.

Somehow I do not want to change the overall system behavior by altering nsswitch.conf. I wonder if we should not enable "disable-host-ookups" by default as an IP address is in the end more useful then a resolved PTR. A PTR record can be changed later on when dissecting the logfiles but an IP lookup should stay longer.

default as an IP address is in the end more useful then a resolved PTR

I disagree with this a bunch. Especially if you have a number of logins happening

I really feel like the nsswitch fix is the cleanest from my research. The only thing is might potentially impact is if you did a

ssh vyos@localhost

vs impacting ANY remote login with that default you are suggesting

Does DNS static-host-mapping still work with the nssswich.conf change? I‘m just curious about the side effects.

It should. This should really be a non-breaking change as it's a fallback for something else that already exists in /etc/hosts.

127.0.1.1       edge.lan.kroy.io edge

Since the order is:

files dns #myhostname

The myhostname is ONLY ever for the local host, and shouldn't actually ever be used since the system's hostname is in /etc/hosts, which is called first.

It apparently causes some weird behavior when the IP of the ssh client doesn't have a PTR record.

For example, if I make my nsswitch.conf look like:

hosts:          files dns myhostname

If my IP has a PTR record either in DNS or hosts, the memory usage is only a few megabytes. My IPv4 has a PTR record, but my IPv6 doesn't, because of privacy extensions. So if I ssh -4, no massive memory usage.

admin@edge:~$ who
admin    pts/0        2020-07-05 19:19 (2001:dead:beed:9:fd1f:a68e:76c:44b6)

So my memory usage before a static host mapping over ipv6:

              total        used        free      shared  buff/cache   available
Mem:           7976         954        6131          86         890        6668
Swap:             0           0           0

After setting my temporary ipv6 address in a static entry with:

set system static-host-mapping host-name test3 inet 2001:dead:beef:9:fd1f:a68e:76c:44b6c

and logged out and back in again:

admin@edge:~$ free -m
              total        used        free      shared  buff/cache   available
Mem:           7976         762        6323          86         890        6860
Swap:             0           0           0

And this is there whether myhostname is commented out or not

admin@edge:~$ who
admin    pts/0        2020-07-05 19:23 (test3)

This PR should correct the issue.