Page MenuHomeVyOS Platform
Create Task
Maniphest T91

Memory leak in the Perl bindings for CStore (Vyatta::Config)
Closed, ResolvedPublic
Actions

Description

it seems this service

https://github.com/vyos/vyatta-cfg-system/blob/b4be922411d61216aca52202ec3ee432551a4f89/scripts/vyos-intfwatchd

watches the output of the command "ip monitor link" and takes action when links come back up. The problem is after high uptime or many link state changes (in our case, openvpn interfaces that have a down endpoint, making them reset every 30 seconds), the handle in perl becomes very large. After 2 weeks of uptime on one of our vyos, it was consuming 330MB.

transcript from conversation in IRC about it:

<LesB> anyone here know where i should go to troubleshoot vyos-intfwatchd consuming a large amount of RAM
<LesB> vyos crashed a couple weeks ago, so i bumped it to 1GB, and that process is sitting at 330MB consumed
<LesB> leaps and bounds higher than any other vyos we have running (we have something like 80 in use)
<LesB> best i can tell, vyos-intfwatchd watches the output of "ip monitor link". We have about 30 openvpns, a couple of which are always down at one point or another. Seems like that may be what is bloating this script
<LesB> dmbaturin, i see you were the last contrib to that script, any insight?
<dmbaturin> LesB: Oh. Yes, that's my script, and I have to admit it's rather sloppy.
<dmbaturin> It was made as a fix for the fact that since some version, linux kernel stopped getting IPv6 addresses back after interface flaps...
<LesB> dmbaturin, my perl knowledge is lacking. Any way you know of to have the buffer of "ip monitor link" release so it's not just continuously getting bigger?
<LesB> since we don't have ipv6 on any of our vyos, i just removed execute rights on the script and stopped the service... but i'd like that to not be the perm solution
<dmbaturin> LesB: Agree. So it seems perl garbage collection is giving us troubles. Could you create a task about it in phabricator so that I don't forget about it?

Details

Difficulty level
Easy (less than an hour)
Version
1.2

Related Objects

Event Timeline

LBegnaud created this task.Jun 14 2016, 12:46 PM
syncer claimed this task.Aug 23 2016, 7:59 PM
syncer reassigned this task from syncer to jhendryUK.
syncer merged a task: T112: Memory leak in vyos-intfwatchd.
syncer edited projects, added VyOS 1.1.x (1.1.8); removed VyOS 1.1.x.
syncer added subscribers: jhendryUK, syncer.
syncer added subscribers: VyOS 1.1.x, VyOS 2.0.x.
mickvav added a subscriber: mickvav.EditedAug 24 2016, 12:24 PM

I can suggest trying to do fork() on before line 139 and exit this forked child later - this should keep parent daemon's memory footprint constant.

syncer claimed this task.Nov 22 2016, 4:14 PM
syncer reassigned this task from syncer to jhendryUK.
syncer edited projects, added VyOS 1.2 Crux; removed VyOS 1.1.x (1.1.8).Dec 14 2016, 1:31 PM
syncer edited subscribers, added: VyOS 1.2 Crux; removed: VyOS 1.1.x.
syncer reassigned this task from jhendryUK to dmbaturin.Dec 21 2016, 8:54 AM
genta added a subscriber: genta.Jan 16 2018, 3:24 AM

I've found memory leak bug in Cstore perl binding (perlxs).
This binding is a part of vyatta-cfg.

https://github.com/vyos/vyatta-cfg/tree/helium/perl_dmod/Cstore

The vyos-intfwatched calling Vyatta::Interface::path() function.
This function instantiate Vyatta::Config each time when called, and Vyatta::Config uses Cstore, then memory exhaustion will happen.

You can confirm this issue running below scripts. (NOTE: DON'T RUN ON YOUR PRODUCTION ENVIRONMENT PLEASE! This causes memory exhaustion in a second.)

#!/usr/bin/perl
# demo #1: this code was splitted from vyos-intfwatched.
use lib "/opt/vyatta/share/perl5";
use strict;
use warnings;
use Vyatta::Interface;

while (1)
{
    my $intf = new Vyatta::Interface('vti1');
    my $intf_addr_path = $intf->path() . " address"; # <= memory leak here
}
#!/usr/bin/perl
# demo #2: Vyatta::Interface::path() calling new Vyatta::Config.
#          This also run out all of your memory.
use lib "/opt/vyatta/share/perl5";
use strict;
use warnings;
use Vyatta::Config;

while (1)
{
    my $config = new Vyatta::Config;    # <= memory leak here
}
#!/usr/bin/perl
# demo #3: Vyatta::Config calling new Cstore() (Perl binding).
#          This also run out all of your memory.
use lib "/opt/vyatta/share/perl5";
use strict;
use warnings;
use Cstore;

while (1)
{
    my $cstore = new Cstore();    # <= memory leak here
}

The main reason of this memory leak was a lack of DESTROY function in perlxs.
Implementing DESTORY in Cstore perlxs will fix this issue.

I've just sent a pull request to solve this issue.

https://github.com/vyos/vyatta-cfg/pull/10

After applying this patch, this issue solved.
Please consider to merge please.

Thanks,

dmbaturin renamed this task from vyosintfwatchd bug to Memory leak in the Perl bindings for CStore (Vyatta::Config).May 30 2018, 2:41 AM
dmbaturin closed this task as Resolved.
dmbaturin edited projects, added VyOS 1.2 Crux (VyOS 1.2.0-rc1); removed VyOS 1.2 Crux.
dmbaturin removed a subscriber: VyOS 2.0.x.

intfwatchd is no more (T669), so if it had any other memory leaks, they are also not a problem now.

dmbaturin moved this task from Needs Triage to Finished on the VyOS 1.2 Crux (VyOS 1.2.0-rc1) board.May 30 2018, 2:43 AM