Page MenuHomeVyOS Platform

arp_filter blocks ARPs for VRRP virtual addresses in 1.2.x
Closed, ResolvedPublicBUG

Description

When a VRRP virtual address is configured in 1.2.x I see no ARP responses from the virtual addresses unless I manually disable ARP filter for both the host interface and the VRRP virtual interface

Details

Version
999.201711232137

Event Timeline

syncer triaged this task as Normal priority.

I'll check, meanwhile, could you verify that you still see this issue in the latest build?

rps raised the priority of this task from Normal to Urgent!.Jul 20 2018, 6:28 PM
rps subscribed.

Moving this to critical because a router that won't respond to ARP is not very useful :-)

TLDR is that for VRRP interfaces VyOS will not respond to ARP requests for the virtual IP due to arp_filter being set to 1 on the MACVLAN interface used by keepalived.

I have confirmed this problem is still in the 1.2 nightly build as of 2018-07-20.

When VRRP is in use arp_filter being set to 1 on the MACVLAN interface makes it impossible for the virtual interface to respond to ARP making VRRP broken. Note if you only enable VRRP in your test setup after having cached ARP you may need to flush your ARP table to reproduce this.

The fix is to have VRRP MACVLAN interfaces set to arp_filter to 0. Note that the parent interface should still remain at 1.

For my test setup I'm using an 802.1Q VLAN interface of eth2.899 with VRRP MACVLAN interface being eth2.899v99, default state:

/proc/sys/net/ipv4/conf/all/arp_filter is set to 0
/proc/sys/net/ipv4/conf/default/arp_filter is set to 1
/proc/sys/net/ipv4/conf/eth2/arp_filter is set to 1
/proc/sys/net/ipv4/conf/eth2.899/arp_filter is set to 1
/proc/sys/net/ipv4/conf/eth2.899v99/arp_filter is set to 1

The result is that ARP for the VRRP virtual IP goes unanswered:

12:01:33.228530 10:dd:b1:d0:e8:22 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 100.64.32.1 tell 100.64.32.10, length 28
12:01:34.228732 10:dd:b1:d0:e8:22 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 100.64.32.1 tell 100.64.32.10, length 28
12:01:35.229252 10:dd:b1:d0:e8:22 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 100.64.32.1 tell 100.64.32.10, length 28
12:01:36.234206 10:dd:b1:d0:e8:22 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 100.64.32.1 tell 100.64.32.10, length 28
12:01:37.235071 10:dd:b1:d0:e8:22 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 100.64.32.1 tell 100.64.32.10, length 28
12:01:39.916162 10:dd:b1:d0:e8:22 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 100.64.32.2 tell 100.64.32.10, length 28

Changing the MACVLAN interface arp_filter to 0 (disable) allows the MACVLAN interface to correctly respond to ARP:

13:01:15.304036 10:dd:b1:d0:e8:22 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 58: Request who-has 100.64.32.1 tell 100.64.32.10, length 44
13:01:15.304250 00:00:5e:00:01:63 > 10:dd:b1:d0:e8:22, ethertype ARP (0x0806), length 60: Reply 100.64.32.1 is-at 00:00:5e:00:01:63, length 46
13:01:16.307966 10:dd:b1:d0:e8:22 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 58: Request who-has 100.64.32.1 tell 100.64.32.10, length 44
13:01:16.308163 00:00:5e:00:01:63 > 10:dd:b1:d0:e8:22, ethertype ARP (0x0806), length 60: Reply 100.64.32.1 is-at 00:00:5e:00:01:63, length 46
13:01:17.311685 10:dd:b1:d0:e8:22 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 58: Request who-has 100.64.32.1 tell 100.64.32.10, length 44
13:01:17.311872 00:00:5e:00:01:63 > 10:dd:b1:d0:e8:22, ethertype ARP (0x0806), length 60: Reply 100.64.32.1 is-at 00:00:5e:00:01:63, length 46

Having this set on all VRRP group member instances does not trigger BACKUP nodes to respond so it is safe to set when the VRRP interface is created regardless of role.

The need for this configuration is hinted at here: https://fossies.org/linux/keepalived/doc/NOTE_vrrp_vmac.txt

Also note that if any member of a sync-group is missing arp_filter = 0 on the MACVLAN interface the prempt-delay will fail and the master will switch over instantly.

As a work-around, the I attempted to add a fix to /config/scripts/vyatta-postconfig-bootup.script , but this script executes before MACVLAN interfaces are created.

Instead, the work-around must be called by keepalived with a notify_master hook being the most logical option. The following script is created:

#!/bin/bash

# This is a work-around for VRRP virtual interfaces (MACVLAN) being created 
# with "arp_filter = 1" resulting int ARP who-has requests to the virtual IP 
# being ignored.
#
# Call this script upon transitioning to a MASTER role using the following 
# configuration on at least one interface per sync-group:
#
# set interfaces ethernet eth0 vrrp vrrp-group 1 run-transition-scripts master '/config/scripts/macvlan-fix.script'

for i in `ls -1 /proc/sys/net/ipv4/conf/eth*v*/arp_filter`; do echo 0 > $i; done;

And called using notify_master using the following configuration:

set interfaces ethernet eth0 vrrp vrrp-group 1 run-transition-scripts master '/config/scripts/macvlan-fix.script'

Note the fix above is a bit quick-and-dirty as a proof-of-concept. The ultimate fix might be best handled in cluster-level hooks such as the /opt/vyatta/sbin/vyatta-vrrp-conntracksync.sh script to avoid conflict with interface-level notify_ scripts.

Quick update that using the transition-script method does not work (I think because the script isn't run as root).

I must have tested it with the values already set.

The fix is indeed to set arp_filter correctly but I'm still looking at how to make it persistent.

The move to systemd for the init process is making it so postconfig isn't being run after other services have started. In fact keepalived is only started AFTER the postconfig script.

I tried using the task scheduler with an interval of 1 min and that seems to work -ish. A pretty ugly work-around.

Really need postconfig to only execute after the system has completed startup.

On a side note the reason I'm digging into this in 1.2 is that 1.1.8 currently has a bug where unicast ARP directed to a VRRP virtual IP is not responded to (while ARP sent to broadcast works fine).

This is affecting Android 8.1 devices as they now use unicast ARP responses as a method of checking network connectivity and deauthenticate from wireless and attempt to reconnect if they do not see a response to a unicast ARP to the gateway.

VyOS 1.2 is not affected by this problem and correctly responds to unicast ARP but only with the fixes mentioned above for ARP to work at all.

Thanks for getting back to us
@dmbaturin where we set this ?

I'd need to check.

Also, are we talking about "normal" or "RFC-compliant" VRRP setup?

This is for rfc3768-compatibility enabled

Does the problem also exist with the other kind of VRRP?

Since I'm reimplementing VRRP now, I also would like to know if RFC-compliant is actually working, this issue aside, and whether we should keep the option.

I have never attempted the non-RFC-compliant implementation but will test it Monday.

Does the non-compliant one not use a virtual MAC? If that's the case it would be unusable for failover in most environments since a host holding on to a stale ARP entry for the gateway would be too disruptive. Either way will take a look.

Just adding a note to confirm that RFC-compliant VRRP does work correctly provided the arp_filter settings are fixed. I am able to forward traffic through the routing using its virtual MAC as the next-hop and firewall policy remains correctly applied.

In terms of where to fix this, I haven't been able to find where keepalived is actually started, but that would be the place. I tried looking at Keepalived.pm and the start_daemon function but that didn't seem to be used anymore (?)

Anyway as a proof-of-concept I created the following perl script to set the correct arp_filter values on MACVLAN interfaces:

#!/usr/bin/perl

use lib '/opt/vyatta/share/perl5/';
use Vyatta::Interface;
use Vyatta::Misc;

use strict;
use warnings;

sub get_vrrp_intf_group {
    my @array;

    #
    # return an array of hashes that contains all the intf/group pairs
    #
    my $config = new Vyatta::Config;

    foreach my $name ( getInterfaces() ) {
        my $intf = new Vyatta::Interface($name);
        next unless $intf;
	my $path = $intf->path();
	$config->setLevel($path);
	if ($config->existsOrig('vrrp')) {
	    $path = "$path vrrp vrrp-group";
	    $config->setLevel($path);
	    my @groups = $config->listOrigNodes();
	    foreach my $group (@groups) {
		my %hash;
		$hash{'intf'}  = $name;
		$hash{'group'} = $group;
		$hash{'path'}  = "$path $group";
		push @array, {%hash};
	    }
	}
    }

    return @array;
}

my @vrrp_instances = get_vrrp_intf_group();
foreach my $hash (@vrrp_instances) {
	my $intf  = $hash->{'intf'};
	my $group = $hash->{'group'};
	my $macvlan = "$intf" . 'v' . "$group";
    if (-d "/proc/sys/net/ipv4/conf/$macvlan") {
        system("sudo echo 0 > /proc/sys/net/ipv4/conf/$macvlan/arp_filter");
    }
}

If this or similar could be called once keepalived has been started or restarted it should work. You might also check the config for RFC compatibility being enabled and skip it if not.

Just to add a quick note:

While Android behavior was what hinted at this issue in a more obvious way it appears to have been a longstanding bug. This evening I was able to confirm it was responsible for an 5 min (ish) outage of VyOS being used as a firewall where a Cisco router lost the ARP entry of a VyOS pair's VRRP address and the static routes pointing at VyOS for a next-hop were no longer active.

Looking back I now believe this has been the root cause of 5 or so unexplained outages that fit this pattern over the past 2 years. VyOS has otherwise been stable and reliable in this role.

So I took a step back and started wondering why we have /proc/sys/net/ipv4/conf/default/arp_filter set to 1 to begin with.

It looks like it appears in /etc/sysctl.conf with the obscure comment of "reset promiscous arp response" which doesn't really make sense. This looks like a hold-over from Vyatta as early on as VC 4 that was never questioned.

Simply removing the global arp_filter default fixes VRRP for 1.2

In terms of other things that might be affected I think the impact should be minor as arp_filter of 1 only comes into play for something like the physical interface for VRRP (which is being set either by keepalived or VyOS already when default is 0) or a situation where you have multiple network interfaces on the same subnet which is a bit of a weird and broken configuration in most cases anyway.

In 1.1 set interfaces <ethernet pseudo-ethernet bridge bonding> ... ip disable-arp-filter was introduced so my proposal would be to make the system default to 0 and update this configuration statement to be enable-arp-filter or similar.

For reference Cumulus also makes use of a global default of 0:

https://support.cumulusnetworks.com/hc/en-us/articles/203859616-Default-ARP-Settings-in-Cumulus-Linux

Edit: Looks like keepalived was updated to fix setting arp_filter on interfaces it needs in version 2.0.6

http://www.keepalived.org/changelog.html

In the vyos-build repository /data/live-build-config/hooks/08-sysconf.chroot needs to be updated to remove:

update_sysctl_conf net.ipv4.conf.default.arp_filter 1 \
    "reset promiscous arp response"

https://github.com/vyos/vyos-build/blob/db57dd817a9765122625a24b2bd02fe46ebed573/data/live-build-config/hooks/08-sysconf.chroot

This is resulting in the value being in /etc/sysctl.conf on the production system.

We prob. need to compare this against /vyatta-cfg-system/sysconf/vyatta-sysctl.conf as well as there might be other duplication and inconsistencies going on between the static file generated by the build process and VyOS. For example after removing ...default.arp_filter = 1 from /etc/sysctl.conf we likely don't need to be setting all.arp_filter = 0 here.

I think we should take care about this in the VRRP scripts and leave the base system untouched. Meaning, take your proposal and remove

update_sysctl_conf net.ipv4.conf.default.arp_filter 1 \
    "reset promiscous arp response"

from data/live-build-config/hooks/08-sysconf.chroot

It looks like the commit has fixed this issue.