I'll check, meanwhile, could you verify that you still see this issue in the latest build?

Quick update that using the transition-script method does not work (I think because the script isn't run as root).

I must have tested it with the values already set.

The fix is indeed to set arp_filter correctly but I'm still looking at how to make it persistent.

The move to systemd for the init process is making it so postconfig isn't being run after other services have started. In fact keepalived is only started AFTER the postconfig script.

I tried using the task scheduler with an interval of 1 min and that seems to work -ish. A pretty ugly work-around.

Really need postconfig to only execute after the system has completed startup.

On a side note the reason I'm digging into this in 1.2 is that 1.1.8 currently has a bug where unicast ARP directed to a VRRP virtual IP is not responded to (while ARP sent to broadcast works fine).

This is affecting Android 8.1 devices as they now use unicast ARP responses as a method of checking network connectivity and deauthenticate from wireless and attempt to reconnect if they do not see a response to a unicast ARP to the gateway.

VyOS 1.2 is not affected by this problem and correctly responds to unicast ARP but only with the fixes mentioned above for ARP to work at all.

In case you don't see the updates:

@dmbaturin
@syncer

Thanks for getting back to us
@dmbaturin where we set this ?

I'd need to check.

Also, are we talking about "normal" or "RFC-compliant" VRRP setup?

This is for rfc3768-compatibility enabled

Does the problem also exist with the other kind of VRRP?

Since I'm reimplementing VRRP now, I also would like to know if RFC-compliant is actually working, this issue aside, and whether we should keep the option.

For the reference: https://github.com/acassen/keepalived/blob/master/doc/NOTE_vrrp_vmac.txt

I have never attempted the non-RFC-compliant implementation but will test it Monday.

Does the non-compliant one not use a virtual MAC? If that's the case it would be unusable for failover in most environments since a host holding on to a stale ARP entry for the gateway would be too disruptive. Either way will take a look.

Just adding a note to confirm that RFC-compliant VRRP does work correctly provided the arp_filter settings are fixed. I am able to forward traffic through the routing using its virtual MAC as the next-hop and firewall policy remains correctly applied.

In terms of where to fix this, I haven't been able to find where keepalived is actually started, but that would be the place. I tried looking at Keepalived.pm and the start_daemon function but that didn't seem to be used anymore (?)

Anyway as a proof-of-concept I created the following perl script to set the correct arp_filter values on MACVLAN interfaces:

#!/usr/bin/perl

use lib '/opt/vyatta/share/perl5/';
use Vyatta::Interface;
use Vyatta::Misc;

use strict;
use warnings;

sub get_vrrp_intf_group {
    my @array;

    #
    # return an array of hashes that contains all the intf/group pairs
    #
    my $config = new Vyatta::Config;

    foreach my $name ( getInterfaces() ) {
        my $intf = new Vyatta::Interface($name);
        next unless $intf;
	my $path = $intf->path();
	$config->setLevel($path);
	if ($config->existsOrig('vrrp')) {
	    $path = "$path vrrp vrrp-group";
	    $config->setLevel($path);
	    my @groups = $config->listOrigNodes();
	    foreach my $group (@groups) {
		my %hash;
		$hash{'intf'}  = $name;
		$hash{'group'} = $group;
		$hash{'path'}  = "$path $group";
		push @array, {%hash};
	    }
	}
    }

    return @array;
}

my @vrrp_instances = get_vrrp_intf_group();
foreach my $hash (@vrrp_instances) {
	my $intf  = $hash->{'intf'};
	my $group = $hash->{'group'};
	my $macvlan = "$intf" . 'v' . "$group";
    if (-d "/proc/sys/net/ipv4/conf/$macvlan") {
        system("sudo echo 0 > /proc/sys/net/ipv4/conf/$macvlan/arp_filter");
    }
}

If this or similar could be called once keepalived has been started or restarted it should work. You might also check the config for RFC compatibility being enabled and skip it if not.

Just to add a quick note:

While Android behavior was what hinted at this issue in a more obvious way it appears to have been a longstanding bug. This evening I was able to confirm it was responsible for an 5 min (ish) outage of VyOS being used as a firewall where a Cisco router lost the ARP entry of a VyOS pair's VRRP address and the static routes pointing at VyOS for a next-hop were no longer active.

Looking back I now believe this has been the root cause of 5 or so unexplained outages that fit this pattern over the past 2 years. VyOS has otherwise been stable and reliable in this role.

So I took a step back and started wondering why we have /proc/sys/net/ipv4/conf/default/arp_filter set to 1 to begin with.

It looks like it appears in /etc/sysctl.conf with the obscure comment of "reset promiscous arp response" which doesn't really make sense. This looks like a hold-over from Vyatta as early on as VC 4 that was never questioned.

Simply removing the global arp_filter default fixes VRRP for 1.2

In terms of other things that might be affected I think the impact should be minor as arp_filter of 1 only comes into play for something like the physical interface for VRRP (which is being set either by keepalived or VyOS already when default is 0) or a situation where you have multiple network interfaces on the same subnet which is a bit of a weird and broken configuration in most cases anyway.

In 1.1 set interfaces <ethernet pseudo-ethernet bridge bonding> ... ip disable-arp-filter was introduced so my proposal would be to make the system default to 0 and update this configuration statement to be enable-arp-filter or similar.

For reference Cumulus also makes use of a global default of 0:

https://support.cumulusnetworks.com/hc/en-us/articles/203859616-Default-ARP-Settings-in-Cumulus-Linux

Edit: Looks like keepalived was updated to fix setting arp_filter on interfaces it needs in version 2.0.6

http://www.keepalived.org/changelog.html

In the vyos-build repository /data/live-build-config/hooks/08-sysconf.chroot needs to be updated to remove:

update_sysctl_conf net.ipv4.conf.default.arp_filter 1 \
    "reset promiscous arp response"

https://github.com/vyos/vyos-build/blob/db57dd817a9765122625a24b2bd02fe46ebed573/data/live-build-config/hooks/08-sysconf.chroot

This is resulting in the value being in /etc/sysctl.conf on the production system.

We prob. need to compare this against /vyatta-cfg-system/sysconf/vyatta-sysctl.conf as well as there might be other duplication and inconsistencies going on between the static file generated by the build process and VyOS. For example after removing ...default.arp_filter = 1 from /etc/sysctl.conf we likely don't need to be setting all.arp_filter = 0 here.

I think we should take care about this in the VRRP scripts and leave the base system untouched. Meaning, take your proposal and remove

update_sysctl_conf net.ipv4.conf.default.arp_filter 1 \
    "reset promiscous arp response"

from data/live-build-config/hooks/08-sysconf.chroot