Today I tested VRF route leaking and NAT. It works on 1.3.1-S1. Simple configuration:

This feature would be very helpfull for hairpin nat as we can see from the mentions.
Might also be helpfull for ipv6 as well.
I am aware its a different product but edgeos from ubiquiti does something like this (looks to be a managed address group that populates dynamically) for nat and fw:

In T3933#124952, @sandwichdoge wrote:

@Viacheslav I tested your fix in my environment. The inbound filtering worked as expected after the fix. However it did not work correctly for the case we where we want both inbound and outbound firewalls on a single vrf member interface (or any case that has more than 2 directions on the same interface).

table ip filter {
	chain VYOS_FW_LOCAL {
		type filter hook input priority filter; policy accept;
		oifname "ONE" counter packets 63 bytes 6024 jump NAME_FOO # <<< Problem here, oifname should be eth0, not vrf name
		iifname "ONE" counter packets 63 bytes 6024 jump NAME_FOO
		jump VYOS_POST_FW
	}
...
	chain NAME_FOO {
		ip saddr 8.8.8.8 counter packets 79 bytes 6636 drop comment "FOO-10"
		counter packets 3 bytes 984 return comment "FOO default-action accept"
	}
}

I dug a little deeper, it appears that calling 2 http APIs in parallel results in vyos-http-api library crash.

Jul  5 08:47:39 cxr vyos-http-api[107198]: Configuration modified via HTTP API using key 'ccube-dev'
Jul  5 08:47:39 cxr vyos-http-api[107198]: INFO:     None:0 - "POST /configure HTTP/1.0" 200 OK
Jul  5 08:47:39 cxr netplugd[907]: br4: can't get flags: No such device
Jul  5 08:47:39 cxr vyos-http-api[107198]: processing form data
Jul  5 08:47:39 cxr netplugd[907]: br4: can't get flags: No such device
Jul  5 08:47:39 cxr netplugd[907]: message repeated 3 times: [ br4: can't get flags: No such device]
Jul  5 08:47:39 cxr netplugd[907]: br5: can't get flags: No such device
Jul  5 08:47:39 cxr netplugd[907]: br4: can't get flags: No such device
Jul  5 08:47:39 cxr netplugd[907]: message repeated 27 times: [ br4: can't get flags: No such device]
Jul  5 08:47:40 cxr vyos-http-api[107198]: INFO:     None:0 - "POST /config-file HTTP/1.0" 200 OK
Jul  5 08:47:40 cxr vyos-http-api[107198]: processing form data
Jul  5 08:47:40 cxr vyos-http-api[107198]: INFO:     None:0 - "POST /retrieve HTTP/1.0" 400 Bad Request
Jul  5 08:47:40 cxr vyos-http-api[107198]: processing form data
Jul  5 08:47:40 cxr vyos-http-api[107198]: processing form data
Jul  5 08:47:40 cxr ntpd[3893]: Listen normally on 87 vti1 169.254.231.46:123
Jul  5 08:47:40 cxr ntpd[3893]: new interface(s) found: waking up resolver
Jul  5 08:47:41 cxr kernel: [104872.825731] vyos-http-api-s[107280]: segfault at 1020 ip 00007f792d30391d sp 00007f792dce93e0 error 4 in libvyosconfig.so.0[7f792d2ae000+10c000]
Jul  5 08:47:41 cxr kernel: [104872.825745] Code: 20 48 83 c4 08 c3 e8 f2 d9 fa ff eb c6 48 83 ec 48 48 8b 40 10 48 89 44 24 18 48 8b 40 20 48 8b 58 20 48 8b 5b 20 48 8b 7b 20 <48> 8b 77 20 48 89 74 24 10 48 8b 56 20 48 89 54 24 20 48 8b 7f 08
Jul  5 08:47:41 cxr kernel: [104872.833057] net_ratelimit: 24 callbacks suppressed
Jul  5 08:47:41 cxr kernel: [104872.833059] IPv4: martian source 10.10.10.18 from 10.10.10.1, on dev eth2
Jul  5 08:47:41 cxr kernel: [104872.833061] ll header: 00000000: ff ff ff ff ff ff 6a 2c d7 cd 51 fd 08 06
Jul  5 08:47:41 cxr systemd[1]: vyos-http-api.service: Main process exited, code=killed, status=11/SEGV

I'm also getting the same error when calling the https API from localhost. In my case it only happens occasionally.

PR for 1.3 https://github.com/vyos/vyos-1x/pull/1391

https://docs.strongswan.org/docs/5.9/config/IPv6.html
https://docs.strongswan.org/docs/5.9/features/routeBasedVpn.html

In T2455#68732, @dmbaturin wrote:

VTI is secretly IPIP, so it doesn't support IPv6. The real issue is that we don't support the IPv6 variant of VTI yet.

PR for 1.3 https://github.com/vyos/vyos-1x/pull/1372

@Viacheslav As for your other concern, you can filter the actual inbound interface (eth4 in this my case) in mangle-PREROUTING. Maybe you could try packet marking in mangle-PREROUTING, then filter them later in VYOS_FW_FORWARD/VYOS_FW_LOCAL in the filter table?
Something like this:

@Viacheslav I tested your fix in my environment. The inbound filtering worked as expected after the fix. However it did not work correctly for the case we where we want both inbound and outbound firewalls on a single vrf member interface (or any case that has more than 2 directions on the same interface).

I have also hit this into the latest rolling version:

I think it should check this parameter per commit and it is a bug with validation as we don't have a tunnel interface yet
But after commit it will be valid value

@kroy Are you still having trouble with it?

PR https://github.com/vyos/vyos-1x/pull/1360

PR https://github.com/vyos/vyos-1x/pull/1358

set protocols failover route 203.0.113.1/32 next-hop 192.168.100.1 check target '192.168.100.1'
set protocols failover route 203.0.113.1/32 next-hop 192.168.100.1 check timeout '10'
set protocols failover route 203.0.113.1/32 next-hop 192.168.100.1 check type 'icmp'
set protocols failover route 203.0.113.1/32 next-hop 192.168.100.1 interface 'eth1'
set protocols failover route 203.0.113.1/32 next-hop 192.168.100.1 metric '2'

Thanks for the pointer, but I think it should still be considered a "bug" that you can no longer use an empty group (I'm just going to assume that this would apply to any kind of group, but most are probably using this for host/network groups, as this is where it would be most useful). Judging from the comments in T4147, I'm clearly not the only one who was taking advantage of managing sets outside of the system. Alas, my boot times for 1.4 (what this discussion is about) are not really valid, as my configuration didn't really get migrated from 1.3.1->1.4, or better said, it doesn't actually commit, and I actually ended up with a mostly empty firewall config on boot, which is perhaps why its booting so quickly now.

Fix Regex for addresses and python ckecks https://github.com/vyos/vyos-1x/pull/1354

@panachoi There is a task for groups T4147

Indeed, I figured that out. I also found that my openvpn config was not migrated properly (T3642?); all of the tls configuration stuff (previously kept under /config/auth somewhere) was gone. After doing run import pki for all of the necessary bits it was able at least to commit openvpn properly.

1. Some domains can't be added, for example dns.google

vyos@r12# set firewall group domain-group DOMAINS address dns.google

Fix smoketest https://github.com/vyos/vyos-1x/pull/1352

Yes. New 1.4 has more restricted checks on addresses and networks.
Actually, if you are using /22, the correct network for this case is 192.168.44.0/22.
You can use this online tool for checking ipv4 networks and subnets.

Sorry its taken me so long to follow up on this

Added more options. PR https://github.com/vyos/vyos-1x/pull/1350

PR for the current https://github.com/vyos/vyos-1x/pull/1346

In T973#124168, @superq wrote:

I wouldn't call telegraf a very good option. It does a very bad job of producing Prometheus metrics.

I wouldn't call telegraf a very good option. It does a very bad job of producing Prometheus metrics.

Prometheus-client already in 1.4
https://docs.vyos.io/en/latest/configuration/service/monitoring.html#prometheus-client

Has any progress on this been made? I am still having this issue on 1.4-rolling-202205250217.

This vm started out with 4G of memory and 2CPUs; I doubled quickly everything when I hit the out of memory error the first time, but that didn't help. I can quickly install the latest rolling and test

Yes, you error with "root" user is a known issue: T4281.

Reset added in T4442

I've debugged this further, by breaking up my configuration into various sections (system, interfaces, firewall,nat,service,vpn etc) and running them on a new VM.

Still not much luck here. But I've let the boot run a bit longer, and notice the following:

PR https://github.com/vyos/vyos-1x/pull/1340

set service event-handler event first filter pattern '.*ssh2.*'
set service event-handler event first script arguments '192.0.2.5'
set service event-handler event first script environment interface value 'eth0'
set service event-handler event first script path '/config/scripts/hello.sh'

The current salt-minion version 3003.4+ds-1
@maznu Do we need anything else for it?

PR https://github.com/vyos/vyos-1x/pull/1338

Okay, thats the only rule where I was using a port-group combined with protocol all; the others that use protocol all dont have a port or port group in the rule, so they are okay?

In T1230#123939, @panachoi wrote:

1.4 rolling does not help me, so there must be something "wrong" with my configuration. I've attached the private config, it would be awesome if someone might find what's broken.

private.cfg127 KBDownload

For a better analysis, can you share your firewall and nat config without hidden data? You can send it to my email: n.fort@vyos.io

1.4 rolling does not help me, so there must be something "wrong" with my configuration. I've attached the private config, it would be awesome if someone might find what's broken.

@panachoi , for me moving to 1.4 rolling release did the trick. Boot times went from > 10 mins in 1.2 to 2-3 minutes in 1.4. Hope that helps

@panachoi If you can share the anonymized config that works in 1.2.8 that would be useful. I'd expect migrating to 1.4 to see a decent improvement in firewall load times.

I'm still having issues moving past anything higher than 1.2.8. Booting 1.2.8 looks thusly:

There is an issue with vrf device for LOCAL direction
Imagine if you have 50 interfaces in one VRF and you want to drop all traffic from one interface for example - eth2 and don't touch other interfaces
You set firewall on eth2 Local - drop all traffic for device vrf and it will be affected to another 49 interfaces as iifname VRF_DEVICE the same

PR https://github.com/vyos/vyos-1x/pull/1330

set firewall name FOO default-action 'accept'
set firewall name FOO description 'desc'
set firewall name FOO rule 10 action 'drop'
set firewall name FOO rule 10 source address '8.8.8.8'
set interfaces ethernet eth0 firewall local name 'FOO'
set interfaces ethernet eth0 vrf 'ONE'
set vrf name ONE table '150'

Check:

table ip filter {
	chain VYOS_FW_LOCAL {
		type filter hook input priority filter; policy accept;
		iifname "ONE" counter packets 63 bytes 6024 jump NAME_FOO
		jump VYOS_POST_FW
	}
...
	chain NAME_FOO {
		ip saddr 8.8.8.8 counter packets 79 bytes 6636 drop comment "FOO-10"
		counter packets 3 bytes 984 return comment "FOO default-action accept"
	}
}

PR https://github.com/vyos/vyatta-op-firewall/pull/3

PR https://github.com/vyos/vyos-1x/pull/1327

@RyVolodya could you recheck it?

It is not only for dhcp, any service can be affected to this
There are no many check when we remove interface

@pyaskowski try zone-policy firewall

Has there been any movement on this? I've been following for a while since I've noticed the behavior myself and am hoping it can be resolved at some point.

Confirmed working.

Did similar tests with your big config >20k lines:

@n.fort I have upgraded following the path 1.2.6-S1 --> 1.3-S1 --> 1.4 Following the tips of not using port names for bootps and ssmtp and not using "set system ipv6 disable" and everything went fine.

Not sure if this works but there is an upload button in the toolbar: