NAT Problem with VRF
Closed, ResolvedPublicBUG
Actions

Assigned To

Authored By

	rherold
	Jun 28 2021, 9:57 AM

Description

hi,

I start using VRF and stumbled over a nasty nat bug:

Device:

eth0 192.168.0.100/24 gw 192.168.0.1 VRF OOBM
eth1 192.168.0.1/24 VRF default
eth2 no IP VRF default
pppoe0 dynamic public IP from ISP VRF default

eth0 and eth1 are conntected to the same switch and can ping each other

NAT RULE:

set nat source rule 100 outbound-interface 'pppoe0'
set nat source rule 100 protocol 'all'
set nat source rule 100 translation address 'masquerade'

The nat works for all other devices in 192.168.0/24. But all packets from 192.168.0.100 goes without masquerade out of pppoe0.

Details

Difficulty level: Unknown (require assessment)
Version: VyOS 1.4 / VyOS 1.5
Why the issue appeared?: Will be filled on close
Is it a breaking change?: Perfectly compatible
Issue type: Bug (incorrect behavior)

Related Objects

Mentioned In: T6097: vrf_zones blocking ipv6 traffic
T5116: Better VRF support
T3913: VRF traffic fails after upgrade from 1.3.0-RC6 to 1.3.0-EPA1/2

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes

@zsdc
please take a look on this
it might be some similar issue in this patch?
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=0fb4d21956f4a9af225594a46857ccf29bd747bc

because PREROUTING will be called twice

Hi @tj2852847

thanks for your comment , we are testing first with @rherold , I understand that your case is similar but it's not the same (you have an explicit route-leaking between default vrf and vrf X ). So we also need to test it and try to sure the version solved it .

Please take a look at the commit 9213ce6672582bc12f02c1530726fe97030d2cfe for kernel 5.13.

Hello Everyone,
I am testing 1.4 vrf leak from vrf x to default with NAT and is not working as expected. Outbound traffic is get forwarded to gateway NAT applied, but REPLY never forwarded to originator .

Version

vyos@PE-01:~$ show vers

Version:          VyOS 1.4-rolling-202109191513
Release train:    sagitta

Built by:         [email protected]
Built on:         Sun 19 Sep 2021 15:13 UTC
Build UUID:       6837bfa3-73ca-4621-abca-522358e9eec3
Build commit ID:  07555c06452524

Architecture:     x86_64
Boot via:         installed image
System type:      KVM guest

Hardware vendor:  QEMU
Hardware model:   Standard PC (i440FX + PIIX, 1996)
Hardware S/N:     
Hardware UUID:    43015507-c1f4-4857-9139-f3cb2e0d3597

Copyright:        VyOS maintainers and contributors

Traffic

10.41.100.139 is IP of outbound interface toward default gateway

15:50:55.850337 br10.255 In  IP (tos 0x0, ttl 62, id 39811, offset 0, flags [none], proto ICMP (1), length 84)
    72.15.151.138 > 8.8.8.8: ICMP echo request, id 42240, seq 4, length 64
15:50:55.850372 eth2  Out IP (tos 0x0, ttl 61, id 39811, offset 0, flags [none], proto ICMP (1), length 84)
    **10.41.100.139** > 8.8.8.8: ICMP echo request, id 42240, seq 4, length 64
15:50:55.863155 eth2  In  IP (tos 0x0, ttl 115, id 0, offset 0, flags [none], proto ICMP (1), length 84)
    8.8.8.8 > 10.41.100.139: ICMP echo reply, id 42240, seq 4, length 64

Route table default

vyos@PE-01:~$ show ip route 
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

S>* 0.0.0.0/0 [1/0] via 10.41.100.1, eth2, weight 1, 19:53:50
S>r 9.9.9.1/32 [1/0] via 192.0.2.1 (recursive), weight 1, 01:17:03
  r                    via 192.0.2.1, br10.255 onlink, weight 1, 01:17:03
C>* 10.41.100.0/24 is directly connected, eth2, 01w4d13h
S>* 72.15.151.136/29 [1/0] is directly connected, br10.255 (vrf OVERLAY), weight 1, 01:17:03
S>* 192.0.2.0/24 [1/0] is directly connected, br10.255 (vrf OVERLAY), weight 1, 01:17:03
vyos@PE-01:~$

Route table vrf X

vyos@PE-01:~$ show ip route vrf OVERLAY 
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
       F - PBR, f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup

VRF OVERLAY:
S>* 0.0.0.0/0 [1/0] via 10.41.100.1, eth2 (vrf default), weight 1, 01:12:13
K * 0.0.0.0/0 [255/8192] unreachable (ICMP unreachable), 01w1d22h
S>* 10.41.100.1/32 [1/0] is directly connected, eth2 (vrf default), weight 1, 01:12:13
C>* 192.0.2.0/24 is directly connected, br10.255, 01w1d14h
vyos@PE-01:~$

yes, It is an issues related with the conntrack+ nat/vrf leak , I share something where the problem is clearer :

https://serverfault.com/questions/1073012/conntrack-failed-to-nat-its-own-tcp-packets-from-another-vrf

The question how to disable connection tracking.

Is any work around for this scenario ?

not yet , we 've been trying with different CT but it's not solve the main problem . I understand that disabling conntrack is not possible because is used for nat.

syncer changed the task status from Open to In progress.Oct 17 2021, 2:56 PM

syncer triaged this task as High priority.

syncer edited projects, added VyOS 1.3 Equuleus (1.3.0-epa3), VyOS 1.4 Sagitta, Known issue; removed VyOS 1.3 Equuleus.

anything new here?

syncer edited projects, added VyOS 1.3 Equuleus (1.3.0); removed VyOS 1.3 Equuleus (1.3.0-epa3).Nov 1 2021, 10:11 PM

c-po mentioned this in T3913: VRF traffic fails after upgrade from 1.3.0-RC6 to 1.3.0-EPA1/2.Nov 8 2021, 6:01 PM

danhusan added a subscriber: danhusan.Nov 8 2021, 8:09 PM

drfill added a subscriber: drfill.Nov 12 2021, 7:32 PM

bbabich added a subscriber: bbabich.Jan 14 2022, 9:37 AM

forkwhilefork added a subscriber: forkwhilefork.Feb 1 2022, 1:26 AM

mpueschel added a subscriber: mpueschel.May 11 2022, 10:42 PM

janegil added a subscriber: janegil.May 18 2022, 11:01 PM

zakwan added a subscriber: zakwan.May 25 2022, 9:14 AM

torbj added a subscriber: torbj.May 25 2022, 6:22 PM

Today I tested VRF route leaking and NAT. It works on 1.3.1-S1. Simple configuration:

Снимок экрана от 2022-07-11 14-48-25.png (514×583 px, 23 KB)

set vrf name red table '1200'

set interfaces ethernet eth0 address 'dhcp'
set interfaces ethernet eth1 address '100.64.0.1/24'
set interfaces ethernet eth1 vrf 'red'

set nat source rule 10 outbound-interface 'eth0'
set nat source rule 10 source address '100.64.0.0/24'
set nat source rule 10 translation address 'masquerade'

set protocols static interface-route 100.64.0.0/24 next-hop-interface eth1 next-hop-vrf 'red'
set protocols vrf red static route 0.0.0.0/0 next-hop 192.168.255.1 next-hop-vrf 'default'

Viacheslav removed a project: VyOS 1.3 Equuleus (1.3.0).Jul 11 2022, 12:02 PM

I've re-tested this issues with the initial configuration, nat source / mesquered/ destination , it seems to work as @Dmitry said. The conntrack doesn't show the connection as [UNREPLIED] , it's established :

#configuration 

set interfaces ethernet eth1 address '192.168.0.100/24'
set interfaces ethernet eth1 hw-id '50:00:00:09:00:01'
set interfaces ethernet eth1 vrf 'OOBM'
set interfaces ethernet eth3 address '192.168.0.1/24'
set interfaces loopback lo
set nat destination rule 110 description 'NAT test- INSIDE'
set nat destination rule 110 destination port '2022'
set nat destination rule 110 inbound-interface 'eth0'
set nat destination rule 110 protocol 'tcp'
set nat destination rule 110 translation address '192.168.0.40'
set nat source rule 100 outbound-interface 'eth0'
set nat source rule 100 protocol 'all'
set nat source rule 100 source address '192.168.0.0/24'
set nat source rule 100 translation address 'masquerade'
set protocols vrf OOBM static route 0.0.0.0/0 next-hop 192.168.122.1 next-hop-vrf 'default'

egress traffic :

vyos@vyos:~$ traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 60 byte packets
 1  192.168.0.100 (192.168.0.100)  7.225 ms  6.194 ms  6.321 ms
 2  192.168.122.1 (192.168.122.1)  13.563 ms  12.694 ms  11.442 ms
 3  * * *
 4  200.51.241.1 (200.51.241.1)  49.754 ms  48.548 ms  44.612 ms
 5  74.125.32.151 (74.125.32.151)  44.074 ms 72.14.208.91 (72.14.208.91)  43.949 ms  43.823 ms
 6  74.125.52.126 (74.125.52.126)  43.683 ms 74.125.51.138 (74.125.51.138)  20.603 ms 74.125.52.126 (74.125.52.126)  20.230 ms
 7  74.125.242.193 (74.125.242.193)  28.352 ms 172.253.53.33 (172.253.53.33)  27.843 ms 74.125.242.193 (74.125.242.193)  27.035 ms
 8  142.251.239.165 (142.251.239.165)  26.905 ms 142.251.79.143 (142.251.79.143)  26.180 ms 142.250.46.111 (142.250.46.111)  25.734 ms
 9  8.8.8.8 (8.8.8.8)  30.609 ms  28.951 ms  24.783 ms
vyos@vyos:~$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=117 time=12.0 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=117 time=10.7 ms

conntrack reply ingress connection behind nat/vrf:

vyos@vyos:~$  conntrack -L
tcp      6 431988 ESTABLISHED src=192.168.122.49 dst=192.168.122.151 sport=44230 dport=2022 src=192.168.0.40 dst=192.168.122.49 sport=2022 dport=44230 [ASSURED] mark=0 use=1



vyos@vyos:~$ conntrack -E
    [NEW] tcp      6 120 SYN_SENT src=192.168.122.49 dst=192.168.122.151 sport=46156 dport=2022 [UNREPLIED] src=192.168.0.40 dst=192.168.122.49 sport=2022 dport=46156
 [UPDATE] tcp      6 60 SYN_RECV src=192.168.122.49 dst=192.168.122.151 sport=46156 dport=2022 src=192.168.0.40 dst=192.168.122.49 sport=2022 dport=46156
 [UPDATE] tcp      6 299 ESTABLISHED src=192.168.122.49 dst=192.168.122.151 sport=46156 dport=2022 src=192.168.0.40 dst=192.168.122.49 sport=2022 dport=46156 [ASSURED]
 6 60 SYN_RECV src=192.168.122.49 dst=192.168.122.151 sport=46156 dport=2022 src=192.168.0.40 dst=192.168.122.49 sport=2022 dport=46156
 [UPDATE] tcp      6 299 ESTABLISHED src=192.168.122.49 dst=192.168.122.151 sport=46156 dport=2022 src=192.168.0.40 dst=192.168.122.49 sport=2022 dport=46156 [ASSURED^Cconntrack v1.4.6 (conntrack-to.

Could you try with @rherold ? it should work using 1.3.1-S1

Hi, but one more thing related NAT and VRF in 1.4 rolling. As you know it uses NF MAP, to isolate conntrack tables, so we need to create some design to fix this moment. Matbe with adding some mark

I have NAT working with vrf in VyOS 1.4-rolling-202208290458 + custom nat offload

set interfaces ethernet eth0 address '192.168.122.14/24'
set interfaces ethernet eth1 address '192.0.2.1/24'
set interfaces ethernet eth1 vrf 'foo'
set protocols static route 192.0.2.0/24 interface eth1 vrf 'foo'
set system conntrack
set vrf name foo protocols static route 0.0.0.0/0 next-hop 192.168.122.1 interface 'eth0'
set vrf name foo protocols static route 0.0.0.0/0 next-hop 192.168.122.1 vrf 'default'
set vrf name foo table '1010'

Nftables

root@r14:/home/vyos# cat nat.nft 
flush ruleset

table ip filter {
	flowtable fastnat {
		hook ingress priority filter
		devices = { eth0, eth1 }
	}

	chain forward {
		type filter hook forward priority filter; policy accept;
		ip protocol { tcp, udp } flow add @fastnat
	}
}
table ip nat {
	chain POSTROUTING {
		type nat hook postrouting priority srcnat; policy accept;
		ip saddr 192.0.2.0/24 oif "eth0" snat to 192.168.122.14 persistent
	}

	chain PREROUTING {
		type nat hook prerouting priority dstnat; policy accept;
	}
}

Conntrack table

vyos@r14:~$ sudo conntrack -F
conntrack v1.4.6 (conntrack-tools): connection tracking table has been emptied.
vyos@r14:~$ 
vyos@r14:~$ sudo conntrack -L
tcp      6 431999 ESTABLISHED src=192.168.122.14 dst=192.168.122.1 sport=22 dport=44462 src=192.168.122.1 dst=192.168.122.14 sport=44462 dport=22 [ASSURED] mark=0 use=1
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=33018 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=33018 [OFFLOAD] mark=0 use=2
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=37517 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=37517 [OFFLOAD] mark=0 use=2
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=59794 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=59794 [OFFLOAD] mark=0 use=2
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=39288 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=39288 [OFFLOAD] mark=0 use=2
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=39616 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=39616 [OFFLOAD] mark=0 use=2
icmp     1 29 src=192.0.2.2 dst=1.1.1.1 type=8 code=0 id=12387 src=1.1.1.1 dst=192.168.122.14 type=0 code=0 id=12387 mark=0 use=1
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=41155 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=41155 [OFFLOAD] mark=0 use=2
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=39829 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=39829 [OFFLOAD] mark=0 use=2
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=33655 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=33655 [OFFLOAD] mark=0 use=2
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=44835 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=44835 [OFFLOAD] mark=0 use=2
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=40213 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=40213 [OFFLOAD] mark=0 use=2
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=33729 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=33729 [OFFLOAD] mark=0 use=2
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=48344 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=48344 [OFFLOAD] mark=0 use=2
conntrack v1.4.6 (conntrack-tools): 14 flow entries have been shown.
vyos@r14:~$

as I remember ... it has been working by this PR:

https://phabricator.vyos.net/rVYOSONEX22791e26f444766dc9f9e1729b72893208f58079

but I am not sure , if a kernel update fixed it because as I understood it was a well-known issue on contrackd .

Viacheslav added a comment.Aug 30 2022, 6:33 AM

This comment was removed by Viacheslav.

aohanian added a subscriber: aohanian.Oct 1 2022, 2:25 PM

Is there a way to isolate a NAT rule to operate within a VRF?

For example, let's say I have the following configuration:

set vrf name red table '101'
set vrf name blue table '102'

set interfaces ethernet eth0 vif 101 vrf red
set interfaces ethernet eth0 vif 101 address 100.64.0.2/30

set interfaces ethernet eth0 vif 102 vrf blue
set interfaces ethernet eth0 vif 102 address 100.64.0.6/30

set interfaces ethernet eth1 vif 101 vrf red
set interfaces ethernet eth1 vif 101 address 192.168.0.1/24

set interfaces ethernet eth1 vif 102 vrf blue
set interfaces ethernet eth1 vif 102 address 192.168.0.1/24

How can I NAT vrf red source traffic from 192.168.0.0/24 to 100.64.0.2 and vrf blue source traffic from 192.168.0.0/24 to 100.64.0.6?

I'm looking for a command such as

set nat source rule 100 vrf red

The problem here is that the two vrf tables share the client source address of 192.168.0.0/24, but they should have different translated addresses. I realize this is slightly different from the discussion about NATing between VRFs.

At least on my lab, with one of the latest 1.4, this is working for me:

set interfaces ethernet eth0 vif 101 address '100.64.0.2/30'
set interfaces ethernet eth0 vif 101 vrf 'red'
set interfaces ethernet eth0 vif 102 address '100.64.0.6/30'
set interfaces ethernet eth0 vif 102 vrf 'blue'

set interfaces ethernet eth1 vif 101 address '192.168.0.1/24'
set interfaces ethernet eth1 vif 101 vrf 'red'
set interfaces ethernet eth1 vif 102 address '192.168.0.1/24'
set interfaces ethernet eth1 vif 102 vrf 'blue'

set vrf name blue protocols static route 0.0.0.0/0 next-hop 100.64.0.5
set vrf name blue table '102'
set vrf name red protocols static route 0.0.0.0/0 next-hop 100.64.0.1
set vrf name red table '101'

set nat source rule 10 outbound-interface 'eth0.101'
set nat source rule 10 translation address 'masquerade'
set nat source rule 20 outbound-interface 'eth0.102'
set nat source rule 20 translation address 'masquerade'

Then, pinging from two host, one on each vrf (both with ip 192.168.0.X), I can see correct translation address on remote router. From host on vrf red ping to 1.1.1.1, and from host in vrf blue ping to 8.8.8.8

tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), snapshot length 262144 bytes
14:17:22.675855 IP 100.64.0.2 > 1.1.1.1: ICMP echo request, id 62190, seq 313, length 64
14:17:22.700520 IP 1.1.1.1 > 100.64.0.2: ICMP echo reply, id 62190, seq 313, length 64
14:17:23.294866 IP 100.64.0.6 > 8.8.8.8: ICMP echo request, id 62446, seq 312, length 64
14:17:23.317557 IP 8.8.8.8 > 100.64.0.6: ICMP echo reply, id 62446, seq 312, length 64

scj643 added a subscriber: scj643.Nov 9 2022, 7:38 PM

Charlie-Root added a subscriber: Charlie-Root.Feb 21 2023, 1:45 PM

This comment was removed by Charlie-Root.

I have almost same problem here. Can't NAT between two VRFs correctly. The outgoing packet has been NATed correctly but the incoming packet seems be dropped, can't reach the source, it seems the return packet can't be tracked correctly.

vyos@bsp-asbr2-cm# comp
         description "MPLS BB originated from CM"
         hw-id "00:0c:29:33:09:f8"
         vrf "cm_up"
+    }
+    ethernet eth4 {
+        address "100.64.0.1/24"
+        description "Experimental VRF"
     }
     loopback     lo { }
     tunnel tun0 {
         }
         table "101"
     }
 }
 

[edit]
vyos@bsp-asbr2-cm# set int eth eth4 vrf experimental 
[edit]
vyos@bsp-asbr2-cm# commit
[edit]
vyos@bsp-asbr2-cm# save
Saving configuration to '/config/config.boot'...
Done
[edit]
vyos@bsp-asbr2-cm# show nat
 destination {
     rule 10 {
         destination {
             port 10000-64000
         }
         inbound-interface eth0
         protocol tcp_udp
         translation {
             address 192.168.124.34
         }
     }
 }
 source {
     rule 10 {
         outbound-interface eth0
         protocol all
         translation {
             address masquerade
         }
     }
 }
[edit]
vyos@bsp-asbr2-cm# show vrf
 name cm_up {
     protocols {
         static {
             route 0.0.0.0/0 {
                 next-hop XXX.XXX.XX.1 {
                 }
             }
         }
     }
     table 101
 }
 name experimental {
     protocols {
         static {
             route 0.0.0.0/0 {
                 next-hop XXX.XXX.XX.1 {
                     vrf cm_up
                 }
             }
         }
     }
     table 1000
 }
[edit]
vyos@bsp-asbr2-cm# tcpdump -i eth4 icmp
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth4, link-type EN10MB (Ethernet), snapshot length 262144 bytes
14:16:26.844140 IP 100.64.0.2 > 202.101.224.68: ICMP echo request, id 20655, seq 1, length 64
14:16:27.844653 IP 100.64.0.2 > 202.101.224.68: ICMP echo request, id 20655, seq 2, length 64
14:16:28.868656 IP 100.64.0.2 > 202.101.224.68: ICMP echo request, id 20655, seq 3, length 64
14:16:29.892643 IP 100.64.0.2 > 202.101.224.68: ICMP echo request, id 20655, seq 4, length 64
14:16:30.916754 IP 100.64.0.2 > 202.101.224.68: ICMP echo request, id 20655, seq 5, length 64
^C
5 packets captured
5 packets received by filter
0 packets dropped by kernel
vyos@bsp-asbr2-cm# tcpdump -i eth0 host 202.101.224.68
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
14:20:21.316994 IP XXX.XXX.XX.XXX > 202.101.224.68: ICMP echo request, id 19836, seq 13, length 64
14:20:21.351321 IP 202.101.224.68 > XXX.XXX.XX.XXX: ICMP echo reply, id 19836, seq 13, length 64
14:20:22.340995 IP XXX.XXX.XX.XXX > 202.101.224.68: ICMP echo request, id 19836, seq 14, length 64
14:20:22.374407 IP 202.101.224.68 > XXX.XXX.XX.XXX: ICMP echo reply, id 19836, seq 14, length 64
14:20:23.364925 IP XXX.XXX.XX.XXX > 202.101.224.68: ICMP echo request, id 19836, seq 15, length 64
14:20:23.399367 IP 202.101.224.68 > XXX.XXX.XX.XXX: ICMP echo reply, id 19836, seq 15, length 64
14:20:24.388917 IP XXX.XXX.XX.XXX > 202.101.224.68: ICMP echo request, id 19836, seq 16, length 64
14:20:24.423237 IP 202.101.224.68 > XXX.XXX.XX.XXX: ICMP echo reply, id 19836, seq 16, length 64
^C
8 packets captured
8 packets received by filter
0 packets dropped by kernel
[edit]
vyos@bsp-asbr2-cm:~$ show ver
Version:          VyOS 1.4-rolling-202302150317
Release train:    current

Built by:         [email protected]
Built on:         Wed 15 Feb 2023 03:17 UTC
Build UUID:       e62b2d4d-c09c-4dd6-a722-884b782e4d13
Build commit ID:  5207b6f510d677

Architecture:     x86_64
Boot via:         installed image
System type:      VMware guest

Hardware vendor:  VMware, Inc.
Hardware model:   VMware20,1
Hardware S/N:     VMware-56 4d fb 13 43 c4 e5 63-b0 9e 49 a8 b7 33 09 da
Hardware UUID:    13fb4d56-c443-63e5-b09e-49a8b73309da

Copyright:        VyOS maintainers and contributors
vyos@bsp-asbr2-cm:~$ uname -a
Linux bsp-asbr2-cm 6.1.11-amd64-vyos #1 SMP PREEMPT_DYNAMIC Fri Feb 10 07:40:24 UTC 2023 x86_64 GNU/Linux
vyos@bsp-asbr2-cm:~$ 
vyos@bsp-asbr2-cm:~$ show ip route vrf all
Codes: K - kernel route, C - connected, S - static, R - RIP,
       O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
       T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
       f - OpenFabric,
       > - selected route, * - FIB route, q - queued, r - rejected, b - backup
       t - trapped, o - offload failure

VRF cm_up:
S>* 0.0.0.0/0 [1/0] via XXX.XXX.XX.1, eth0, weight 1, 21:01:40
C>* XXX.XXX.XX.0/25 is directly connected, eth0, 21:01:42
C>* 192.168.124.32/28 is directly connected, eth3, 21:01:43
C>* 192.168.127.32/32 is directly connected, dum0, 21:01:43

VRF default:
O>* 10.96.3.0/24 [110/20] via 192.168.124.2, eth1, weight 1, 21:01:33
O>* 10.96.4.0/24 [110/2] via 192.168.124.2, eth1, weight 1, 21:01:34
O>* 10.96.255.0/30 [110/3] via 192.168.124.2, eth1, weight 1, 21:01:34
O   10.96.255.8/30 [110/1] is directly connected, tun0, weight 1, 20:57:35
C>* 10.96.255.8/30 is directly connected, tun0, 20:57:56
O>* 192.168.0.0/25 [110/2] via 10.96.255.10, tun0, weight 1, 19:44:45
O>* 192.168.0.128/25 [110/2] via 10.96.255.10, tun0, weight 1, 19:44:45
O>* 192.168.2.0/25 [110/2] via 10.96.255.10, tun0, weight 1, 19:44:45
O>* 192.168.8.0/25 [110/2] via 10.96.255.10, tun0, weight 1, 19:44:45
O>* 192.168.14.0/28 [110/2] via 10.96.255.10, tun0, weight 1, 19:44:45
O>* 192.168.16.0/25 [110/2] via 10.96.255.10, tun0, weight 1, 19:44:45
O>* 192.168.16.128/25 [110/2] via 10.96.255.10, tun0, weight 1, 19:44:45
O>* 192.168.63.32/32 [110/2] via 10.96.255.10, tun0, weight 1, 19:44:45
O>* 192.168.65.0/24 [110/3] via 192.168.124.2, eth1, weight 1, 21:01:34
O>* 192.168.66.0/27 [110/20] via 192.168.124.2, eth1, weight 1, 21:01:33
O>* 192.168.68.0/24 [110/3] via 192.168.124.2, eth1, weight 1, 21:01:34
O>* 192.168.69.0/25 [110/3] via 192.168.124.2, eth1, weight 1, 21:01:34
O>* 192.168.72.0/24 [110/3] via 192.168.124.2, eth1, weight 1, 21:01:34
O>* 192.168.79.0/24 [110/3] via 192.168.124.2, eth1, weight 1, 21:01:34
O>* 192.168.80.128/25 [110/3] via 192.168.124.2, eth1, weight 1, 21:01:34
O   192.168.124.0/28 [110/1] is directly connected, eth1, weight 1, 21:01:39
C>* 192.168.124.0/28 is directly connected, eth1, 21:01:42
O>* 192.168.124.64/28 [110/2] via 192.168.124.2, eth1, weight 1, 21:01:34
O>* 192.168.127.1/32 [110/1] via 192.168.124.2, eth1, weight 1, 21:01:34
O>* 192.168.127.33/32 [110/3] via 192.168.124.2, eth1, weight 1, 21:01:34
O   192.168.127.34/32 [110/1] via 0.0.0.0, dum1 onlink, weight 1, 21:01:39
C>* 192.168.127.34/32 is directly connected, dum1, 21:01:44

VRF experimental:
S>* 0.0.0.0/0 [1/0] via XXX.XXX.XX.1, eth0 (vrf cm_up), weight 1, 00:10:22
C>* 100.64.0.0/24 is directly connected, eth4, 00:22:18
vyos@bsp-asbr2-cm:~$ sudo conntrack -L |grep 202.101
icmp     1 29 src=100.64.0.2 dst=202.101.224.68 type=8 code=0 id=30080 [UNREPLIED] src=202.101.224.68 dst=183.221.22.100 type=0 code=0 id=30080 zone=1000 use=1
conntrack v1.4.6 (conntrack-tools): 183 flow entries have been shown.
vyos@bsp-asbr2-cm:~$

it doesn't seem the same problem as here, this logic that was applied over this version was vrf not on the table . Could you share full configuration ? there is some point over vrfs / vrf default /leaking that are not clear. So I can replicate the scenery and we see what is going on .

In T3655#143947, @fernando wrote:

it doesn't seem the same problem as here, this logic that was applied over this version was vrf not on the table . Could you share full configuration ? there is some point over vrfs / vrf default /leaking that are not clear. So I can replicate the scenery and we see what is going on .

My configuration is listed below:

login as: vyos
[email protected]'s password:
Welcome to VyOS!

Check out project news at https://blog.vyos.io
and feel free to report bugs at https://vyos.dev

You can change this banner using "set system login banner post-login" command.

VyOS is a free software distribution that includes multiple components,
you can check individual component licenses under /usr/share/doc/*/copyright
Last login: Fri Mar  3 18:22:31 2023 from 192.168.68.2
vyos@bsp-asbr2-cm:~$ show conf
interfaces {
    dummy dum0 {
        address 192.168.127.32/32
        description "GRE over IPSec originate loopback"
        vrf cm_up
    }
    dummy dum1 {
        address 192.168.127.34/32
    }
    ethernet eth0 {
        address XXX.XXX.XX.100/25
        description "To China Mobile static access"
        hw-id 00:0c:29:33:09:da
        vrf cm_up
    }
    ethernet eth1 {
        address 192.168.124.1/28
        description "Downstream to vSRX"
        hw-id 00:0c:29:33:09:e4
    }
    ethernet eth2 {
        address 192.168.124.66/28
        description "MPLS BB between 2 HV"
        disable
        hw-id 00:0c:29:33:09:ee
    }
    ethernet eth3 {
        address 192.168.124.33/28
        description "MPLS BB originate from CM"
        hw-id 00:0c:29:33:09:f8
        vrf cm_up
    }
    ethernet eth4 {
        address 100.64.0.1/24
        description "Experimental VRF"
        hw-id 00:0c:29:33:09:02
        vrf experimental
    }
    loopback lo {
    }
    tunnel tun0 {
        address 10.96.255.9/30
        description "S2S VPN 1"
        encapsulation gre
        ip {
            adjust-mss clamp-mss-to-pmtu
        }
        mtu 1428
        remote 192.168.63.32
        source-address 192.168.127.32
        source-interface dum0
    }
}
nat {
    destination {
        rule 10 {
            destination {
                port 10000-64000
            }
            inbound-interface eth0
            protocol tcp_udp
            translation {
                address 192.168.124.34
            }
        }
    }
    source {
        rule 10 {
            outbound-interface eth0
            protocol all
            translation {
                address masquerade
            }
        }
    }
}
pki {
    key-pair ipsec-CDSLCM {
        private {
            key ****************
        }
        public {
            key ****************
        }
    }
    key-pair ipsec-CDSLCU {
        public {
            key ****************
        }
    }
    key-pair ipsec-JXNCCT {
        public {
            key ****************
        }
    }
}
protocols {
    ospf {
        area 0.0.0.0 {
            network 192.168.0.0/15
            network 10.96.0.0/16
        }
        parameters {
            router-id 192.168.127.32
        }
    }
}
qos {
    policy {
        shaper test {
            bandwidth 300mbit
            default {
                bandwidth 200mbit
                queue-type fair-queue
            }
        }
    }
}
service {
    ntp {
        allow-client {
            address 0.0.0.0/0
            address ::/0
        }
        server time1.vyos.net {
        }
        server time2.vyos.net {
        }
        server time3.vyos.net {
        }
    }
    ssh {
        listen-address 192.168.124.1
    }
}
system {
    config-management {
        commit-revisions 100
    }
    conntrack {
        modules {
            ftp
            h323
            nfs
            pptp
            sip
            sqlnet
            tftp
        }
    }
    console {
        device ttyS0 {
            speed 115200
        }
    }
    host-name bsp-asbr2-cm
    login {
        user vyos {
            authentication {
                encrypted-password ****************
            }
        }
    }
    name-server 114.114.114.114
    syslog {
        global {
            facility all {
                level info
            }
            facility protocols {
                level debug
            }
        }
    }
    time-zone Asia/Shanghai
}
vpn {
    ipsec {
        esp-group MyESPGroup {
            proposal 1 {
                encryption aes128
                hash aes128gmac
            }
        }
        ike-group MyIKEGroup {
            proposal 1 {
                dh-group 2
                encryption aes128
                hash sha1
            }
        }
        interface eth0
        site-to-site {
            peer JXNCCT {
                authentication {
                    local-id cdslcm.ras.meit.su
                    mode rsa
                    remote-id zion.lv2.pw
                    rsa {
                        local-key ****************
                        remote-key ****************
                    }
                }
                connection-type respond
                default-esp-group MyESPGroup
                ike-group MyIKEGroup
                local-address XXX.XXX.XX.100
                remote-address any
                tunnel 1 {
                    local {
                        prefix 192.168.127.32/32
                    }
                    remote {
                        prefix 192.168.63.32/32
                    }
                }
            }
        }
    }
}
vrf {
    name cm_up {
        protocols {
            static {
                route 0.0.0.0/0 {
                    next-hop XXX.XXX.XX.1 {
                    }
                }
            }
        }
        table 101
    }
    name experimental {
        protocols {
            static {
                route 0.0.0.0/0 {
                    next-hop XXX.XXX.XX.1 {
                        vrf cm_up
                    }
                }
            }
        }
        table 1000
    }
}
vyos@bsp-asbr2-cm:~$

In T3655#143947, @fernando wrote:

it doesn't seem the same problem as here, this logic that was applied over this version was vrf not on the table . Could you share full configuration ? there is some point over vrfs / vrf default /leaking that are not clear. So I can replicate the scenery and we see what is going on .

Hi, can you reproduce this behavior in lab?

diodep mentioned this in T5116: Better VRF support.Mar 28 2023, 7:24 AM

In T3655#143947, @fernando wrote:

it doesn't seem the same problem as here, this logic that was applied over this version was vrf not on the table . Could you share full configuration ? there is some point over vrfs / vrf default /leaking that are not clear. So I can replicate the scenery and we see what is going on .

I don't know how to tell Linux NAT properly NAT between VRFs, maybe we should check nftable rule generator by dump rules...

Btw, can VyOS support cone NAT?

SrividyaA added a subscriber: SrividyaA.May 9 2023, 5:15 PM

trilom added a subscriber: trilom.May 10 2023, 7:09 PM

In T3655#131502, @Viacheslav wrote:

I have NAT working with vrf in VyOS 1.4-rolling-202208290458 + custom nat offload

set interfaces ethernet eth0 address '192.168.122.14/24'
set interfaces ethernet eth1 address '192.0.2.1/24'
set interfaces ethernet eth1 vrf 'foo'
set protocols static route 192.0.2.0/24 interface eth1 vrf 'foo'
set system conntrack
set vrf name foo protocols static route 0.0.0.0/0 next-hop 192.168.122.1 interface 'eth0'
set vrf name foo protocols static route 0.0.0.0/0 next-hop 192.168.122.1 vrf 'default'
set vrf name foo table '1010'

Nftables

root@r14:/home/vyos# cat nat.nft 
flush ruleset

table ip filter {
	flowtable fastnat {
		hook ingress priority filter
		devices = { eth0, eth1 }
	}

	chain forward {
		type filter hook forward priority filter; policy accept;
		ip protocol { tcp, udp } flow add @fastnat
	}
}
table ip nat {
	chain POSTROUTING {
		type nat hook postrouting priority srcnat; policy accept;
		ip saddr 192.0.2.0/24 oif "eth0" snat to 192.168.122.14 persistent
	}

	chain PREROUTING {
		type nat hook prerouting priority dstnat; policy accept;
	}
}

Conntrack table

vyos@r14:~$ sudo conntrack -F
conntrack v1.4.6 (conntrack-tools): connection tracking table has been emptied.
vyos@r14:~$ 
vyos@r14:~$ sudo conntrack -L
tcp      6 431999 ESTABLISHED src=192.168.122.14 dst=192.168.122.1 sport=22 dport=44462 src=192.168.122.1 dst=192.168.122.14 sport=44462 dport=22 [ASSURED] mark=0 use=1
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=33018 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=33018 [OFFLOAD] mark=0 use=2
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=37517 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=37517 [OFFLOAD] mark=0 use=2
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=59794 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=59794 [OFFLOAD] mark=0 use=2
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=39288 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=39288 [OFFLOAD] mark=0 use=2
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=39616 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=39616 [OFFLOAD] mark=0 use=2
icmp     1 29 src=192.0.2.2 dst=1.1.1.1 type=8 code=0 id=12387 src=1.1.1.1 dst=192.168.122.14 type=0 code=0 id=12387 mark=0 use=1
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=41155 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=41155 [OFFLOAD] mark=0 use=2
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=39829 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=39829 [OFFLOAD] mark=0 use=2
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=33655 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=33655 [OFFLOAD] mark=0 use=2
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=44835 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=44835 [OFFLOAD] mark=0 use=2
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=40213 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=40213 [OFFLOAD] mark=0 use=2
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=33729 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=33729 [OFFLOAD] mark=0 use=2
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=48344 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=48344 [OFFLOAD] mark=0 use=2
conntrack v1.4.6 (conntrack-tools): 14 flow entries have been shown.
vyos@r14:~$

This works for me too on current rolling releases from Jan-2023 to now.

In T3655#148609, @bbabich wrote:

In T3655#131502, @Viacheslav wrote:

I have NAT working with vrf in VyOS 1.4-rolling-202208290458 + custom nat offload

set interfaces ethernet eth0 address '192.168.122.14/24'
set interfaces ethernet eth1 address '192.0.2.1/24'
set interfaces ethernet eth1 vrf 'foo'
set protocols static route 192.0.2.0/24 interface eth1 vrf 'foo'
set system conntrack
set vrf name foo protocols static route 0.0.0.0/0 next-hop 192.168.122.1 interface 'eth0'
set vrf name foo protocols static route 0.0.0.0/0 next-hop 192.168.122.1 vrf 'default'
set vrf name foo table '1010'

Nftables

root@r14:/home/vyos# cat nat.nft 
flush ruleset

table ip filter {
	flowtable fastnat {
		hook ingress priority filter
		devices = { eth0, eth1 }
	}

	chain forward {
		type filter hook forward priority filter; policy accept;
		ip protocol { tcp, udp } flow add @fastnat
	}
}
table ip nat {
	chain POSTROUTING {
		type nat hook postrouting priority srcnat; policy accept;
		ip saddr 192.0.2.0/24 oif "eth0" snat to 192.168.122.14 persistent
	}

	chain PREROUTING {
		type nat hook prerouting priority dstnat; policy accept;
	}
}

Conntrack table

vyos@r14:~$ sudo conntrack -F
conntrack v1.4.6 (conntrack-tools): connection tracking table has been emptied.
vyos@r14:~$ 
vyos@r14:~$ sudo conntrack -L
tcp      6 431999 ESTABLISHED src=192.168.122.14 dst=192.168.122.1 sport=22 dport=44462 src=192.168.122.1 dst=192.168.122.14 sport=44462 dport=22 [ASSURED] mark=0 use=1
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=33018 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=33018 [OFFLOAD] mark=0 use=2
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=37517 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=37517 [OFFLOAD] mark=0 use=2
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=59794 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=59794 [OFFLOAD] mark=0 use=2
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=39288 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=39288 [OFFLOAD] mark=0 use=2
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=39616 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=39616 [OFFLOAD] mark=0 use=2
icmp     1 29 src=192.0.2.2 dst=1.1.1.1 type=8 code=0 id=12387 src=1.1.1.1 dst=192.168.122.14 type=0 code=0 id=12387 mark=0 use=1
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=41155 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=41155 [OFFLOAD] mark=0 use=2
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=39829 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=39829 [OFFLOAD] mark=0 use=2
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=33655 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=33655 [OFFLOAD] mark=0 use=2
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=44835 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=44835 [OFFLOAD] mark=0 use=2
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=40213 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=40213 [OFFLOAD] mark=0 use=2
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=33729 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=33729 [OFFLOAD] mark=0 use=2
udp      17 src=192.0.2.2 dst=1.1.1.1 sport=48344 dport=53 src=1.1.1.1 dst=192.168.122.14 sport=53 dport=48344 [OFFLOAD] mark=0 use=2
conntrack v1.4.6 (conntrack-tools): 14 flow entries have been shown.
vyos@r14:~$

This works for me too on current rolling releases from Jan-2023 to now.

But I don't want to leak VRF foo's route to the public...

I've done test , regarding the original issues that it was nat+route-leaking (default + foo) , which is working on the last rolling (VyOS 1.4-rolling-202305140317). however, I've tried some test using two vrf+route-leaking and NAT , I can replicated the issue:

set interfaces ethernet eth0 address '172.16.50.12/24'
set interfaces ethernet eth0 description 'RED'
set interfaces ethernet eth0 vrf 'RED'
set interfaces ethernet eth1 address '192.168.130.100/24'
set interfaces ethernet eth1 description 'BLUE'
set interfaces ethernet eth1 vrf 'BLUE'

set nat destination rule 110 description 'NAT ssh- INSIDE'
set nat destination rule 110 destination port '2022'
set nat destination rule 110 inbound-interface 'eth0'
set nat destination rule 110 protocol 'tcp'
set nat destination rule 110 translation address '192.168.130.40'
set nat source rule 100 outbound-interface 'eth0'
set nat source rule 100 protocol 'all'
set nat source rule 100 source address '192.168.130.0/24'
set nat source rule 100 translation address 'masquerade'
set vrf bind-to-all
set vrf name BLUE protocols static route 0.0.0.0/0 next-hop 172.16.50.1 vrf 'RED'
set vrf name BLUE protocols static route 172.16.50.0/24 interface eth0 vrf 'RED'
set vrf name BLUE table '1010'
set vrf name RED protocols static route 0.0.0.0/0 next-hop 172.16.50.1
set vrf name RED protocols static route 192.168.130.0/24 interface eth1 vrf 'BLUE'
set vrf name RED table '2020'

@diodep Cloud you try the following command ? sudo nft flush table inet vrf_zones , it should solve this behavior :

vyos@vyos-rt1:~$ sudo conntrack -L
icmp     1 29 src=192.168.130.40 dst=8.8.8.8 type=8 code=0 id=59272 src=8.8.8.8 dst=172.16.50.12 type=0 code=0 id=59272 mark=0 use=1

}                sudo conntrack -L
tcp      6 431989 ESTABLISHED src=172.16.50.9 dst=172.16.50.12 sport=34044 dport=2022 src=192.168.130.40 dst=172.16.50.9 sport=2022 dport=34044 [ASSURED] mark=0 helper=tns use=1
conntrack v1.4.6 (conntrack-tools): 1 flow entries have been shown.


table ip vyos_nat { # handle 26
	chain PREROUTING { # handle 1
		type nat hook prerouting priority dstnat; policy accept;
		counter packets 1 bytes 60 jump VYOS_PRE_DNAT_HOOK # handle 5
		iifname "eth0" tcp dport 2022 counter packets 1 bytes 60 dnat to 192.168.130.40 comment "DST-NAT-110" # handle 6
	}

Arc771 added a subscriber: Arc771.Jun 4 2023, 9:48 PM

I have been able to get NAT to work with VRFs with 1.4-rolling-202306080317. However:

Applying the NAT rule to a running system only half worked: 1a) Traffic was being forwarded and the source address is rewritten appropriately. 1b) Replies come in to the NAT'd source address, but they are never forwarded to the flow originator.
After rebooting the router everything worked fine.

*edit*
It turns out this only worked when doing NAT between VRFs on the same router. It was not working for addresses on a remote router in the same L3VPN. For that, like others, I had to run nft flush table inet vrf_zones

I'm also encountering this issue on the latest rolling release.

@fernando - I can also confirm that the nft flush table inet vrf_zones resolves this issue (until reboot, of course)

mertuo added a subscriber: mertuo.Jul 7 2023, 7:42 AM

jvoss added a subscriber: jvoss.Jul 24 2023, 1:41 PM

tuxnet added a subscriber: tuxnet.Sep 1 2023, 5:37 AM

I created a PR to fix this issue by using direction parameter of conntrack zones: https://github.com/vyos/vyos-1x/pull/2236
I have a very basic VRF setup and it works fine. It would be much appreciated if someone could test this with more complex VRF setup.

You can test this approach on a running VyOS router using following commands:

nft -s list table inet vrf_zones > test.nft
sed -E 's/zone set/original zone set/g' ./test.nft > test2.nft
nft delete table inet vrf_zones
nft -f test2.nft

@vfreex the referenced netfilter patch is from 2015, is that really valid for current version thats included in the Linux 6.1 LTS kernel?

If not, how come this wasnt merged already by netfilter team?

@Apachez I am running kernel 6.1.49-amd64-vyos and this works fine with my local setup.
The patch is already in linux kernel since at least 4.3 (you can confirm with https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/log/include/net/netfilter/nf_conntrack_zones.h?h=linux-4.3.y), but it was added to nft command only since Feb 2017: https://git.netfilter.org/nftables/commit/src/ct.c?id=ed66d9966294a3bab6c8611e369861ba57374743

Oh sorry, I missed that this commit was for LTS 1.3.x series.

But isnt the Linux kernel in 1.3.3 newer than 4.3?

As in this commit isnt needed since its already part of the linux kernel?

The 1.4 series uses current Linux 6.1 LTS (6.1.52 as of writing).

@vfreex I've tested in my labs related this issues , I can confirm that it work as expected . this original zone solved the problem when there was a src-nat /dst-nat with different VRFs or leaking with them ,Thanks you for this contribution .

vyos with different VRFs and src/dst nat :

`
# nft tables : 
table inet vrf_zones { # handle 24
	map ct_iface_map { # handle 3
		typeof iifname : ct zone
		elements = { "RED" : 2020,
			     "eth0" : 2020,
			     "eth1" : 1010,
			     "BLUE" : 1010 }
	}

	chain vrf_zones_ct_in { # handle 1
		type filter hook prerouting priority raw; policy accept;
		counter packets 328 bytes 44624 ct original zone set iifname map @ct_iface_map # handle 4
	}

	chain vrf_zones_ct_out { # handle 2
		type filter hook output priority raw; policy accept;
		counter packets 10 bytes 840 ct original zone set oifname map @ct_iface_map # handle 5
	}
}

conntrack matching with this criteria and established the connections :

vyos@vyos-rt1:~$  sudo conntrack -L
tcp      6 431994 ESTABLISHED src=192.168.0.40 dst=172.16.50.12 sport=53794 dport=2022 zone-orig=2020 src=192.168.130.40 dst=192.168.0.40 sport=2022 dport=53794 [ASSURED] mark=0 helper=tns use=1
conntrack v1.4.6 (conntrack-tools): 1 flow entries have been shown.
vyos@vyos-rt1:~$  sudo conntrack -L
icmp     1 29 src=192.168.130.40 dst=8.8.8.8 type=8 code=0 id=14003 zone-orig=1010 src=8.8.8.8 dst=172.16.50.12 type=0 code=0 id=14003 mark=0 use=1
tcp      6 431981 ESTABLISHED src=192.168.0.40 dst=172.16.50.12 sport=53794 dport=2022 zone-orig=2020 src=192.168.130.40 dst=192.168.0.40 sport=2022 dport=53794 [ASSURED] mark=0 helper=tns use=1
conntrack v1.4.6 (conntrack-tools): 2 flow entries have been shown.

@c-po I've tested on 1.4 and 1.5 , so we can merge it 1.4 to mitigate this problem .

fernando changed the task status from In progress to Backport candidate.Sep 12 2023, 4:17 PM

fernando changed Version from 1.3.0-rc4 to 1.4 .

fernando set Issue type to Unspecified (please specify).

command on 1.5 :

vyos@vyos-rt1:~$ sudo conntrack -L
icmp     1 1 src=192.168.130.40 dst=8.8.8.8 type=8 code=0 id=14003 zone-orig=1010 src=8.8.8.8 dst=172.16.50.12 type=0 code=0 id=14003 mark=0 use=1
icmp     1 10 src=192.168.130.40 dst=8.8.8.8 type=8 code=0 id=16813 zone-orig=1010 src=8.8.8.8 dst=172.16.50.12 type=0 code=0 id=16813 mark=0 use=1
conntrack v1.4.6 (conntrack-tools): 2 flow entries have been shown.
vyos@vyos-rt1:~$ sudo conntrack -L
tcp      6 431994 ESTABLISHED src=192.168.0.40 dst=172.16.50.12 sport=57907 dport=2022 zone-orig=2020 src=192.168.130.40 dst=192.168.0.40 sport=2022 dport=57907 [ASSURED] mark=0 helper=tns use=1
conntrack v1.4.6 (conntrack-tools): 1 flow entries have been shown.

vyos@vyos-rt1:~$ show version
Version:          VyOS 1.5-rolling-202309120021
Release train:    current

Built by:         [email protected]
Built on:         Tue 12 Sep 2023 01:43 UTC
Build UUID:       ed2a4347-7f84-4042-a395-fc7e0fee9988
Build commit ID:  0f75e12dd1af2f

Architecture:     x86_64
Boot via:         installed image
System type:      KVM guest

Hardware vendor:  QEMU
Hardware model:   Standard PC (i440FX + PIIX, 1996)
Hardware S/N:
Hardware UUID:    09ee26dd-c103-4243-8997-0fe4bbcf894f

Copyright:        VyOS maintainers and contributors

fernando changed the task status from Backport candidate to Needs testing.Sep 12 2023, 6:59 PM

fernando added a project: VyOS 1.5 Circinus.

fernando changed Version from 1.4 to VyOS 1.4 / VyOS 1.5.

fernando changed Is it a breaking change? from Unspecified (possibly destroys the router) to Perfectly compatible.

fernando changed Issue type from Unspecified (please specify) to Bug (incorrect behavior).

@fernando This is really nice. Thank you for the testing!

Can we close it?

for me , it's ok . I didn't see another issue related it . we can close

@rherold Could you re-check it?

fernando closed this task as Resolved.Oct 4 2023, 7:54 PM

table inet vrf_zones {
        map ct_iface_map {
                typeof iifname : ct zone
                elements = { "HE" : 132,
                             "WAN" : 128,
                             "eth0" : 128,
                             "tun0" : 132,
                             "eth1" : 256,
                             "eth2" : 384,
                             "veth0" : 132,
                             "veth1" : 256,
                             "VMNET" : 256,
                             "FASTNETMON" : 384 }
        }

        chain vrf_zones_ct_in {
                type filter hook prerouting priority raw; policy accept;
                counter packets 37682 bytes 9857007 ct original zone set iifname map @ct_iface_map
        }

        chain vrf_zones_ct_out {
                type filter hook output priority raw; policy accept;
                counter packets 10822 bytes 1502078 ct original zone set oifname map @ct_iface_map
        }
}

this table doesn't work for IPv6:

vyos@vyos:~$ sudo ip vrf exec VMNET ping 198.18.5.0
PING 198.18.5.0 (198.18.5.0) 56(84) bytes of data.
64 bytes from 198.18.5.0: icmp_seq=1 ttl=64 time=0.070 ms
^C
--- 198.18.5.0 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.070/0.070/0.070/0.000 ms
vyos@vyos:~$ sudo ip vrf exec VMNET ping6 2001:470:1f15:1ed:1::1
PING 2001:470:1f15:1ed:1::1(2001:470:1f15:1ed:1::1) 56 data bytes
^C
--- 2001:470:1f15:1ed:1::1 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1039ms

deleting the table fixes it:

vyos@vyos:~$ sudo nft delete table inet vrf_zones
vyos@vyos:~$ sudo ip vrf exec VMNET ping6 2001:470:1f15:1ed:1::1
PING 2001:470:1f15:1ed:1::1(2001:470:1f15:1ed:1::1) 56 data bytes
64 bytes from 2001:470:1f15:1ed:1::1: icmp_seq=1 ttl=64 time=0.063 ms
64 bytes from 2001:470:1f15:1ed:1::1: icmp_seq=2 ttl=64 time=0.051 ms
^C
--- 2001:470:1f15:1ed:1::1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1014ms
rtt min/avg/max/mdev = 0.051/0.057/0.063/0.006 ms

I'm at a bit of a loss for how to "fix" this without having to delete the table, maybe the point here arises from the fact that I'm simply connecting two vrfs together with a veth pair:

set interfaces virtual-ethernet veth0 peer-name veth1
set interfaces virtual-ethernet veth1 peer-name veth0
set interfaces virtual-ethernet veth0 address 2001:470:1f15:1ed:1::1/80
set interfaces virtual-ethernet veth1 address 2001:470:1f15:1ed:1::2/80
set interfaces virtual-ethernet veth0 address 198.18.5.0/23
set interfaces virtual-ethernet veth1 address 198.18.4.1/23
set interfaces virtual-ethernet veth0 vrf HE
set interfaces virtual-ethernet veth1 vrf VMNET

vyos@vyos# sudo ip vrf exec VMNET ping6 2001:470:1f15:1ed:1::1
PING 2001:470:1f15:1ed:1::1(2001:470:1f15:1ed:1::1) 56 data bytes
^C
--- 2001:470:1f15:1ed:1::1 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1033ms

[edit]
vyos@vyos# sudo nft delete table inet vrf_zones
[edit]
vyos@vyos# sudo ip vrf exec VMNET ping6 2001:470:1f15:1ed:1::1
PING 2001:470:1f15:1ed:1::1(2001:470:1f15:1ed:1::1) 56 data bytes
64 bytes from 2001:470:1f15:1ed:1::1: icmp_seq=1 ttl=64 time=0.063 ms
64 bytes from 2001:470:1f15:1ed:1::1: icmp_seq=2 ttl=64 time=0.040 ms
^C
--- 2001:470:1f15:1ed:1::1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1053ms
rtt min/avg/max/mdev = 0.040/0.051/0.063/0.011 ms
[edit]
vyos@vyos#

don't know why really the table class inet should apply but I'm not sure that's the problem, I notice theres a lot of ip6 notrack verdicts throughout the whole ruleset but I wasn't able to isolate it to being the exact cause just that I can delete the vrf_zone table and it works. I don't really understand the concept of firewall zones in VyOS and I wasn't able to arrive at a workaround by virtue of anything that I tried with the firewall zone configuration.

EDIT:
The only thing I kinda understood about firewall zones was the local-zone option as it pertains to traffic going directly to the router and that kind of makes sense here because that's what I'm doing but ultimately the point of this is for one VRF's BGP instance to leak routes to another; so the BGP instances have to act as neighbors and some connectivity and there needs to be connectivity for next-hop routing

@paigeadelethompson it is another issue, open a separate bug report

paigeadelethompson mentioned this in T6097: vrf_zones blocking ipv6 traffic.Mar 4 2024, 6:20 PM

	F2928346: Снимок экрана от 2022-07-11 14-48-25.png
	Jul 11 2022, 11:49 AM

	F1499926: config
	Jun 28 2021, 2:42 PM

NAT Problem with VRFClosed, ResolvedPublicBUGActions

Description

Details

Related Objects

Event Timeline

NAT Problem with VRF
Closed, ResolvedPublicBUG
Actions