Hi Everyone!
I’ve got an interesting one here. I’ve been rattling my brain with this one and have tried multiple ways to try to make this work. Here is a high level description of what I’m trying to achieve.
I’ve created a simplified virtual lab to describe what I’m experiencing, i’ve also tested this on the latest version of the VyOS 1.5 rolling release as well with the same outcome.
- LAN networks will use a specific default gateway for specific UDP traffic
- Any LAN network requiring DNS / TCP (http / https) traffic will use the vrf named “INTERNET”
- Using VETH interfaces to establish an iBGP connection between the vrfs “default” and “INTERNET”.
- I have a very simple PBR in this lab that will redirect all Internet traffic out of the Internet VRF
- ICMP ping works just fine reaching “8.8.8.8” or “1.1.1.2” but name resolution fails, and TCP fails too towards any url, in this I tried “ifconfig.me/ip”
VyOS 1.5 Stream Config:
```
set interfaces ethernet eth0 address '10.4.0.1/24'
set interfaces ethernet eth0 hw-id '0c:76:53:4a:00:00'
set interfaces ethernet eth1 hw-id '0c:76:53:4a:00:01'
set interfaces ethernet eth2 hw-id '0c:76:53:4a:00:02'
set interfaces ethernet eth3 hw-id '0c:76:53:4a:00:03'
set interfaces ethernet eth4 hw-id '0c:76:53:4a:00:04'
set interfaces ethernet eth5 hw-id '0c:76:53:4a:00:05'
set interfaces ethernet eth6 hw-id '0c:76:53:4a:00:06'
set interfaces ethernet eth7 hw-id '0c:76:53:4a:00:07'
set interfaces ethernet eth8 hw-id '0c:76:53:4a:00:08'
set interfaces ethernet eth9 hw-id '0c:76:53:4a:00:09'
set interfaces ethernet eth9 vif 35 vrf 'INTERNET'
set interfaces loopback lo
set interfaces pppoe pppoe0 authentication password '*****************'
set interfaces pppoe pppoe0 authentication username '*****************'
set interfaces pppoe pppoe0 description 'PPPoE'
set interfaces pppoe pppoe0 ip adjust-mss '1452'
set interfaces pppoe pppoe0 mru '1492'
set interfaces pppoe pppoe0 mtu '1492'
set interfaces pppoe pppoe0 no-peer-dns
set interfaces pppoe pppoe0 source-interface 'eth9.35'
set interfaces pppoe pppoe0 vrf 'INTERNET'
set interfaces virtual-ethernet veth0 address '10.255.255.0/31'
set interfaces virtual-ethernet veth0 peer-name 'veth1'
set interfaces virtual-ethernet veth1 address '10.255.255.1/31'
set interfaces virtual-ethernet veth1 peer-name 'veth0'
set interfaces virtual-ethernet veth1 vrf 'INTERNET'
set nat source rule 1 outbound-interface name 'pppoe0'
set nat source rule 1 source address '0.0.0.0/0'
set nat source rule 1 translation address 'masquerade'
set policy route PBR1 interface 'eth0'
set policy route PBR1 rule 20 action 'accept'
set policy route PBR1 rule 20 log
set policy route PBR1 rule 20 set table '150'
set policy route PBR1 rule 20 source address '10.4.0.0/24'
set protocols bgp address-family ipv4-unicast redistribute connected
set protocols bgp neighbor 10.255.255.1 address-family ipv4-unicast nexthop-self
set protocols bgp neighbor 10.255.255.1 remote-as '65000'
set protocols bgp system-as '65000'
set service ntp allow-client address '127.0.0.0/8'
set service ntp allow-client address '169.254.0.0/16'
set service ntp allow-client address '10.0.0.0/8'
set service ntp allow-client address '172.16.0.0/12'
set service ntp allow-client address '192.168.0.0/16'
set service ntp allow-client address '::1/128'
set service ntp allow-client address 'fe80::/10'
set service ntp allow-client address 'fc00::/7'
set service ntp server time1.vyos.net
set service ntp server time2.vyos.net
set service ntp server time3.vyos.net
set system config-management commit-revisions '100'
set system console device ttyS0 speed '115200'
set system host-name 'vyos'
set system login user vyos authentication encrypted-password '$6$rounds=656000$7e7g5/6I9crYHF6i$2Bd/TUkoMUDY66aeOrOfMdDG7KN6Rnu9KkxeqPhCryFcRp2DcGf5OjTAHJmqIK2C/pZJmDIbVhYpOCn4dHa9c.'
set system login user vyos authentication plaintext-password ''
set system syslog global facility all level 'info'
set system syslog global facility local7 level 'debug'
set vrf name INTERNET protocols bgp address-family ipv4-unicast redistribute connected
set vrf name INTERNET protocols bgp neighbor 10.255.255.0 address-family ipv4-unicast nexthop-self
set vrf name INTERNET protocols bgp neighbor 10.255.255.0 remote-as '65000'
set vrf name INTERNET protocols bgp system-as '65000'
set vrf name INTERNET table '150'
```
VyOS Routing Table:
```
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, F - PBR,
f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
t - trapped, o - offload failure
VRF INTERNET:
S>* 0.0.0.0/0 [210/0] is directly connected, pppoe0, weight 1, 00:09:21
B>* 10.4.0.0/24 [200/0] via 10.255.255.0, veth1, weight 1, 00:09:01
C>* 10.255.255.0/31 is directly connected, veth1, 00:10:30
C>* 142.124.37.123/32 is directly connected, pppoe0, 00:09:21
VRF default:
C>* 10.4.0.0/24 is directly connected, eth0, 00:22:03
C>* 10.255.255.0/31 is directly connected, veth0, 00:22:05
B>* 142.124.37.123/32 [200/0] via 10.255.255.1, veth0, weight 1, 00:09:01
[edit]
```
PBR Log (ping 1.1.1.2):
```
Apr 15 16:02:02 vyos kernel: [ 1488.870210] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=1.1.1.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=43612 DF PROTO=ICMP TYPE=8 CODE=0 ID=169 SEQ=1
Apr 15 16:02:03 vyos kernel: [ 1489.872255] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=1.1.1.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=43848 DF PROTO=ICMP TYPE=8 CODE=0 ID=169 SEQ=2
Apr 15 16:02:04 vyos kernel: [ 1490.874009] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=1.1.1.2 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=44340 DF PROTO=ICMP TYPE=8 CODE=0 ID=169 SEQ=3
```
PBR Log (curl ifconfig.me/ip)
```
Apr 15 16:03:28 vyos kernel: [ 1575.145498] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=1.1.1.2 LEN=57 TOS=0x00 PREC=0x00 TTL=64 ID=26673 DF PROTO=UDP SPT=40384 DPT=53 LEN=37
Apr 15 16:03:28 vyos kernel: [ 1575.145576] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=1.1.1.2 LEN=57 TOS=0x00 PREC=0x00 TTL=64 ID=26674 DF PROTO=UDP SPT=40384 DPT=53 LEN=37
Apr 15 16:03:33 vyos kernel: [ 1580.150448] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=1.1.1.2 LEN=57 TOS=0x00 PREC=0x00 TTL=64 ID=26675 DF PROTO=UDP SPT=40384 DPT=53 LEN=37
Apr 15 16:03:33 vyos kernel: [ 1580.150495] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=1.1.1.2 LEN=57 TOS=0x00 PREC=0x00 TTL=64 ID=26676 DF PROTO=UDP SPT=40384 DPT=53 LEN=37
```
PBR Log (ping google.ca)
```
Apr 15 16:02:50 vyos kernel: [ 1536.871047] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=1.1.1.2 LEN=55 TOS=0x00 PREC=0x00 TTL=64 ID=30648 DF PROTO=UDP SPT=56541 DPT=53 LEN=35
Apr 15 16:02:50 vyos kernel: [ 1536.871145] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=1.1.1.2 LEN=55 TOS=0x00 PREC=0x00 TTL=64 ID=30649 DF PROTO=UDP SPT=56541 DPT=53 LEN=35
Apr 15 16:02:55 vyos kernel: [ 1541.876231] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=1.1.1.2 LEN=55 TOS=0x00 PREC=0x00 TTL=64 ID=30650 DF PROTO=UDP SPT=56541 DPT=53 LEN=35
Apr 15 16:02:55 vyos kernel: [ 1541.876291] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=1.1.1.2 LEN=55 TOS=0x00 PREC=0x00 TTL=64 ID=30651 DF PROTO=UDP SPT=56541 DPT=53 LEN=35
```
Results from Sample Host (10.4.0.200):
```
root@UbuntuDockerGuest-4:~# ping 1.1.1.2 -c 3
PING 1.1.1.2 (1.1.1.2) 56(84) bytes of data.
64 bytes from 1.1.1.2: icmp_seq=1 ttl=54 time=5.68 ms
64 bytes from 1.1.1.2: icmp_seq=2 ttl=54 time=5.62 ms
64 bytes from 1.1.1.2: icmp_seq=3 ttl=54 time=6.97 ms
--- 1.1.1.2 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 5.618/6.089/6.974/0.626 ms
root@UbuntuDockerGuest-4:~# ping google.ca
ping: google.ca: Temporary failure in name resolution
root@UbuntuDockerGuest-4:~# curl ifconfig.me/ip
curl: (6) Could not resolve host: ifconfig.me
root@UbuntuDockerGuest-4:~#
```
Now if i ended up doing this:
```
set protocols static route 0.0.0.0/0 next-hop 10.255.255.1 int veth0
delete policy route PBR1 rule 20 set table '150'
```
DNS and TCP will work normally – but this defeats the purpose… i can’t add static routes for ALL individual tcp/udp ip addresses in the main routing table:
```
Apr 15 16:05:56 vyos kernel: [ 1723.070514] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=1.1.1.2 LEN=55 TOS=0x00 PREC=0x00 TTL=64 ID=52136 DF PROTO=UDP SPT=49392 DPT=53 LEN=35
Apr 15 16:05:56 vyos kernel: [ 1723.070606] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=1.1.1.2 LEN=55 TOS=0x00 PREC=0x00 TTL=64 ID=52137 DF PROTO=UDP SPT=49392 DPT=53 LEN=35
Apr 15 16:05:56 vyos kernel: [ 1723.081009] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=142.251.41.35 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=19104 DF PROTO=ICMP TYPE=8 CODE=0 ID=170 SEQ=1
Apr 15 16:05:56 vyos kernel: [ 1723.085917] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=1.1.1.2 LEN=72 TOS=0x00 PREC=0x00 TTL=64 ID=49296 DF PROTO=UDP SPT=48706 DPT=53 LEN=52
Apr 15 16:05:57 vyos kernel: [ 1724.082562] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=142.251.41.35 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=19275 DF PROTO=ICMP TYPE=8 CODE=0 ID=170 SEQ=2
Apr 15 16:05:58 vyos kernel: [ 1725.084504] [ipv4-route-PBR1-20-A]IN=eth0 OUT= MAC=0c:76:53:4a:00:00:e2:39:0a:ee:aa:aa:08:00 SRC=10.4.0.200 DST=142.251.41.35 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=19406 DF PROTO=ICMP TYPE=8 CODE=0 ID=170 SEQ=3
```
Also, worth noting, if i used `set protocols static table ## route` without using VRF’s i won’t run into this issue.
Any insight on this would be greatly appreciated!!!
I'm aware that VETH pairs aren't required for this, but I've tried SEVERAL options, the VETH pairs are only there for me to be able to properly do Intra-VRF routing locally at the VyOS level, without a next-hop IP Address in the routing table doesn't work as expected when it has only an exit interface.
But regardless of that I'm having the issue of TCP / UDP traffic flowing through a VRF.
Update: I added "VyOS Rolling" on this as well, because this issue is present in the Rolling Releases as well for 1.5, please remove it if you see fit!