Problem is because of LSP MTU missmatch

Kernel pipeline merged

Maybe I found the issue:

@rob it fixed in the latest 1.3 with commit https://github.com/vyos/vyos-1x/commit/c7430fbb8738d76e63a6972b7399fa39572e2254
probably just not hit at that time in 1.3-rc4

this error also occure in VyOS 1.3 RC4.

Implemented, wait for approval

Don't think this is related to the i40e driver. I think it's because vyos sees

vyos@vyos# set interfaces ethernet eth3 description Test
[edit]
vyos@vyos# commit
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth3'
DEBUG/IFCONFIG returned (out):
[{"ifindex":5,"ifname":"eth3","flags":["BROADCAST","MULTICAST","UP","LOWER_UP"],"mtu":1500,"qdisc":"mq","operstate":"UP","linkmode":"DEFAULT ","group":"default","txqlen":1000,"link_type":"ether","address":"3c:fd:fe:dd:bd:89","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min_mtu ":68,"max_mtu":9702,"inet6_addr_gen_mode":"none","num_tx_queues":16,"num_rx_queues":16,"gso_max_size":65536,"gso_max_segs":65535,"phys_port_ id":"3cfdfeddbd89","vfinfo_list":[]}]
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth3'
DEBUG/IFCONFIG returned (out):
[{"ifindex":5,"ifname":"eth3","flags":["BROADCAST","MULTICAST","UP","LOWER_UP"],"mtu":1500,"qdisc":"mq","operstate":"UP","linkmode":"DEFAULT ","group":"default","txqlen":1000,"link_type":"ether","address":"3c:fd:fe:dd:bd:89","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min_mtu ":68,"max_mtu":9702,"inet6_addr_gen_mode":"none","num_tx_queues":16,"num_rx_queues":16,"gso_max_size":65536,"gso_max_segs":65535,"phys_port_ id":"3cfdfeddbd89","vfinfo_list":[]}]
DEBUG/IFCONFIG cmd 'ethtool --show-pause eth3'
DEBUG/IFCONFIG returned (out):
Pause parameters for eth3:
Autonegotiate: off
RX: off
TX: off
DEBUG/IFCONFIG cmd 'ethtool --pause eth3 autoneg on tx on rx on'
could not set flowcontrol for eth3
DEBUG/IFCONFIG cmd 'ethtool -K eth3 gro off'
DEBUG/IFCONFIG cmd 'ethtool -K eth3 gso off'
DEBUG/IFCONFIG write '0' > '/sys/class/net/eth3/queues/rx-0/rps_cpus'
DEBUG/IFCONFIG cmd 'ethtool -K eth3 sg off'
DEBUG/IFCONFIG cmd 'ethtool -K eth3 tso off'
DEBUG/IFCONFIG cmd 'ethtool -K eth3 ufo off'
DEBUG/IFCONFIG cmd 'ethtool eth3'
DEBUG/IFCONFIG returned (out):
Settings for eth3:

Supported ports: [ FIBRE ]
Supported link modes:   1000baseX/Full
                        10000baseSR/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes:  1000baseX/Full
                        10000baseSR/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Port: FIBRE
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
                       drv probe link
Link detected: yes

DEBUG/IFCONFIG cmd 'ethtool -s eth3 autoneg on'
{'description': 'Test',
'duplex': 'auto',
'ifname': 'eth3',
'ip': {'arp_cache_timeout': '30'},
'mtu': '1500',
'speed': 'auto',
'vif': {'80': {'address': ['1.3.3.7/29'],

'ip': {'arp_cache_timeout': '30'},
'mtu': '1500'}}}

DEBUG/IFCONFIG cmd 'ip link set dev eth3 alias "Test"'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv4/conf/eth3/link_filter'
DEBUG/IFCONFIG cmd 'sudo systemctl stop dhclient@eth3.service'
DEBUG/IFCONFIG cmd 'sudo systemctl stop dhcp6c@eth3.service'
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth3'
DEBUG/IFCONFIG returned (out):
[{"ifindex":5,"ifname":"eth3","flags":["NO-CARRIER","BROADCAST","MULTICAST","UP"],"mtu":1500,"qdisc":"mq","operstate":"DOWN","linkmode":"DEF AULT","group":"default","txqlen":1000,"link_type":"ether","address":"3c:fd:fe:dd:bd:89","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min _mtu":68,"max_mtu":9702,"inet6_addr_gen_mode":"none","num_tx_queues":16,"num_rx_queues":16,"gso_max_size":65536,"gso_max_segs":65535,"phys_p ort_id":"3cfdfeddbd89","ifalias":"Test","vfinfo_list":[]}]
DEBUG/IFCONFIG cmd 'ip link set dev eth3 nomaster'
DEBUG/IFCONFIG write '30000' > '/proc/sys/net/ipv4/neigh/eth3/base_reachable_time_ms'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv4/conf/eth3/arp_filter'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3/arp_accept'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3/arp_announce'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3/arp_ignore'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3/proxy_arp'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3/proxy_arp_pvlan'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv4/conf/eth3/forwarding'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3/rp_filter'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv6/conf/eth3/forwarding'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv6/conf/eth3/accept_ra'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv6/conf/eth3/autoconf'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv6/conf/eth3/dad_transmits'
DEBUG/IFCONFIG cmd 'ip link set dev eth3 mtu 1500'
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth3'
DEBUG/IFCONFIG returned (out):
[{"ifindex":5,"ifname":"eth3","flags":["NO-CARRIER","BROADCAST","MULTICAST","UP"],"mtu":1500,"qdisc":"mq","operstate":"DOWN","linkmode":"DEF AULT","group":"default","txqlen":1000,"link_type":"ether","address":"3c:fd:fe:dd:bd:89","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min _mtu":68,"max_mtu":9702,"inet6_addr_gen_mode":"none","num_tx_queues":16,"num_rx_queues":16,"gso_max_size":65536,"gso_max_segs":65535,"phys_p ort_id":"3cfdfeddbd89","ifalias":"Test","vfinfo_list":[]}]
DEBUG/IFCONFIG cmd 'xdp_loader -d eth3 -U --auto-mode'
DEBUG/IFCONFIG returned (out):
INFO: xdp_link_detach() no curr XDP prog on ifindex:5
DEBUG/IFCONFIG cmd 'tc qdisc del dev eth3 handle ffff: ingress 2> /dev/null;tc qdisc del dev eth3 handle 1: root prio 2> /dev/null;set $?=0'
DEBUG/IFCONFIG cmd 'ip link set dev eth3 up'
{'address': ['1.3.3.7/29'],
'ifname': 'eth3.80',
'ip': {'arp_cache_timeout': '30'},
'mtu': '1500'}
DEBUG/IFCONFIG cmd 'ip link set dev eth3.80 alias ""'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv4/conf/eth3.80/link_filter'
DEBUG/IFCONFIG cmd 'sudo systemctl stop dhclient@eth3.80.service'
DEBUG/IFCONFIG cmd 'sudo systemctl stop dhcp6c@eth3.80.service'
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth3.80'
DEBUG/IFCONFIG returned (out):
[{"ifindex":8,"link":"eth3","ifname":"eth3.80","flags":["NO-CARRIER","BROADCAST","MULTICAST","UP"],"mtu":1500,"qdisc":"noqueue","operstate": "LOWERLAYERDOWN","linkmode":"DEFAULT","group":"default","txqlen":1000,"link_type":"ether","address":"3c:fd:fe:dd:bd:89","broadcast":"ff:ff:f f:ff:ff:ff","promiscuity":0,"min_mtu":0,"max_mtu":65535,"linkinfo":{"info_kind":"vlan","info_data":{"protocol":"802.1Q","id":80,"flags":["RE ORDER_HDR"]}},"inet6_addr_gen_mode":"none","num_tx_queues":1,"num_rx_queues":1,"gso_max_size":65536,"gso_max_segs":65535}]
DEBUG/IFCONFIG cmd 'ip link set dev eth3.80 nomaster'
DEBUG/IFCONFIG write '30000' > '/proc/sys/net/ipv4/neigh/eth3.80/base_reachable_time_ms'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv4/conf/eth3.80/arp_filter'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3.80/arp_accept'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3.80/arp_announce'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3.80/arp_ignore'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3.80/proxy_arp'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3.80/proxy_arp_pvlan'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv4/conf/eth3.80/forwarding'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3.80/rp_filter'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv6/conf/eth3.80/forwarding'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv6/conf/eth3.80/accept_ra'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv6/conf/eth3.80/autoconf'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv6/conf/eth3.80/dad_transmits'
DEBUG/IFCONFIG cmd 'ip link set dev eth3.80 mtu 1500'
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth3.80'
DEBUG/IFCONFIG returned (out):
[{"ifindex":8,"link":"eth3","ifname":"eth3.80","flags":["NO-CARRIER","BROADCAST","MULTICAST","UP"],"mtu":1500,"qdisc":"noqueue","operstate": "LOWERLAYERDOWN","linkmode":"DEFAULT","group":"default","txqlen":1000,"link_type":"ether","address":"3c:fd:fe:dd:bd:89","broadcast":"ff:ff:f f:ff:ff:ff","promiscuity":0,"min_mtu":0,"max_mtu":65535,"linkinfo":{"info_kind":"vlan","info_data":{"protocol":"802.1Q","id":80,"flags":["RE ORDER_HDR"]}},"inet6_addr_gen_mode":"none","num_tx_queues":1,"num_rx_queues":1,"gso_max_size":65536,"gso_max_segs":65535}]
DEBUG/IFCONFIG cmd 'xdp_loader -d eth3.80 -U --auto-mode'
DEBUG/IFCONFIG returned (out):
INFO: xdp_link_detach() no curr XDP prog on ifindex:8
DEBUG/IFCONFIG cmd 'ip link set dev eth3.80 up'

One solution for it:

vyos_bld@3c88687662fe:/vyos/work/T3516/vyatta-cfg-quagga$ git diff
diff --git a/templates/protocols/static/interface-route/node.tag/next-hop-interface/node.def b/templates/protocols/static/interface-route/node.tag/next-hop-interface/node.def
index f8bc9270..991a30ee 100644
--- a/templates/protocols/static/interface-route/node.tag/next-hop-interface/node.def
+++ b/templates/protocols/static/interface-route/node.tag/next-hop-interface/node.def
@@ -14,7 +14,10 @@ end:
       if ! ${vyatta_sbindir}/vyatta-next-hop-check $VAR(../@) ipv4 interface; then
         exit 1;
       fi
+      DIST=`cli-shell-api returnEffectiveValue protocols static interface-route $VAR(../@) next-hop-interface $VAR(@) distance`
+
       vtysh -c "configure terminal" \
+                   -c "no ip route $VAR(../@) $VAR(@) $DIST" \
                    -c "no ip route $VAR(../@) $VAR(@)"
     else
       if [[ -n "$VAR(./distance/@)" ]]; then
diff --git a/templates/protocols/static/route/node.tag/next-hop/node.def b/templates/protocols/static/route/node.tag/next-hop/node.def
index 0574781d..44a39bfb 100644
--- a/templates/protocols/static/route/node.tag/next-hop/node.def
+++ b/templates/protocols/static/route/node.tag/next-hop/node.def
@@ -16,10 +16,12 @@ end:
       if ! ${vyatta_sbindir}/vyatta-next-hop-check $VAR(../@) ipv4 address; then
         exit 1;
       fi
+      DIST=`cli-shell-api returnEffectiveValue protocols static route $VAR(../@) next-hop $VAR(@) distance`
       if ${vyatta_sbindir}/vyatta-gateway-static_route-check.pl \
           "$VAR(../@)" "$VAR(@)"
       then
         vtysh -c "configure terminal" \
+                     -c "no ip route $VAR(../@) $VAR(@) $DIST" \
                      -c "no ip route $VAR(../@) $VAR(@)"
       fi
     else

I have boiled down your commands to the following:

set interfaces ethernet eth0 hw-id 'ac:1f:6b:76:95:98'
set interfaces ethernet eth1 hw-id 'ac:1f:6b:76:95:99'
set interfaces ethernet eth2 hw-id 'ac:1f:6b:76:98:7a'
set interfaces ethernet eth3 hw-id 'ac:1f:6b:76:98:7b'
set interfaces ethernet eth4 hw-id '3c:fd:fe:dd:bd:88'
set interfaces ethernet eth5 hw-id '3c:fd:fe:dd:bd:89'
set interfaces ethernet eth5 vif 58 address 'x.x.x.x'
set interfaces ethernet eth5 vif 80 address 'x.x.x.x'
set interfaces ethernet eth5 vif 80 ip ospf cost '500'
set interfaces ethernet eth5 vif 80 ip ospf dead-interval '30'
set interfaces ethernet eth5 vif 81 address 'x.x.x.x'
set interfaces ethernet eth5 vif 81 ip ospf cost '1'
set interfaces ethernet eth5 vif 81 ip ospf dead-interval '30'
set interfaces ethernet eth5 vif 81 ip ospf network 'point-to-point'

Can't reproduce on VyOS 1.3.0-rc4, even after a reboot I get a single ipv6 route ::/0 2a01:9820:5::10 table 100 line

Forgot to include the version info

Hi @hard, I think behavior should be another for VM. Did you try to change speed when you connected via serial?

https://github.com/vyos/libpam-radius-auth/commit/cf571ca6c722d3d2b0c359dddf835a3f406b194b
https://github.com/vyos/libnss-mapuser/commit/2ded63a0e6748018dada646691843462797cc7fc

Checked on VM

vyos@vyos-dev# set system console device ttyS0 speed 115200
[edit]
vyos@vyos-dev# commit
[edit]
vyos@vyos-dev# show system console
 device ttyS0 {
     speed 115200
 }
[edit]

The 2001:19f0:ffff::1 is the neighbour, it is accessible via the Kernel route and I'm able to pull routes from it. The setup is multihop. But I think you are on something. I think the peer should return the gateway as the next hop and not itself. I'm experimenting with Vultr and their setup is questionable, so this is just one more thing that they messed up. Anyhow, thanks for your help. Feel free to close the ticket.

mii-mon was fine. Link was enabled on the switch and the link for the interface was up. But on the LACP was not enabled on the switch.
So I would exepect to see the sub interfaces from the bond as up but the bond has to be down.

I see no connected network in your FIB for 2001:19f0:ffff::1 I guess that's the reason.

Hmm, all other routes have longer prefixes, so they should take precedence. As far as I understand the priority is prefix length (longest first) -> admin distance -> metric. So, your explanation contradicts this unless I miss something.