NIC flap at any interface change
Closed, ResolvedPublicBUG
Actions

Assigned To

Authored By

	shyrwall
	May 4 2021, 9:33 AM

Description

HW: X710-DA2, installed directly, no hypervisor etc.

I have a problem where any interface change, even description, results in the nic being brought down and then up again. Possibly being related to not being able to set flowcontrol.Unfortunately I had to rollback to 1.2.7 on the specified server so I'm unable to debug any further.

Here is some logs,

[email protected]# set interfaces ethernet eth5 description Test
[edit]
[email protected]# commit
could not set flowcontrol for eth5
[edit]
[email protected]#

May 3 08:16:11 vyos vyos-configd[757]: config session pid is 5761
May 3 08:16:11 vyos vyos-configd[757]: Received message: {"type": "node", "data": "/usr/libexec/vyos/conf_mode/interfaces-ethernet.pyVYOS_TAGNODE_VALUE=eth5"}
May 3 08:16:11 vyos kernel: [ 863.781620] i40e 0000:05:00.1 eth5: To change autoneg please use: ethtool -s <dev> autoneg <on|off>
May 3 08:16:12 vyos kernel: [ 863.817177] i40e 0000:05:00.1 eth5: NIC Link is Down
May 3 08:16:12 vyos netplugd[1091]: eth5: state ACTIVE flags 0x00011043 UP,BROADCAST,RUNNING,MULTICAST,10000 -> 0x00001003 UP,BROADCAST,MULTICAST
May 3 08:16:12 vyos netplugd[1091]: eth5.58: state ACTIVE flags 0x00011043 UP,BROADCAST,RUNNING,MULTICAST,10000 -> 0x00001003 UP,BROADCAST,MULTICAST
May 3 08:16:12 vyos netplugd[5926]: /etc/netplug/netplug eth5 out -> pid 5926
May 3 08:16:12 vyos netplugd[1091]: eth5.80: state ACTIVE flags 0x00011043 UP,BROADCAST,RUNNING,MULTICAST,10000 -> 0x00001003 UP,BROADCAST,MULTICAST
May 3 08:16:12 vyos netplugd[1091]: eth5.81: state ACTIVE flags 0x00011043 UP,BROADCAST,RUNNING,MULTICAST,10000 -> 0x00001003 UP,BROADCAST,MULTICAST
May 3 08:16:12 vyos netplugd[5927]: /etc/netplug/netplug eth5.58 out -> pid 5927
May 3 08:16:12 vyos netplugd[5928]: /etc/netplug/netplug eth5.80 out -> pid 5928
May 3 08:16:12 vyos netplugd[5929]: /etc/netplug/netplug eth5.81 out -> pid 5929
May 3 08:16:12 vyos isisd[1239]: circuit already disconnected
May 3 08:16:12 vyos netplugd[1091]: eth5.81: state OUTING pid 5929 exited status 0
May 3 08:16:12 vyos netplugd[1091]: eth5: state OUTING pid 5926 exited status 0
May 3 08:16:12 vyos netplugd[1091]: eth5.58: state OUTING pid 5927 exited status 0
May 3 08:16:12 vyos netplugd[1091]: eth5.80: state OUTING pid 5928 exited status 0
May 3 08:16:12 vyos isisd[1239]: circuit already disconnected
May 3 08:16:12 vyos isisd[1239]: circuit already disconnected
May 3 08:16:12 vyos isisd[1239]: circuit already disconnected
May 3 08:16:12 vyos vyos-configd[757]: Sending response 1
May 3 08:16:12 vyos systemd[1]: opt-vyatta-config-tmp-new_config_5761.mount: Succeeded.
May 3 08:16:12 vyos systemd[3195]: opt-vyatta-config-tmp-new_config_5761.mount: Succeeded.
May 3 08:16:13 vyos kernel: [ 864.961037] i40e 0000:05:00.1 eth5: NIC Link is Up, 10 Gbps Full Duplex, Flow Control: None
May 3 08:16:13 vyos netplugd[1091]: eth5: state INACTIVE flags 0x00001003 UP,BROADCAST,MULTICAST -> 0x00011043 UP,BROADCAST,RUNNING,MULTICAST,10000
May 3 08:16:13 vyos netplugd[1091]: eth5.58: state INACTIVE flags 0x00001003 UP,BROADCAST,MULTICAST -> 0x00011043 UP,BROADCAST,RUNNING,MULTICAST,10000
May 3 08:16:13 vyos netplugd[1091]: eth5.80: state INACTIVE flags 0x00001003 UP,BROADCAST,MULTICAST -> 0x00011043 UP,BROADCAST,RUNNING,MULTICAST,10000
May 3 08:16:13 vyos netplugd[6018]: /etc/netplug/netplug eth5 in -> pid 6018
May 3 08:16:13 vyos netplugd[6019]: /etc/netplug/netplug eth5.58 in -> pid 6019
May 3 08:16:13 vyos netplugd[1091]: eth5.81: state INACTIVE flags 0x00001003 UP,BROADCAST,MULTICAST -> 0x00011043 UP,BROADCAST,RUNNING,MULTICAST,10000
May 3 08:16:13 vyos netplugd[6020]: /etc/netplug/netplug eth5.80 in -> pid 6020
May 3 08:16:13 vyos netplugd[6021]: /etc/netplug/netplug eth5.81 in -> pid 6021
May 3 08:16:13 vyos commit: Successful change to active configuration by user zib on /dev/pts/2

Details

Difficulty level: Normal (likely a few hours)
Version: vyos-1.3-rolling-202105011026-amd64 and vyos-1.4-rolling-202105020417-amd64
Why the issue appeared?: Will be filled on close
Is it a breaking change?: Unspecified (possibly destroys the router)
Issue type: Unspecified (please specify)

Related Objects

Mentioned Here: T2738: Modifying configuration in the "interfaces" section from VRRP transition scripts causes configuration lockup and high CPU utilization

Event Timeline

shyrwall created this task.May 4 2021, 9:33 AM

set interfaces ethernet eth0 hw-id 'ac:1f:6b:76:95:98'
set interfaces ethernet eth1 hw-id 'ac:1f:6b:76:95:99'
set interfaces ethernet eth2 hw-id 'ac:1f:6b:76:98:7a'
set interfaces ethernet eth3 hw-id 'ac:1f:6b:76:98:7b'
set interfaces ethernet eth4 hw-id '3c:fd:fe:dd:bd:88'
set interfaces ethernet eth5 hw-id '3c:fd:fe:dd:bd:89'
set interfaces ethernet eth5 vif 58 address 'x.x.x.x'
set interfaces ethernet eth5 vif 80 address 'x.x.x.x'
set interfaces ethernet eth5 vif 80 ip ospf cost '500'
set interfaces ethernet eth5 vif 80 ip ospf dead-interval '30'
set interfaces ethernet eth5 vif 81 address 'x.x.x.x'
set interfaces ethernet eth5 vif 81 ip ospf cost '1'
set interfaces ethernet eth5 vif 81 ip ospf dead-interval '30'
set interfaces ethernet eth5 vif 81 ip ospf network 'point-to-point'

c-po changed the task status from Open to In progress.May 4 2021, 5:56 PM

c-po claimed this task.

I have boiled down your commands to the following:

set interfaces ethernet eth2 vif 58 address '10.0.58.254/24'
set interfaces ethernet eth2 vif 80 address '10.0.80.254/24'
set interfaces ethernet eth2 vif 81 address '10.0.81.254/24'

and after applying this interface configuration I changed the description of eth2 set interfaces ethernet eth2 description test2 which resulted in the following happening on the system:

May  4 19:50:19 vyos DEBUG/IFCONFIG returned (err):
May  4 19:50:19 vyos Cannot change udp-fragmentation-offload
May  4 19:50:19 vyos isisd[889]: circuit already connected
May  4 19:50:19 vyos isisd[889]: circuit already connected
May  4 19:50:19 vyos isisd[889]: circuit already connected
May  4 19:50:20 vyos isisd[889]: circuit already connected
May  4 19:50:20 vyos systemd[1824]: opt-vyatta-config-tmp-new_config_4244.mount: Succeeded.
May  4 19:50:20 vyos systemd[1]: opt-vyatta-config-tmp-new_config_4244.mount: Succeeded.
May  4 19:50:21 vyos commit: Successful change to active configuration by user cpo on /dev/pts/0

I assume the problem could be with the famous i40e NIC driver.

Please re-run your commands but before typing commit enable debugging by running: touch /tmp/vyos.ifconfig.debug.

All was done using: VyOS 1.4 rolling 202105020417

c-po changed the task status from In progress to On hold.May 4 2021, 7:06 PM

c-po triaged this task as High priority.

c-po changed Difficulty level from Unknown (require assessment) to Normal (likely a few hours).

[email protected]# set interfaces ethernet eth3 description Test
[edit]
[email protected]# commit
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth3'
DEBUG/IFCONFIG returned (out):
[{"ifindex":5,"ifname":"eth3","flags":["BROADCAST","MULTICAST","UP","LOWER_UP"],"mtu":1500,"qdisc":"mq","operstate":"UP","linkmode":"DEFAULT ","group":"default","txqlen":1000,"link_type":"ether","address":"3c:fd:fe:dd:bd:89","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min_mtu ":68,"max_mtu":9702,"inet6_addr_gen_mode":"none","num_tx_queues":16,"num_rx_queues":16,"gso_max_size":65536,"gso_max_segs":65535,"phys_port_ id":"3cfdfeddbd89","vfinfo_list":[]}]
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth3'
DEBUG/IFCONFIG returned (out):
[{"ifindex":5,"ifname":"eth3","flags":["BROADCAST","MULTICAST","UP","LOWER_UP"],"mtu":1500,"qdisc":"mq","operstate":"UP","linkmode":"DEFAULT ","group":"default","txqlen":1000,"link_type":"ether","address":"3c:fd:fe:dd:bd:89","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min_mtu ":68,"max_mtu":9702,"inet6_addr_gen_mode":"none","num_tx_queues":16,"num_rx_queues":16,"gso_max_size":65536,"gso_max_segs":65535,"phys_port_ id":"3cfdfeddbd89","vfinfo_list":[]}]
DEBUG/IFCONFIG cmd 'ethtool --show-pause eth3'
DEBUG/IFCONFIG returned (out):
Pause parameters for eth3:
Autonegotiate: off
RX: off
TX: off
DEBUG/IFCONFIG cmd 'ethtool --pause eth3 autoneg on tx on rx on'
could not set flowcontrol for eth3
DEBUG/IFCONFIG cmd 'ethtool -K eth3 gro off'
DEBUG/IFCONFIG cmd 'ethtool -K eth3 gso off'
DEBUG/IFCONFIG write '0' > '/sys/class/net/eth3/queues/rx-0/rps_cpus'
DEBUG/IFCONFIG cmd 'ethtool -K eth3 sg off'
DEBUG/IFCONFIG cmd 'ethtool -K eth3 tso off'
DEBUG/IFCONFIG cmd 'ethtool -K eth3 ufo off'
DEBUG/IFCONFIG cmd 'ethtool eth3'
DEBUG/IFCONFIG returned (out):
Settings for eth3:

Supported ports: [ FIBRE ]
Supported link modes:   1000baseX/Full
                        10000baseSR/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes:  1000baseX/Full
                        10000baseSR/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Port: FIBRE
PHYAD: 0
Transceiver: internal
Auto-negotiation: off
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
                       drv probe link
Link detected: yes

DEBUG/IFCONFIG cmd 'ethtool -s eth3 autoneg on'
{'description': 'Test',
'duplex': 'auto',
'ifname': 'eth3',
'ip': {'arp_cache_timeout': '30'},
'mtu': '1500',
'speed': 'auto',
'vif': {'80': {'address': ['1.3.3.7/29'],

'ip': {'arp_cache_timeout': '30'},
'mtu': '1500'}}}

DEBUG/IFCONFIG cmd 'ip link set dev eth3 alias "Test"'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv4/conf/eth3/link_filter'
DEBUG/IFCONFIG cmd 'sudo systemctl stop [email protected]'
DEBUG/IFCONFIG cmd 'sudo systemctl stop [email protected]'
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth3'
DEBUG/IFCONFIG returned (out):
[{"ifindex":5,"ifname":"eth3","flags":["NO-CARRIER","BROADCAST","MULTICAST","UP"],"mtu":1500,"qdisc":"mq","operstate":"DOWN","linkmode":"DEF AULT","group":"default","txqlen":1000,"link_type":"ether","address":"3c:fd:fe:dd:bd:89","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min _mtu":68,"max_mtu":9702,"inet6_addr_gen_mode":"none","num_tx_queues":16,"num_rx_queues":16,"gso_max_size":65536,"gso_max_segs":65535,"phys_p ort_id":"3cfdfeddbd89","ifalias":"Test","vfinfo_list":[]}]
DEBUG/IFCONFIG cmd 'ip link set dev eth3 nomaster'
DEBUG/IFCONFIG write '30000' > '/proc/sys/net/ipv4/neigh/eth3/base_reachable_time_ms'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv4/conf/eth3/arp_filter'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3/arp_accept'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3/arp_announce'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3/arp_ignore'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3/proxy_arp'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3/proxy_arp_pvlan'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv4/conf/eth3/forwarding'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3/rp_filter'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv6/conf/eth3/forwarding'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv6/conf/eth3/accept_ra'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv6/conf/eth3/autoconf'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv6/conf/eth3/dad_transmits'
DEBUG/IFCONFIG cmd 'ip link set dev eth3 mtu 1500'
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth3'
DEBUG/IFCONFIG returned (out):
[{"ifindex":5,"ifname":"eth3","flags":["NO-CARRIER","BROADCAST","MULTICAST","UP"],"mtu":1500,"qdisc":"mq","operstate":"DOWN","linkmode":"DEF AULT","group":"default","txqlen":1000,"link_type":"ether","address":"3c:fd:fe:dd:bd:89","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min _mtu":68,"max_mtu":9702,"inet6_addr_gen_mode":"none","num_tx_queues":16,"num_rx_queues":16,"gso_max_size":65536,"gso_max_segs":65535,"phys_p ort_id":"3cfdfeddbd89","ifalias":"Test","vfinfo_list":[]}]
DEBUG/IFCONFIG cmd 'xdp_loader -d eth3 -U --auto-mode'
DEBUG/IFCONFIG returned (out):
INFO: xdp_link_detach() no curr XDP prog on ifindex:5
DEBUG/IFCONFIG cmd 'tc qdisc del dev eth3 handle ffff: ingress 2> /dev/null;tc qdisc del dev eth3 handle 1: root prio 2> /dev/null;set $?=0'
DEBUG/IFCONFIG cmd 'ip link set dev eth3 up'
{'address': ['1.3.3.7/29'],
'ifname': 'eth3.80',
'ip': {'arp_cache_timeout': '30'},
'mtu': '1500'}
DEBUG/IFCONFIG cmd 'ip link set dev eth3.80 alias ""'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv4/conf/eth3.80/link_filter'
DEBUG/IFCONFIG cmd 'sudo systemctl stop [email protected]'
DEBUG/IFCONFIG cmd 'sudo systemctl stop [email protected]'
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth3.80'
DEBUG/IFCONFIG returned (out):
[{"ifindex":8,"link":"eth3","ifname":"eth3.80","flags":["NO-CARRIER","BROADCAST","MULTICAST","UP"],"mtu":1500,"qdisc":"noqueue","operstate": "LOWERLAYERDOWN","linkmode":"DEFAULT","group":"default","txqlen":1000,"link_type":"ether","address":"3c:fd:fe:dd:bd:89","broadcast":"ff:ff:f f:ff:ff:ff","promiscuity":0,"min_mtu":0,"max_mtu":65535,"linkinfo":{"info_kind":"vlan","info_data":{"protocol":"802.1Q","id":80,"flags":["RE ORDER_HDR"]}},"inet6_addr_gen_mode":"none","num_tx_queues":1,"num_rx_queues":1,"gso_max_size":65536,"gso_max_segs":65535}]
DEBUG/IFCONFIG cmd 'ip link set dev eth3.80 nomaster'
DEBUG/IFCONFIG write '30000' > '/proc/sys/net/ipv4/neigh/eth3.80/base_reachable_time_ms'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv4/conf/eth3.80/arp_filter'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3.80/arp_accept'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3.80/arp_announce'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3.80/arp_ignore'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3.80/proxy_arp'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3.80/proxy_arp_pvlan'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv4/conf/eth3.80/forwarding'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv4/conf/eth3.80/rp_filter'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv6/conf/eth3.80/forwarding'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv6/conf/eth3.80/accept_ra'
DEBUG/IFCONFIG write '0' > '/proc/sys/net/ipv6/conf/eth3.80/autoconf'
DEBUG/IFCONFIG write '1' > '/proc/sys/net/ipv6/conf/eth3.80/dad_transmits'
DEBUG/IFCONFIG cmd 'ip link set dev eth3.80 mtu 1500'
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth3.80'
DEBUG/IFCONFIG returned (out):
[{"ifindex":8,"link":"eth3","ifname":"eth3.80","flags":["NO-CARRIER","BROADCAST","MULTICAST","UP"],"mtu":1500,"qdisc":"noqueue","operstate": "LOWERLAYERDOWN","linkmode":"DEFAULT","group":"default","txqlen":1000,"link_type":"ether","address":"3c:fd:fe:dd:bd:89","broadcast":"ff:ff:f f:ff:ff:ff","promiscuity":0,"min_mtu":0,"max_mtu":65535,"linkinfo":{"info_kind":"vlan","info_data":{"protocol":"802.1Q","id":80,"flags":["RE ORDER_HDR"]}},"inet6_addr_gen_mode":"none","num_tx_queues":1,"num_rx_queues":1,"gso_max_size":65536,"gso_max_segs":65535}]
DEBUG/IFCONFIG cmd 'xdp_loader -d eth3.80 -U --auto-mode'
DEBUG/IFCONFIG returned (out):
INFO: xdp_link_detach() no curr XDP prog on ifindex:8
DEBUG/IFCONFIG cmd 'ip link set dev eth3.80 up'

Don't think this is related to the i40e driver. I think it's because vyos sees

Auto-negotiation: off

when running ethtool and tries to set it to on because it's not off in the config.

DEBUG/IFCONFIG cmd 'ethtool -s eth3 autoneg on' <--- nic flap occurs here.

If I set speed 10000 and duplex full the error does not occur anymore.

The nic is using DACs so autoneg is not supported so the driver is correct.

ethtool however does siganl "Supports auto-negotiation: Yes" which is wrong. I've seen this with other nics also not using i40e so not sure how to work around it.

pasik added a subscriber: pasik.May 5 2021, 11:12 AM

@shyrwall

I tried reproducing this issue on VyOS 1.3.0-rc4 but I see no link flaps on my IGB card connected to a Cisco Catalyst 1000.

Can you please try on your Intel i40e installation if the nic flap really occurs when you enter:

sudo ethtool -s eth2 autoneg on

Yes. Tried before posting the above comment.

Well then somehow this feels to me NIC related as I can not see it here.

Well yeah kinda :) The driver VYOS is using is reporting wrong information to ethtool about autoneg support. No idea if it's fixed in a newer driver. Can't really play around much and I only have this type of nic in a single server. Let's hope someone comes along later with a X710 and can try more.

In the meanwhile i'll just comment out the autoneg stuff from the vyos 1.3/1.4 scripts.

c-po moved this task from Need Triage to Backlog on the VyOS 1.4 Sagitta board.May 24 2021, 7:41 PM

We are seeing this on an X722 NIC that's still the i40e driver (the server in question also has an X710 NIC, but there's no cables connected to the X710 but I should be able to get cables connected to the X710 next week). We are using 1.4 on this machine.

We tried disabling flow control on our NICs but we still get this interface flap whenever we make a change. I've not tried with 1.2 or 1.3.

If there's any commands / builds / etc I can try on this machine then I am happy to (but as this machine is remote it may take me a few days to try anything as I have to schedule a maintenance period)

c-po changed the task status from On hold to In progress.Aug 29 2021, 7:05 PM

c-po set Issue type to Unspecified (please specify).

Can you please upgrade to the latest 1.3 beta version from https://vyos.net/get/nightly-builds/ a lot has been changed since then.

If the issue re-occurs please enable debugging by: touch /tmp/vyos.ifconfig.debug and update this task.

c-po changed the task status from In progress to Needs testing.Sep 1 2021, 7:48 AM

We tested this on 1.4-rolling-202103040218 on our router, unfortunately as this is a production device I couldn't spend long diagnosing this. After upgrading the system image the device didn't come back up (I have layer 2 connectivity to the device via a VLAN on eth1 normally). So I logged in via our serial console and did a show interfaces

Traceback (most recent call last):
  File "/usr/libexec/vyos/op_mode/show_interfaces.py", line 309, in <module>
    actions.get(args.action, missing)(
  File "/usr/libexec/vyos/op_mode/show_interfaces.py", line 48, in handled_function
    function(*args, **kwargs)
  File "/usr/libexec/vyos/op_mode/show_interfaces.py", line 222, in run_show_intf_brief
    for interface in filtered_interfaces(ifnames, iftypes, vif, vrrp):
  File "/usr/libexec/vyos/op_mode/show_interfaces.py", line 77, in filtered_interfaces
    interface = klass(ifname, create=False, debug=False)
  File "/usr/lib/python3/dist-packages/vyos/ifconfig/ethernet.py", line 81, in __init__
    self.ethtool = Ethtool(ifname)
  File "/usr/lib/python3/dist-packages/vyos/ethtool.py", line 60, in __init__
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface        IP Address                        S/L  Description
---------        ----------                        ---  -----------
eth0             -                                 A/D
eth1             -                                 A/D
eth2             -                                 u/D
eth3             -                                 A/D
eth4             -                                 A/D
eth5             -                                 u/u
eth6             -                                 A/D
eth7             -                                 A/D
eth8             -                                 u/D
eth9             -                                 A/D
eth10            -                                 A/D
eth11            -                                 u/D
eth12            -                                 u/D  Core: Interface Description
    raise ValueError(f'Could not determine driver for interface {ifname}!')
ValueError: Could not determine driver for interface eth12.3099!

I don't know if this was caused specifically by this bug, and I needed this router back online so I could not enable debugging. All interfaces that show a state of u on that output are active and have active descriptions except they've clearly not applied here. eth12 is an X710 NIC. eth1 is an X722 NIC. eth6 is an i210 using the igb driver.

If you need further information and no one is forth coming with that information I can schedule a longer maintenance period on this device but it is a critical device for us so hopefully someone will come along who can test this more easily.

Okay, a lot has changed in the recent 1.4 version in terms of changing interface parameters due to T2738 - can you please retry one of the latest 1.4 images starting from 2021-09-21?

Thanks for testing!

Sorry for the late reply, I've been waiting for a maintenance window to test this in. Again as this is a production device I only have limited debug info before having to roll back to the working version. I installed 1.4-rolling-202110150613 and on booting show interfaces does indeed show all the interfaces. The igb driver interfaces were showing as up (u/u). The i40e interfaces were showing as down (u/D).

Even running ip link set dev eth0 up did not bring the interface up. I did get some output from dmesg before rolling back:

[   41.898492] i40e 0000:17:00.3 eth4: set new mac address 8c:04:ba:da:92:48
[   41.989185] 8021q: adding VLAN 0 to HW filter on device eth4
[   42.098012] i40e 0000:b8:00.3 eth10: To change autoneg please use: ethtool -s <dev> autoneg <on|off>
[   42.110801] i40e 0000:b8:00.3 eth10: set new mac address e4:43:4b:9f:02:91
[   42.171970] IPv6: ADDRCONF(NETDEV_CHANGE): eth4: link becomes ready
[   42.195306] 8021q: adding VLAN 0 to HW filter on device eth10
[   42.327549] i40e 0000:17:00.2 eth3: set new mac address 8c:04:ba:da:92:47
[   42.438529] 8021q: adding VLAN 0 to HW filter on device eth3
[   42.549518] i40e 0000:b8:00.2 eth9: To change autoneg please use: ethtool -s <dev> autoneg <on|off>
[   42.562208] i40e 0000:b8:00.2 eth9: set new mac address e4:43:4b:9f:02:90
[   42.645239] 8021q: adding VLAN 0 to HW filter on device eth9
[   42.891592] 8021q: adding VLAN 0 to HW filter on device eth8
[   42.999669] i40e 0000:b8:00.1 eth7: To change autoneg please use: ethtool -s <dev> autoneg <on|off>
[   43.012543] i40e 0000:b8:00.1 eth7: set new mac address 8c:04:ba:da:92:42
[   43.124534] 8021q: adding VLAN 0 to HW filter on device eth7
[   43.269372] i40e 0000:b8:00.0 eth6: To change autoneg please use: ethtool -s <dev> autoneg <on|off>
[   43.282176] i40e 0000:b8:00.0 eth6: set new mac address 8c:04:ba:da:92:41
[   43.402761] 8021q: adding VLAN 0 to HW filter on device eth6
[   43.402785] IPv6: ADDRCONF(NETDEV_CHANGE): eth6: link becomes ready
[   43.610090] i40e 0000:17:00.0 eth0: To change autoneg please use: ethtool -s <dev> autoneg <on|off>
[   43.622869] i40e 0000:17:00.0 eth0: set new mac address 8c:04:ba:da:92:44
[   43.714226] 8021q: adding VLAN 0 to HW filter on device eth0
[   43.964666] 8021q: adding VLAN 0 to HW filter on device eth2
[   44.334260] igb 0000:02:00.1 eth5: igb: eth5 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX

We've now built a test x710 box for this, so we can test this more easily.

We are running

Version:          VyOS 1.4-rolling-202110240217
Release train:    sagitta

Built by:         [email protected]
Built on:         Sun 24 Oct 2021 02:17 UTC

And a basic test on this x710 NIC (linked at 10gig) as follows:

[email protected]:~$ conf
[edit]
[email protected]# set interface eth eth3 description "Something"
[edit]
[email protected]# commit
Could not set flowcontrol for eth3
[edit]
[email protected]#

In dmesg we get:

[ 1587.984767] i40e 0000:04:00.1 eth3: To change autoneg please use: ethtool -s <dev> autoneg <on|off>

As for the debug info, we can now finally get this for you:

[email protected]# commit
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth3'
DEBUG/IFCONFIG returned (out):
[{"ifindex":5,"ifname":"eth3","flags":["BROADCAST","MULTICAST","UP","LOWER_UP"],"mtu":1500,"qdisc":"mq","operstate":"UP","linkmode":"DEFAULT","group":"default","txqlen":1000,"link_type":"ether","address":"3c:fd:fe:a4:55:15","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min_mtu":68,"max_mtu":9702,"inet6_addr_gen_mode":"none","num_tx_queues":4,"num_rx_queues":4,"gso_max_size":65536,"gso_max_segs":65535,"phys_port_id":"3cfdfea45515","ifalias":"Something","vfinfo_list":[]}]
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth3'
DEBUG/IFCONFIG returned (out):
[{"ifindex":5,"ifname":"eth3","flags":["BROADCAST","MULTICAST","UP","LOWER_UP"],"mtu":1500,"qdisc":"mq","operstate":"UP","linkmode":"DEFAULT","group":"default","txqlen":1000,"link_type":"ether","address":"3c:fd:fe:a4:55:15","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min_mtu":68,"max_mtu":9702,"inet6_addr_gen_mode":"none","num_tx_queues":4,"num_rx_queues":4,"gso_max_size":65536,"gso_max_segs":65535,"phys_port_id":"3cfdfea45515","ifalias":"Something","vfinfo_list":[]}]
DEBUG/IFCONFIG cmd 'ethtool --pause eth3 autoneg on tx on rx on'
Could not set flowcontrol for eth3
DEBUG/IFCONFIG write '0' > '/sys/class/net/eth3/queues/rx-0/rps_cpus'
DEBUG/IFCONFIG NIC driver does not support changing speed/duplex settings!
{'address': ['dhcp'],
 'description': 'something else',
 'duplex': 'auto',
 'hw_id': '3c:fd:fe:a4:55:15',
 'ifname': 'eth3',
 'ip': {'arp_cache_timeout': '30'},
 'mtu': '1500',
 'offload': {'gro': {}, 'gso': {}, 'sg': {}, 'tso': {}},
 'speed': 'auto'}
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth3'
DEBUG/IFCONFIG returned (out):
[{"ifindex":5,"ifname":"eth3","flags":["BROADCAST","MULTICAST","UP","LOWER_UP"],"mtu":1500,"qdisc":"mq","operstate":"UP","linkmode":"DEFAULT","group":"default","txqlen":1000,"link_type":"ether","address":"3c:fd:fe:a4:55:15","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min_mtu":68,"max_mtu":9702,"inet6_addr_gen_mode":"none","num_tx_queues":4,"num_rx_queues":4,"gso_max_size":65536,"gso_max_segs":65535,"phys_port_id":"3cfdfea45515","ifalias":"Something","vfinfo_list":[]}]
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth3'
DEBUG/IFCONFIG returned (out):
[{"ifindex":5,"ifname":"eth3","flags":["BROADCAST","MULTICAST","UP","LOWER_UP"],"mtu":1500,"qdisc":"mq","operstate":"UP","linkmode":"DEFAULT","group":"default","txqlen":1000,"link_type":"ether","address":"3c:fd:fe:a4:55:15","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min_mtu":68,"max_mtu":9702,"inet6_addr_gen_mode":"none","num_tx_queues":4,"num_rx_queues":4,"gso_max_size":65536,"gso_max_segs":65535,"phys_port_id":"3cfdfea45515","ifalias":"Something","vfinfo_list":[]}]
DEBUG/IFCONFIG cmd 'ip link set dev eth3 alias "something else"'
DEBUG/IFCONFIG read '1' < '/proc/sys/net/ipv4/conf/eth3/link_filter'
DEBUG/IFCONFIG cmd 'systemctl stop [email protected]'
DEBUG/IFCONFIG cmd 'systemctl restart [email protected]'
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth3'
DEBUG/IFCONFIG returned (out):
[{"ifindex":5,"ifname":"eth3","flags":["BROADCAST","MULTICAST","UP","LOWER_UP"],"mtu":1500,"qdisc":"mq","operstate":"UP","linkmode":"DEFAULT","group":"default","txqlen":1000,"link_type":"ether","address":"3c:fd:fe:a4:55:15","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min_mtu":68,"max_mtu":9702,"inet6_addr_gen_mode":"none","num_tx_queues":4,"num_rx_queues":4,"gso_max_size":65536,"gso_max_segs":65535,"phys_port_id":"3cfdfea45515","ifalias":"something else","vfinfo_list":[]}]
DEBUG/IFCONFIG cmd 'ip link set dev eth3 nomaster'
DEBUG/IFCONFIG cmd 'nft -c delete element inet vrf_zones ct_iface_map { "eth3" }'
DEBUG/IFCONFIG cmd 'iptables-save -t mangle'
DEBUG/IFCONFIG returned (out):
# Generated by iptables-save v1.8.7 on Wed Oct 27 14:25:13 2021
*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
COMMIT
# Completed on Wed Oct 27 14:25:13 2021
DEBUG/IFCONFIG cmd 'ip6tables-save -t mangle'
DEBUG/IFCONFIG returned (out):
# Generated by ip6tables-save v1.8.7 on Wed Oct 27 14:25:13 2021
*mangle
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:VYOS_DNPT_HOOK - [0:0]
:VYOS_SNPT_HOOK - [0:0]
-A PREROUTING -j VYOS_DNPT_HOOK
-A POSTROUTING -j VYOS_SNPT_HOOK
-A VYOS_DNPT_HOOK -j RETURN
-A VYOS_SNPT_HOOK -j RETURN
COMMIT
# Completed on Wed Oct 27 14:25:13 2021
DEBUG/IFCONFIG read '30000' < '/proc/sys/net/ipv4/neigh/eth3/base_reachable_time_ms'
DEBUG/IFCONFIG read '1' < '/proc/sys/net/ipv4/conf/eth3/arp_filter'
DEBUG/IFCONFIG read '0' < '/proc/sys/net/ipv4/conf/eth3/arp_accept'
DEBUG/IFCONFIG read '0' < '/proc/sys/net/ipv4/conf/eth3/arp_announce'
DEBUG/IFCONFIG read '0' < '/proc/sys/net/ipv4/conf/eth3/arp_ignore'
DEBUG/IFCONFIG read '0' < '/proc/sys/net/ipv4/conf/eth3/proxy_arp'
DEBUG/IFCONFIG read '0' < '/proc/sys/net/ipv4/conf/eth3/proxy_arp_pvlan'
DEBUG/IFCONFIG read '1' < '/proc/sys/net/ipv4/conf/eth3/forwarding'
DEBUG/IFCONFIG read '0' < '/proc/sys/net/ipv4/conf/eth3/rp_filter'
DEBUG/IFCONFIG read '1' < '/proc/sys/net/ipv6/conf/eth3/forwarding'
DEBUG/IFCONFIG read '1' < '/proc/sys/net/ipv6/conf/eth3/accept_ra'
DEBUG/IFCONFIG read '0' < '/proc/sys/net/ipv6/conf/eth3/autoconf'
DEBUG/IFCONFIG read '1' < '/proc/sys/net/ipv6/conf/eth3/dad_transmits'
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth3'
DEBUG/IFCONFIG returned (out):
[{"ifindex":5,"ifname":"eth3","flags":["BROADCAST","MULTICAST","UP","LOWER_UP"],"mtu":1500,"qdisc":"mq","operstate":"UP","linkmode":"DEFAULT","group":"default","txqlen":1000,"link_type":"ether","address":"3c:fd:fe:a4:55:15","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min_mtu":68,"max_mtu":9702,"inet6_addr_gen_mode":"none","num_tx_queues":4,"num_rx_queues":4,"gso_max_size":65536,"gso_max_segs":65535,"phys_port_id":"3cfdfea45515","ifalias":"something else","vfinfo_list":[]}]
DEBUG/IFCONFIG cmd 'ip -json -detail link list dev eth3'
DEBUG/IFCONFIG returned (out):
[{"ifindex":5,"ifname":"eth3","flags":["BROADCAST","MULTICAST","UP","LOWER_UP"],"mtu":1500,"qdisc":"mq","operstate":"UP","linkmode":"DEFAULT","group":"default","txqlen":1000,"link_type":"ether","address":"3c:fd:fe:a4:55:15","broadcast":"ff:ff:ff:ff:ff:ff","promiscuity":0,"min_mtu":68,"max_mtu":9702,"inet6_addr_gen_mode":"none","num_tx_queues":4,"num_rx_queues":4,"gso_max_size":65536,"gso_max_segs":65535,"phys_port_id":"3cfdfea45515","ifalias":"something else","vfinfo_list":[]}]
DEBUG/IFCONFIG cmd 'xdp_loader -d eth3 -U --auto-mode'
DEBUG/IFCONFIG returned (out):
INFO: xdp_link_detach() no curr XDP prog on ifindex:5
DEBUG/IFCONFIG cmd 'tc qdisc del dev eth3 handle ffff: ingress 2> /dev/null;tc qdisc del dev eth3 handle 1: root prio 2> /dev/null;set $?=0'
DEBUG/IFCONFIG cmd 'ip link set dev eth3 up'

Hopefully this helps! We've got this x710 box for as long as we need it to assist in getting this bug resolved.

That is great news, any chance I can get SSH access to this machine for testing myself?

Yep, that's fine - what's the best way the details over to you?

Send it by email - [email protected]

This is now resolved for us.

Thanks for getting back to us!

c-po closed this task as Resolved.Nov 6 2021, 10:42 AM

syncer moved this task from Need Triage to Finished on the VyOS 1.3 Equuleus board.Nov 6 2021, 11:43 AM

syncer edited projects, added VyOS 1.3 Equuleus (1.3.0); removed VyOS 1.3 Equuleus.Aug 29 2022, 12:16 PM

syncer moved this task from Need Triage to Finished on the VyOS 1.3 Equuleus (1.3.0) board.

NIC flap at any interface changeClosed, ResolvedPublicBUGActions

Description

Details

Related Objects

Event Timeline

NIC flap at any interface change
Closed, ResolvedPublicBUG
Actions