Page MenuHomeVyOS Platform

vyos-configd throws an exception during an image upgrade
Open, NormalPublicBUG

Description

Dear Vyos experts,

The image version I am currently running (1.5-rolling-202407280023)has always worked perfectly.
However, when I tried to upgrade to the new images( 1.5-rolling-202410180006 ), I encountered an exception that caused the upgrade to fail. Here is the error I encountered:

Oct 23 17:09:52 vyos vyos-configd[692]: Sending reply: 1
Oct 23 17:09:52 vyos vyos-configd[692]: Received request: msg_size
Oct 23 17:09:52 vyos vyos-configd[692]: Sending reply: 0
Oct 23 17:09:52 vyos sudo[2331]: pam_unix(sudo:session): session closed for user root
Oct 23 17:09:52 vyos sudo[2358]:     root : PWD=/ ; USER=root ; COMMAND=/usr/bin/sh -c '/usr/sbin/vyshim VYOS_TAGNODE_VALUE=\'lo\' /usr/libexec/vyos/conf_mode/interfaces_loopback.py'
Oct 23 17:09:52 vyos sudo[2358]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Oct 23 17:09:52 vyos vyos-configd[692]: Received message: {"type": "node", "last": false, "data": "VYOS_TAGNODE_VALUE=lo/usr/libexec/vyos/conf_mode/interfaces_loopback.py"}
Oct 23 17:09:53 vyos vyos-configd[692]: Sending reply: 1
Oct 23 17:09:53 vyos vyos-configd[692]: Received request: msg_size
Oct 23 17:09:53 vyos vyos-configd[692]: Sending reply: 0
Oct 23 17:09:53 vyos sudo[2358]: pam_unix(sudo:session): session closed for user root
Oct 23 17:09:53 vyos sudo[2397]:     root : PWD=/ ; USER=root ; COMMAND=/usr/bin/sh -c '/usr/sbin/vyshim VYOS_TAGNODE_VALUE=\'br0\' /usr/libexec/vyos/conf_mode/interfaces_bridge.py'
Oct 23 17:09:53 vyos sudo[2397]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=0)
Oct 23 17:09:53 vyos vyos-configd[692]: Received message: {"type": "node", "last": false, "data": "VYOS_TAGNODE_VALUE=br0/usr/libexec/vyos/conf_mode/interfaces_bridge.py"}
Oct 23 17:09:53 vyos zebra[1646]: [H5BRT-63K8T] Cannot find bridge-vlan IF (7) for vlan update
Oct 23 17:09:53 vyos (udev-worker)[2241]: Network interface NamePolicy= disabled on kernel command line.
Oct 23 17:09:53 vyos kernel: br0: port 1(eth0) entered blocking state
Oct 23 17:09:53 vyos kernel: br0: port 1(eth0) entered disabled state
Oct 23 17:09:53 vyos kernel: igc 0000:01:00.0 eth0: entered allmulticast mode
Oct 23 17:09:53 vyos kernel: igc 0000:01:00.0 eth0: entered promiscuous mode
Oct 23 17:09:53 vyos kernel: br0: port 2(eth1) entered blocking state
Oct 23 17:09:53 vyos kernel: br0: port 2(eth1) entered disabled state
Oct 23 17:09:53 vyos kernel: igc 0000:02:00.0 eth1: entered allmulticast mode
Oct 23 17:09:53 vyos kernel: igc 0000:02:00.0 eth1: entered promiscuous mode
Oct 23 17:09:53 vyos netplugd[1246]: br0: state DOWN flags 0x00001002 BROADCAST,MULTICAST -> 0x00001003 UP,BROADCAST,MULTICAST
Oct 23 17:09:53 vyos vyos-configd[692]: Sending reply: 1
Oct 23 17:09:53 vyos vyos-configd[692]: Received request: msg_size
Oct 23 17:09:53 vyos vyos-configd[692]: Sending reply: 0
Oct 23 17:09:53 vyos vyos-configd[692]: Received message: send
Oct 23 17:09:53 vyos vyos-configd[692]: VyOS had an issue completing a command.
Oct 23 17:09:53 vyos vyos-configd[692]: We are sorry that you encountered a problem while using VyOS.
Oct 23 17:09:53 vyos vyos-configd[692]: There are a few things you can do to help us (and yourself):
Oct 23 17:09:53 vyos vyos-configd[692]: - Contact us using the online help desk if you have a subscription:
Oct 23 17:09:53 vyos vyos-configd[692]:   https://support.vyos.io/
Oct 23 17:09:53 vyos vyos-configd[692]: - Make sure you are running the latest version of VyOS available at:
Oct 23 17:09:53 vyos vyos-configd[692]:   https://vyos.net/get/
Oct 23 17:09:53 vyos vyos-configd[692]: - Consult the community forum to see how to handle this issue:
Oct 23 17:09:53 vyos vyos-configd[692]:   https://forum.vyos.io
Oct 23 17:09:53 vyos vyos-configd[692]: - Join us on Slack where our users exchange help and advice:
Oct 23 17:09:53 vyos vyos-configd[692]:   https://vyos.slack.com
Oct 23 17:09:53 vyos vyos-configd[692]: When reporting problems, please include as much information as possible:
Oct 23 17:09:53 vyos vyos-configd[692]: - do not obfuscate any data (feel free to contact us privately if your
Oct 23 17:09:53 vyos vyos-configd[692]:   business policy requires it)
Oct 23 17:09:53 vyos vyos-configd[692]: - and include all the information presented below
Oct 23 17:09:53 vyos vyos-configd[692]: Report time:      2024-10-23 09:09:53
Oct 23 17:09:53 vyos vyos-configd[692]: Image version:    VyOS 1.5-rolling-202410180006
Oct 23 17:09:53 vyos vyos-configd[692]: Release train:    current
Oct 23 17:09:53 vyos vyos-configd[692]: Built by:         [email protected]
Oct 23 17:09:53 vyos vyos-configd[692]: Built on:         Fri 18 Oct 2024 00:07 UTC
Oct 23 17:09:53 vyos vyos-configd[692]: Build UUID:       a6dc3e7c-619f-4051-937a-93b4adac485f
Oct 23 17:09:53 vyos vyos-configd[692]: Build commit ID:  2359180068a653
Oct 23 17:09:53 vyos vyos-configd[692]: Architecture:     x86_64
Oct 23 17:09:53 vyos vyos-configd[692]: Boot via:         installed image
Oct 23 17:09:53 vyos vyos-configd[692]: System type:      bare metal
Oct 23 17:09:53 vyos vyos-configd[692]: Hardware vendor:  CncTion
Oct 23 17:09:53 vyos vyos-configd[692]: Hardware model:   N4100-4L
Oct 23 17:09:53 vyos vyos-configd[692]: Hardware S/N:     Default string
Oct 23 17:09:53 vyos vyos-configd[692]: Hardware UUID:    03000200-0400-0500-0006-000700080009
Oct 23 17:09:53 vyos vyos-configd[692]: Traceback (most recent call last):
Oct 23 17:09:53 vyos vyos-configd[692]:   File "/usr/libexec/vyos/services/vyos-configd", line 326, in <module>
Oct 23 17:09:53 vyos vyos-configd[692]:     message = json.loads(msg)
Oct 23 17:09:53 vyos vyos-configd[692]:               ^^^^^^^^^^^^^^^
Oct 23 17:09:53 vyos vyos-configd[692]:   File "/usr/lib/python3.11/json/__init__.py", line 346, in loads
Oct 23 17:09:53 vyos vyos-configd[692]:     return _default_decoder.decode(s)
Oct 23 17:09:53 vyos vyos-configd[692]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 23 17:09:53 vyos vyos-configd[692]:   File "/usr/lib/python3.11/json/decoder.py", line 337, in decode
Oct 23 17:09:53 vyos vyos-configd[692]:     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
Oct 23 17:09:53 vyos vyos-configd[692]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 23 17:09:53 vyos vyos-configd[692]:   File "/usr/lib/python3.11/json/decoder.py", line 355, in raw_decode
Oct 23 17:09:53 vyos vyos-configd[692]:     raise JSONDecodeError("Expecting value", s, err.value) from None
Oct 23 17:09:53 vyos vyos-configd[692]: json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Oct 23 17:09:53 vyos vyos-configd[692]: noteworthy:
Oct 23 17:09:53 vyos vyos-configd[692]: cmd 'nft --check delete element inet vrf_zones ct_iface_map { "lo" }'
Oct 23 17:09:53 vyos vyos-configd[692]: returned (out):
Oct 23 17:09:53 vyos vyos-configd[692]: returned (err):
Oct 23 17:09:53 vyos vyos-configd[692]: Error: Could not process rule: No such file or directory
Oct 23 17:09:53 vyos vyos-configd[692]: delete element inet vrf_zones ct_iface_map { lo }
Oct 23 17:09:53 vyos vyos-configd[692]:                                              ^^
Oct 23 17:09:53 vyos vyos-configd[692]: cmd 'ip link show dev br0'
Oct 23 17:09:53 vyos vyos-configd[692]: returned (out):
Oct 23 17:09:53 vyos vyos-configd[692]: returned (err):
Oct 23 17:09:53 vyos vyos-configd[692]: Device "br0" does not exist.
Oct 23 17:09:53 vyos vyos-configd[692]: cmd 'ip link show dev br0'
Oct 23 17:09:53 vyos vyos-configd[692]: returned (out):
Oct 23 17:09:53 vyos vyos-configd[692]: returned (err):
Oct 23 17:09:53 vyos vyos-configd[692]: Device "br0" does not exist.
Oct 23 17:09:53 vyos vyos-configd[692]: cmd 'nft --check delete element inet vrf_zones ct_iface_map { "br0" }'
Oct 23 17:09:53 vyos vyos-configd[692]: returned (out):
Oct 23 17:09:53 vyos vyos-configd[692]: returned (err):
Oct 23 17:09:53 vyos vyos-configd[692]: Error: Could not process rule: No such file or directory
Oct 23 17:09:53 vyos vyos-configd[692]: delete element inet vrf_zones ct_iface_map { br0 }
Oct 23 17:09:53 vyos vyos-configd[692]:                                              ^^^
Oct 23 17:09:53 vyos python3[692]: Report time:      2024-10-23 09:09:53
Oct 23 17:09:53 vyos python3[692]: Image version:    VyOS 1.5-rolling-202410180006
Oct 23 17:09:53 vyos python3[692]: Release train:    current
Oct 23 17:09:53 vyos python3[692]: Built by:         [email protected]
Oct 23 17:09:53 vyos python3[692]: Built on:         Fri 18 Oct 2024 00:07 UTC
Oct 23 17:09:53 vyos python3[692]: Build UUID:       a6dc3e7c-619f-4051-937a-93b4adac485f
Oct 23 17:09:53 vyos python3[692]: Build commit ID:  2359180068a653
Oct 23 17:09:53 vyos Architecture[692]:     x86_64
Oct 23 17:09:53 vyos python3[692]: Boot via:         installed image
Oct 23 17:09:53 vyos python3[692]: System type:      bare metal
Oct 23 17:09:53 vyos python3[692]: Hardware vendor:  CncTion
Oct 23 17:09:53 vyos python3[692]: Hardware model:   N4100-4L
Oct 23 17:09:53 vyos python3[692]: Hardware S/N:     Default string
Oct 23 17:09:53 vyos python3[692]: Hardware UUID:    03000200-0400-0500-0006-000700080009
Oct 23 17:09:53 vyos python3[692]: Traceback (most recent call last):
Oct 23 17:09:53 vyos python3[692]:   File "/usr/libexec/vyos/services/vyos-configd", line 326, in <module>
Oct 23 17:09:53 vyos python3[692]:     message = json.loads(msg)
Oct 23 17:09:53 vyos python3[692]:               ^^^^^^^^^^^^^^^
Oct 23 17:09:53 vyos python3[692]:   File "/usr/lib/python3.11/json/__init__.py", line 346, in loads
Oct 23 17:09:53 vyos python3[692]:     return _default_decoder.decode(s)
Oct 23 17:09:53 vyos python3[692]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 23 17:09:53 vyos python3[692]:   File "/usr/lib/python3.11/json/decoder.py", line 337, in decode
Oct 23 17:09:53 vyos python3[692]:     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
Oct 23 17:09:53 vyos python3[692]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Oct 23 17:09:53 vyos python3[692]:   File "/usr/lib/python3.11/json/decoder.py", line 355, in raw_decode
Oct 23 17:09:53 vyos python3[692]:     raise JSONDecodeError("Expecting value", s, err.value) from None
Oct 23 17:09:53 vyos json.decoder.JSONDecodeError[692]: Expecting value: line 1 column 1 (char 0)
Oct 23 17:09:53 vyos noteworthy[692]: 
Oct 23 17:09:53 vyos python3[692]: cmd 'nft --check delete element inet vrf_zones ct_iface_map { "lo" }'
Oct 23 17:09:53 vyos python3[692]: returned (out):
Oct 23 17:09:53 vyos python3[692]: returned (err):
Oct 23 17:09:53 vyos Error[692]: Could not process rule: No such file or directory
Oct 23 17:09:53 vyos python3[692]: delete element inet vrf_zones ct_iface_map { lo }
Oct 23 17:09:53 vyos python3[692]:                                              ^^
Oct 23 17:09:53 vyos python3[692]: cmd 'ip link show dev br0'
Oct 23 17:09:53 vyos python3[692]: returned (out):
Oct 23 17:09:53 vyos python3[692]: returned (err):
Oct 23 17:09:53 vyos python3[692]: Device "br0" does not exist.
Oct 23 17:09:53 vyos python3[692]: cmd 'ip link show dev br0'
Oct 23 17:09:53 vyos python3[692]: returned (out):
Oct 23 17:09:53 vyos python3[692]: returned (err):
Oct 23 17:09:53 vyos python3[692]: Device "br0" does not exist.
Oct 23 17:09:53 vyos python3[692]: cmd 'nft --check delete element inet vrf_zones ct_iface_map { "br0" }'
Oct 23 17:09:53 vyos python3[692]: returned (out):
Oct 23 17:09:53 vyos python3[692]: returned (err):
Oct 23 17:09:53 vyos Error[692]: Could not process rule: No such file or directory
Oct 23 17:09:53 vyos python3[692]: delete element inet vrf_zones ct_iface_map { br0 }
Oct 23 17:09:53 vyos python3[692]:                                              ^^^
Oct 23 17:09:53 vyos systemd[1]: vyos-configd.service: Main process exited, code=exited, status=1/FAILURE
Oct 23 17:09:53 vyos systemd[1]: vyos-configd.service: Failed with result 'exit-code'.
Oct 23 17:09:53 vyos systemd[1]: vyos-configd.service: Consumed 8.208s CPU time.
Oct 23 17:09:54 vyos systemd[1]: vyos-configd.service: Scheduled restart job, restart counter is at 1.
Oct 23 17:09:54 vyos systemd[1]: Stopped vyos-configd.service - VyOS configuration daemon.
Oct 23 17:09:54 vyos systemd[1]: vyos-configd.service: Consumed 8.208s CPU time.
Oct 23 17:09:54 vyos systemd[1]: Started vyos-configd.service - VyOS configuration daemon.

and my interface configurations :

set interfaces bridge br0 address 'xxx.xxx.1.254/32'
set interfaces bridge br0 description 'WAN bridge'
set interfaces bridge br0 member interface eth0
set interfaces bridge br0 member interface eth1
set interfaces bridge br0 stp
set interfaces bridge br1 address 'xxx.xxx.0.1/24'
set interfaces bridge br1 address 'xxx.xxx.0.254/32'
set interfaces bridge br1 description 'LAN bridge'
set interfaces bridge br1 member interface eth2
set interfaces bridge br1 member interface eth3
set interfaces bridge br1 stp
set interfaces ethernet eth0 hw-id 'xx:xx:xx:xx:xx:6a'
set interfaces ethernet eth0 offload gro
set interfaces ethernet eth0 offload gso
set interfaces ethernet eth0 offload hw-tc-offload
set interfaces ethernet eth0 offload rfs
set interfaces ethernet eth0 offload rps
set interfaces ethernet eth0 offload sg
set interfaces ethernet eth0 offload tso
set interfaces ethernet eth0 ring-buffer rx '4096'
set interfaces ethernet eth0 ring-buffer tx '4096'
set interfaces ethernet eth1 hw-id 'xx:xx:xx:xx:xx:6b'
set interfaces ethernet eth1 offload gro
set interfaces ethernet eth1 offload gso
set interfaces ethernet eth1 offload hw-tc-offload
set interfaces ethernet eth1 offload rfs
set interfaces ethernet eth1 offload rps
set interfaces ethernet eth1 offload sg
set interfaces ethernet eth1 offload tso
set interfaces ethernet eth1 ring-buffer rx '4096'
set interfaces ethernet eth1 ring-buffer tx '4096'
set interfaces ethernet eth2 hw-id 'xx:xx:xx:xx:xx:6c'
set interfaces ethernet eth2 offload gro
set interfaces ethernet eth2 offload gso
set interfaces ethernet eth2 offload hw-tc-offload
set interfaces ethernet eth2 offload rfs
set interfaces ethernet eth2 offload rps
set interfaces ethernet eth2 offload sg
set interfaces ethernet eth2 offload tso
set interfaces ethernet eth2 ring-buffer rx '4096'
set interfaces ethernet eth2 ring-buffer tx '4096'
set interfaces ethernet eth3 hw-id 'xx:xx:xx:xx:xx:6d'
set interfaces ethernet eth3 offload gro
set interfaces ethernet eth3 offload gso
set interfaces ethernet eth3 offload hw-tc-offload
set interfaces ethernet eth3 offload rfs
set interfaces ethernet eth3 offload rps
set interfaces ethernet eth3 offload sg
set interfaces ethernet eth3 offload tso
set interfaces ethernet eth3 ring-buffer rx '4096'
set interfaces ethernet eth3 ring-buffer tx '4096'
set interfaces loopback lo
set interfaces pppoe pppoe0 authentication password xxxxxx
set interfaces pppoe pppoe0 authentication username xxxxxx
set interfaces pppoe pppoe0 dhcpv6-options pd 0 interface br1 address '1'
set interfaces pppoe pppoe0 dhcpv6-options pd 0 interface br1 sla-id '0'
set interfaces pppoe pppoe0 dhcpv6-options pd 0 length '60'
set interfaces pppoe pppoe0 ip adjust-mss 'clamp-mss-to-pmtu'
set interfaces pppoe pppoe0 ipv6 address autoconf
set interfaces pppoe pppoe0 ipv6 adjust-mss 'clamp-mss-to-pmtu'
set interfaces pppoe pppoe0 mtu '1480'
set interfaces pppoe pppoe0 source-interface 'br0'
set interfaces wireguard wg0 address 'xxx.xxx.0.1/24'
set interfaces wireguard wg0 description 'OSPF over WG HK Peer'
set interfaces wireguard wg0 ip adjust-mss 'clamp-mss-to-pmtu'
set interfaces wireguard wg0 peer to_HK address 'xxx.xxx.0.254'
set interfaces wireguard wg0 peer to_HK allowed-ips 'xxx.xxx.0.0/0'
set interfaces wireguard wg0 peer to_HK allowed-ips 'xxx.xxx.0.0/24'
set interfaces wireguard wg0 peer to_HK persistent-keepalive '5'
set interfaces wireguard wg0 peer to_HK port '5000'
set interfaces wireguard wg0 peer to_HK public-key 'xxxxx'
set interfaces wireguard wg0 private-key xxxxx

The above error caused both my WAN and LAN interfaces to not be up and running. I lost access to the router. I have successfully upgraded my configuration many times before. Please help me identify the reason. Thank you.

Details

Difficulty level
Unknown (require assessment)
Version
1.5-rolling-202410180006
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)