Page MenuHomeVyOS Platform

smoketest: wait for commit lock before cli_set/cli_delete too
Closed, ResolvedPublicBUG

Description

cli_commit() already busy-waits for the CStore commit_lock to clear before committing, since a concurrent process can still hold it open for a moment. cli_set()/cli_delete() had no such guard, so a set/delete issued right after a commit could race against that lingering lock and fail with a generic "Set failed"/"Delete failed" ConfigSessionError.

This is what caused the intermittent test_bgp_27_route_reflector_client failures, whose set/commit/discard loop hammers the session tightly enough to hit the race. Share the wait via a single helper used by all three.

DEBUG - Running Testcase (23/93): /usr/libexec/vyos/tests/smoke/cli/test_protocols_bgp.py
DEBUG - test_bgp_01_simple (__main__.TestProtocolsBGP.test_bgp_01_simple) ... ok
DEBUG - test_bgp_02_neighbors (__main__.TestProtocolsBGP.test_bgp_02_neighbors) ... ok
DEBUG - test_bgp_03_peer_groups (__main__.TestProtocolsBGP.test_bgp_03_peer_groups) ... ok
DEBUG - test_bgp_04_afi_ipv4 (__main__.TestProtocolsBGP.test_bgp_04_afi_ipv4) ... ok
DEBUG - test_bgp_05_afi_ipv6 (__main__.TestProtocolsBGP.test_bgp_05_afi_ipv6) ... ok
DEBUG - test_bgp_06_listen_range (__main__.TestProtocolsBGP.test_bgp_06_listen_range) ... ok
DEBUG - test_bgp_07_l2vpn_evpn (__main__.TestProtocolsBGP.test_bgp_07_l2vpn_evpn) ... ok
DEBUG - test_bgp_08_l2vpn_evpn_advertise_all_vni_conflicts_with_vrf_vni (__main__.TestProtocolsBGP.test_bgp_08_l2vpn_evpn_advertise_all_vni_conflicts_with_vrf_vni) ... ok
DEBUG - test_bgp_08_l2vpn_evpn_advertise_all_vni_restriction (__main__.TestProtocolsBGP.test_bgp_08_l2vpn_evpn_advertise_all_vni_restriction) ... ok
DEBUG - test_bgp_09_distance_and_flowspec (__main__.TestProtocolsBGP.test_bgp_09_distance_and_flowspec) ... ok
DEBUG - test_bgp_10_vrf_simple (__main__.TestProtocolsBGP.test_bgp_10_vrf_simple) ... ok
DEBUG - test_bgp_11_confederation (__main__.TestProtocolsBGP.test_bgp_11_confederation) ... ok
DEBUG - test_bgp_12_v6_link_local (__main__.TestProtocolsBGP.test_bgp_12_v6_link_local) ... ok
DEBUG - test_bgp_13_vpn (__main__.TestProtocolsBGP.test_bgp_13_vpn) ... ok
DEBUG - test_bgp_14_remote_as_peer_group_override (__main__.TestProtocolsBGP.test_bgp_14_remote_as_peer_group_override) ... ok
DEBUG - test_bgp_15_local_as_ebgp (__main__.TestProtocolsBGP.test_bgp_15_local_as_ebgp) ... ok
DEBUG - test_bgp_16_import_rd_rt_compatibility (__main__.TestProtocolsBGP.test_bgp_16_import_rd_rt_compatibility) ... ok
DEBUG - test_bgp_17_import_rd_rt_compatibility (__main__.TestProtocolsBGP.test_bgp_17_import_rd_rt_compatibility) ... ok
DEBUG - test_bgp_18_deleting_import_vrf (__main__.TestProtocolsBGP.test_bgp_18_deleting_import_vrf) ... ok
DEBUG - test_bgp_19_deleting_default_vrf (__main__.TestProtocolsBGP.test_bgp_19_deleting_default_vrf) ... ok
DEBUG - test_bgp_20_import_rd_rt_compatibility (__main__.TestProtocolsBGP.test_bgp_20_import_rd_rt_compatibility) ... ok
DEBUG - test_bgp_21_import_unspecified_vrf (__main__.TestProtocolsBGP.test_bgp_21_import_unspecified_vrf) ... ok
DEBUG - test_bgp_22_interface_mpls_forwarding (__main__.TestProtocolsBGP.test_bgp_22_interface_mpls_forwarding) ... ok
DEBUG - test_bgp_23_vrf_interface_mpls_forwarding (__main__.TestProtocolsBGP.test_bgp_23_vrf_interface_mpls_forwarding) ... ok
DEBUG - test_bgp_24_srv6_sid (__main__.TestProtocolsBGP.test_bgp_24_srv6_sid) ... ok
DEBUG - test_bgp_25_ipv4_labeled_unicast_peer_group (__main__.TestProtocolsBGP.test_bgp_25_ipv4_labeled_unicast_peer_group) ... ok
DEBUG - test_bgp_26_ipv6_labeled_unicast_peer_group (__main__.TestProtocolsBGP.test_bgp_26_ipv6_labeled_unicast_peer_group) ... ok
DEBUG - test_bgp_27_route_reflector_client (__main__.TestProtocolsBGP.test_bgp_27_route_reflector_client) ... ERROR
DEBUG - test_bgp_28_peer_group_member_all_internal_or_external (__main__.TestProtocolsBGP.test_bgp_28_peer_group_member_all_internal_or_external) ... ok
DEBUG - test_bgp_29_peer_group_remote_as_equal_local_as (__main__.TestProtocolsBGP.test_bgp_29_peer_group_remote_as_equal_local_as) ... ok
DEBUG - test_bgp_30_import_vrf_routemap (__main__.TestProtocolsBGP.test_bgp_30_import_vrf_routemap) ... ok
DEBUG - test_bgp_31_as_notation (__main__.TestProtocolsBGP.test_bgp_31_as_notation) ... ok
DEBUG - test_bgp_32_bfd_strict (__main__.TestProtocolsBGP.test_bgp_32_bfd_strict) ... ok
DEBUG - test_bgp_33_vpn_route_target_idempotency (__main__.TestProtocolsBGP.test_bgp_33_vpn_route_target_idempotency) ... ok
DEBUG - test_bgp_34_interface_neighbor (__main__.TestProtocolsBGP.test_bgp_34_interface_neighbor) ... ok
DEBUG - test_bgp_35_queue_limit (__main__.TestProtocolsBGP.test_bgp_35_queue_limit) ... ok
DEBUG - test_bgp_99_bmp (__main__.TestProtocolsBGP.test_bgp_99_bmp) ... ok
DEBUG - 
DEBUG - ======================================================================
DEBUG - ERROR: test_bgp_27_route_reflector_client (__main__.TestProtocolsBGP.test_bgp_27_route_reflector_client)
DEBUG - ----------------------------------------------------------------------
DEBUG - Traceback (most recent call last):
DEBUG -   File "/usr/libexec/vyos/tests/smoke/cli/test_protocols_bgp.py", line 1626, in test_bgp_27_route_reflector_client
DEBUG -     set_neighbor_func(neighbors[func_count], remote_as_type)
DEBUG -   File "/usr/libexec/vyos/tests/smoke/cli/test_protocols_bgp.py", line 1606, in _set_neighbor_1
DEBUG -     self.cli_set(base_path + ['neighbor', neighbor] + interface_cmd + ['remote-as', remote_as_type])
DEBUG -   File "/usr/libexec/vyos/tests/smoke/cli/base_vyostest_shim.py", line 93, in cli_set
DEBUG -     self._session.set(path, value)
DEBUG -   File "/usr/lib/python3/dist-packages/vyos/configsession.py", line 227, in set
DEBUG -     self.__run_command([SET] + path + value)
DEBUG -   File "/usr/lib/python3/dist-packages/vyos/configsession.py", line 216, in __run_command
DEBUG -     raise ConfigSessionError(output)
DEBUG - vyos.configsession.ConfigSessionError: Set failed
DEBUG - 
DEBUG - 
DEBUG - ----------------------------------------------------------------------
DEBUG - Ran 37 tests in 408.293s
DEBUG - 
DEBUG - FAILED (errors=1)

Details

Version
rolling, 1.5.1, 1.4.6
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Event Timeline

c-po changed the task status from Open to In progress.
c-po claimed this task.
c-po triaged this task as Normal priority.