VyOS rolling ISO buildings fails during smoketests.
Description
Details
- Version
- 1.5
- Is it a breaking change?
- Perfectly compatible
- Issue type
- Bug (incorrect behavior)
Related Objects
- Mentioned In
- rVYOSONEXd028be894fcb: T5562: Cleanup netns for smoketest load-balancing wan
rVYOSONEX87ab93326dfd: Merge pull request #2247 from sever-sever/T5562-sag
rVYOSONEX63cf32004caf: T5562: Cleanup netns for smoketest load-balancing wan
rVYOSONEX849499f44f6e: T5562: Cleanup netns for smoketest load-balancing wan
rVYOSONEX795fcb74b646: Merge pull request #2230 from sever-sever/T5562 - Mentioned Here
- T5518: Add MLD protocol support
Event Timeline
Some observations:
2023-09-08T18:08:40.9131605Z DEBUG - Executing command: qemu-system-x86_64 -name "TESTVM-UEFI" -smp sockets=1,cpus=3,cores=1 -cpu host -bios /usr/share/OVMF/OVMF_CODE.fd -m 3G -vga none -nographic -machine accel=kvm -uuid d27cf29e-4419-4407-8f82-dc73d1acd184 -cpu host -boot d -drive file=build/live-image-amd64.hybrid.iso,format=raw,if=none,media=cdrom,id=drive-cd1,readonly=on -device ahci,id=achi0 -device ide-cd,bus=achi0.0,drive=drive-cd1,id=cd1,bootindex=1 -enable-kvm -netdev user,id=n0,net=192.0.2.0/24,dhcpstart=192.0.2.101,dns=192.0.2.10 -device virtio-net-pci,netdev=n0,mac=52:54:00:00:00:00,romfile="" -netdev user,id=n1 -device virtio-net-pci,netdev=n1,mac=52:54:00:00:00:01,romfile="" -netdev user,id=n2 -device virtio-net-pci,netdev=n2,mac=52:54:00:00:00:02,romfile="" -netdev user,id=n3 -device virtio-net-pci,netdev=n3,mac=52:54:00:00:00:03,romfile="" -netdev user,id=n4 -device virtio-net-pci,netdev=n4,mac=52:54:00:00:00:04,romfile="" -netdev user,id=n5 -device virtio-net-pci,netdev=n5,mac=52:54:00:00:00:05,romfile="" -netdev user,id=n6 -device virtio-net-pci,netdev=n6,mac=52:54:00:00:00:06,romfile="" -netdev user,id=n7 -device virtio-net-pci,netdev=n7,mac=52:54:00:00:00:07,romfile="" -device virtio-scsi-pci,id=scsi0 -drive format=raw,file=testinstall-20230908-171217-cbb7.img,if=none,media=disk,id=drive-hd1,readonly=off -device scsi-hd,bus=scsi0.0,drive=drive-hd1,id=hd1,bootindex=2 2023-09-08T18:08:40.9152507Z DEBUG - qemu-system-x86_64: warning: This family of AMD CPU doesn't support hyperthreading(3) 2023-09-08T18:08:40.9153908Z DEBUG - Please configure -smp options properly or try enabling topoext feature.
I think the above:
-smp sockets=1,cpus=3,cores=1
is supposed to be written as:
-smp 3,sockets=1,cores=3,threads=1
And regarding fails/errors:
1. 2023-09-08T18:08:41.0761215Z DEBUG - Running Testcase: /usr/libexec/vyos/tests/smoke/cli/test_load_balancing_wan.py 2023-09-08T18:08:41.0761763Z DEBUG - test_check_chains (__main__.TestLoadBalancingWan.test_check_chains) ... Cannot find device "ceth0" 2023-09-08T18:08:41.0762059Z DEBUG - Cannot find device "ceth1" 2023-09-08T18:08:41.0762350Z DEBUG - Cannot find device "ceth2" 2023-09-08T18:08:41.0762579Z DEBUG - ok 2023-09-08T18:08:41.0763118Z DEBUG - test_table_routes (__main__.TestLoadBalancingWan.test_table_routes) ... Cannot find device "ceth0" 2023-09-08T18:08:41.0763410Z DEBUG - Cannot find device "ceth1" 2023-09-08T18:08:41.0763697Z DEBUG - Cannot find device "ceth2" 2023-09-08T18:08:41.0763912Z DEBUG - ok 2. 2023-09-08T18:08:41.0777997Z DEBUG - Running Testcase: /usr/libexec/vyos/tests/smoke/cli/test_netns.py 2023-09-08T18:08:41.0778408Z DEBUG - test_netns_create (__main__.NetNSTest.test_netns_create) ... FAIL 2023-09-08T18:08:41.0778972Z DEBUG - test_netns_interface (__main__.NetNSTest.test_netns_interface) ... FAIL 2023-09-08T18:08:41.0779177Z DEBUG - 2023-09-08T18:08:41.0779541Z DEBUG - ====================================================================== 2023-09-08T18:08:41.0779943Z DEBUG - FAIL: test_netns_create (__main__.NetNSTest.test_netns_create) 2023-09-08T18:08:41.0780324Z DEBUG - ---------------------------------------------------------------------- 2023-09-08T18:08:41.0780634Z DEBUG - Traceback (most recent call last): 2023-09-08T18:08:41.0781083Z DEBUG - File "/usr/libexec/vyos/tests/smoke/cli/test_netns.py", line 40, in tearDown 2023-09-08T18:08:41.0781379Z DEBUG - self.assertFalse(tmp) 2023-09-08T18:08:41.0781745Z DEBUG - AssertionError: 'ns203 (id: 3)\nnsC (id: 2)' is not false 2023-09-08T18:08:41.0781962Z DEBUG - 2023-09-08T18:08:41.0782331Z DEBUG - ====================================================================== 2023-09-08T18:08:41.0782752Z DEBUG - FAIL: test_netns_interface (__main__.NetNSTest.test_netns_interface) 2023-09-08T18:08:41.0783132Z DEBUG - ---------------------------------------------------------------------- 2023-09-08T18:08:41.0783520Z DEBUG - Traceback (most recent call last): 2023-09-08T18:08:41.0783967Z DEBUG - File "/usr/libexec/vyos/tests/smoke/cli/test_netns.py", line 40, in tearDown 2023-09-08T18:08:41.0784258Z DEBUG - self.assertFalse(tmp) 2023-09-08T18:08:41.0784635Z DEBUG - AssertionError: 'ns203 (id: 3)\nnsC (id: 2)' is not false 3. 2023-09-08T18:08:41.0909390Z DEBUG - Running Testcase: /usr/libexec/vyos/tests/smoke/cli/test_service_https.py 2023-09-08T18:08:41.0909809Z DEBUG - test_api_auth (__main__.TestHTTPSService.test_api_auth) ... ERROR 2023-09-08T18:08:41.0910243Z DEBUG - test_certificate (__main__.TestHTTPSService.test_certificate) ... ok 2023-09-08T18:08:41.0910637Z DEBUG - test_default (__main__.TestHTTPSService.test_default) ... ok 2023-09-08T18:08:41.0911067Z DEBUG - test_server_block (__main__.TestHTTPSService.test_server_block) ... ok 2023-09-08T18:08:41.0911265Z DEBUG - 2023-09-08T18:08:41.0911630Z DEBUG - ====================================================================== 2023-09-08T18:08:41.0912034Z DEBUG - ERROR: test_api_auth (__main__.TestHTTPSService.test_api_auth) 2023-09-08T18:08:41.0912464Z DEBUG - ---------------------------------------------------------------------- 2023-09-08T18:08:41.0912786Z DEBUG - Traceback (most recent call last): 2023-09-08T18:08:41.0913287Z DEBUG - File "/usr/libexec/vyos/tests/smoke/cli/base_vyostest_shim.py", line 114, in wrapped 2023-09-08T18:08:41.0913581Z DEBUG - return f(*args, **kwargs) 2023-09-08T18:08:41.0913842Z DEBUG - ^^^^^^^^^^^^^^^^^^ 2023-09-08T18:08:41.0914361Z DEBUG - File "/usr/libexec/vyos/tests/smoke/cli/test_service_https.py", line 164, in test_api_auth 2023-09-08T18:08:41.0914736Z DEBUG - success = r.json()['data']['SystemStatus']['success'] 2023-09-08T18:08:41.0915005Z DEBUG - ~~~~~~~~^^^^^^^^ 2023-09-08T18:08:41.0915266Z DEBUG - KeyError: 'data' 4. 2023-09-08T18:08:41.0972158Z DEBUG - Running Testcase: /usr/libexec/vyos/tests/smoke/cli/test_system_conntrack.py 2023-09-08T18:08:41.0972647Z DEBUG - test_conntrack_hash_size (__main__.TestSystemConntrack.test_conntrack_hash_size) ... ok 2023-09-08T18:08:41.0973143Z DEBUG - test_conntrack_ignore (__main__.TestSystemConntrack.test_conntrack_ignore) ... ERROR 2023-09-08T18:08:41.0973686Z DEBUG - test_conntrack_module_enable (__main__.TestSystemConntrack.test_conntrack_module_enable) ... ok 2023-09-08T18:08:41.0974172Z DEBUG - test_conntrack_options (__main__.TestSystemConntrack.test_conntrack_options) ... ok 2023-09-08T18:08:41.0974374Z DEBUG - 2023-09-08T18:08:41.0974735Z DEBUG - ====================================================================== 2023-09-08T18:08:41.0975208Z DEBUG - ERROR: test_conntrack_ignore (__main__.TestSystemConntrack.test_conntrack_ignore) 2023-09-08T18:08:41.0975576Z DEBUG - ---------------------------------------------------------------------- 2023-09-08T18:08:41.0975890Z DEBUG - Traceback (most recent call last): 2023-09-08T18:08:41.0976444Z DEBUG - File "/usr/libexec/vyos/tests/smoke/cli/test_system_conntrack.py", line 274, in test_conntrack_ignore 2023-09-08T18:08:41.0976713Z DEBUG - self.cli_commit() 2023-09-08T18:08:41.0977317Z DEBUG - File "/usr/libexec/vyos/tests/smoke/cli/base_vyostest_shim.py", line 76, in cli_commit 2023-09-08T18:08:41.0977615Z DEBUG - self._session.commit() 2023-09-08T18:08:41.0978089Z DEBUG - File "/usr/lib/python3/dist-packages/vyos/configsession.py", line 183, in commit 2023-09-08T18:08:41.0978395Z DEBUG - out = self.__run_command([COMMIT]) 2023-09-08T18:08:41.0978692Z DEBUG - ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2023-09-08T18:08:41.0979204Z DEBUG - File "/usr/lib/python3/dist-packages/vyos/configsession.py", line 139, in __run_command 2023-09-08T18:08:41.0979528Z DEBUG - raise ConfigSessionError(output) 2023-09-08T18:08:41.0979930Z DEBUG - vyos.configsession.ConfigSessionError: [ firewall ] 2023-09-08T18:08:41.0980324Z DEBUG - ConfigError('Failed to apply configuration: /run/nftables- 2023-09-08T18:08:41.0980759Z DEBUG - ct.conf:15:38-53: Error: No\nsuch file or directory; did you mean set 2023-09-08T18:08:41.0981296Z DEBUG - ‘A_conntracktest’ in table ip\n‘vyos_filter’? ip saddr 192.0.2.1 2023-09-08T18:08:41.0981683Z DEBUG - ip daddr @A_conntracktest\ncounter notrack comment 2023-09-08T18:08:41.0982122Z DEBUG - "ignore-2"\n^^^^^^^^^^^^^^^^ /run/nftables-ct.conf:33:38-55: Error: No 2023-09-08T18:08:41.0982652Z DEBUG - such file or\ndirectory; did you mean set ‘A6_conntracktest6’ in table 2023-09-08T18:08:41.0983137Z DEBUG - ip6\n‘vyos_filter’? ip6 saddr fe80::1 ip6 daddr 2023-09-08T18:08:41.0983472Z DEBUG - @A6_conntracktest6\ncounter notrack comment 2023-09-08T18:08:41.0983762Z DEBUG - "ignore-12"\n^^^^^^^^^^^^^^^^^^') 2023-09-08T18:08:41.0983966Z DEBUG - 2023-09-08T18:08:41.0984220Z DEBUG - [[firewall]] failed 2023-09-08T18:08:41.0984486Z DEBUG - [ system conntrack ] 2023-09-08T18:08:41.0984927Z DEBUG - Failed to apply configuration: /run/nftables-ct.conf:15:38-53: Error: No 2023-09-08T18:08:41.0985373Z DEBUG - such file or directory; did you mean set ‘A_conntracktest’ in table ip 2023-09-08T18:08:41.0985793Z DEBUG - ‘vyos_filter’? ip saddr 192.0.2.1 ip daddr @A_conntracktest 2023-09-08T18:08:41.0986111Z DEBUG - counter notrack comment "ignore-2" 2023-09-08T18:08:41.0986587Z DEBUG - ^^^^^^^^^^^^^^^^ /run/nftables-ct.conf:33:38-55: Error: No such file or 2023-09-08T18:08:41.0986999Z DEBUG - directory; did you mean set ‘A6_conntracktest6’ in table ip6 2023-09-08T18:08:41.0987457Z DEBUG - ‘vyos_filter’? ip6 saddr fe80::1 ip6 daddr @A6_conntracktest6 2023-09-08T18:08:41.0987769Z DEBUG - counter notrack comment "ignore-12" 2023-09-08T18:08:41.0988018Z DEBUG - ^^^^^^^^^^^^^^^^^^ 2023-09-08T18:08:41.0988222Z DEBUG - 2023-09-08T18:08:41.0988493Z DEBUG - [[system conntrack]] failed 2023-09-08T18:08:41.0988744Z DEBUG - Commit failed 5. 2023-09-08T18:08:41.1035900Z DEBUG - Running Testcase: /usr/libexec/vyos/tests/smoke/cli/test_vrf.py 2023-09-08T18:08:41.1036351Z DEBUG - test_vrf_assign_interface (__main__.VRFTest.test_vrf_assign_interface) ... ok 2023-09-08T18:08:41.1036720Z DEBUG - test_vrf_bind_all (__main__.VRFTest.test_vrf_bind_all) ... ok 2023-09-08T18:08:41.1037167Z DEBUG - test_vrf_disable_forwarding (__main__.VRFTest.test_vrf_disable_forwarding) ... ok 2023-09-08T18:08:41.1037770Z DEBUG - test_vrf_ip_ipv6_protocol_non_existing_route_map (__main__.VRFTest.test_vrf_ip_ipv6_protocol_non_existing_route_map) ... ok 2023-09-08T18:08:41.1038309Z DEBUG - test_vrf_ip_protocol_route_map (__main__.VRFTest.test_vrf_ip_protocol_route_map) ... ok 2023-09-08T18:08:41.1038805Z DEBUG - test_vrf_ipv6_protocol_route_map (__main__.VRFTest.test_vrf_ipv6_protocol_route_map) ... ok 2023-09-08T18:08:41.1039300Z DEBUG - test_vrf_link_local_ip_addresses (__main__.VRFTest.test_vrf_link_local_ip_addresses) ... ok 2023-09-08T18:08:41.1039723Z DEBUG - test_vrf_loopbacks_ips (__main__.VRFTest.test_vrf_loopbacks_ips) ... ok 2023-09-08T18:08:41.1040144Z DEBUG - test_vrf_static_route (__main__.VRFTest.test_vrf_static_route) ... FAIL 2023-09-08T18:08:41.1040341Z DEBUG - 2023-09-08T18:08:41.1040734Z DEBUG - ====================================================================== 2023-09-08T18:08:41.1041153Z DEBUG - FAIL: test_vrf_static_route (__main__.VRFTest.test_vrf_static_route) 2023-09-08T18:08:41.1041529Z DEBUG - ---------------------------------------------------------------------- 2023-09-08T18:08:41.1041841Z DEBUG - Traceback (most recent call last): 2023-09-08T18:08:41.1042345Z DEBUG - File "/usr/libexec/vyos/tests/smoke/cli/test_vrf.py", line 230, in test_vrf_static_route 2023-09-08T18:08:41.1042748Z DEBUG - self.assertIn(f' ip route {prefix} {next_hop}', frrconfig) 2023-09-08T18:08:41.1043277Z DEBUG - AssertionError: ' ip route 10.0.100.0/24 192.0.100.1' not found in 'vrf red\n vni 100\nexit-vrf\n!'
For the tests above, which configs are actually being used?
For example case 5 above (test_vrf_static_route) I would expect it to fail because 192.0.100.1 is not a valid next-hop according to:
https://github.com/vyos/vyos-1x/blob/current/smoketest/config-tests/basic-vyos
https://github.com/vyos/vyos-1x/blob/current/smoketest/configs/basic-vyos
https://github.com/vyos/vyos-1x/blob/current/smoketest/configs/vrf-basic
Also regarding case 5 I dont think "vni {table}" is actually being set by vyos-configd.
This is for example how a static route within vrf looks like in VyOS 1.5-rolling-202309080021:
set vrf name INTERNET protocols static route 0.0.0.0/0 next-hop 192.168.0.1 distance '1' set vrf name INTERNET table '101'
and here is the output of /run/frr/config/frr.conf:
vrf INTERNET ip route 0.0.0.0/0 192.168.0.1 exit-vrf
https://github.com/vyos/vyos-rolling-nightly-builds/actions/runs/6131805039/job/16642103169
Still errors in:
27875 DEBUG - Running Testcase: /usr/libexec/vyos/tests/smoke/cli/test_interfaces_ethernet.py
28673 DEBUG - Running Testcase: /usr/libexec/vyos/tests/smoke/cli/test_protocols_pim6.py
28995 DEBUG - Running Testcase: /usr/libexec/vyos/tests/smoke/cli/test_system_conntrack.py
Regarding the failing smoketest test_system_conntrack.py (test_conntrack_ignore):
https://github.com/vyos/vyos-1x/blob/current/smoketest/scripts/cli/test_system_conntrack.py#L276
I think its a combo of how verify_nftables is being called and how the arrays nftables_search and nftables6_search are constructed.
Proposed fixes:
1:
Change how verify_nftables is being called (this way the extracted ruleset wont have packet and bytes counters in the output):
self.verify_nftables(nftables_search, 'raw', False, '-s') self.verify_nftables(nftables6_search, 'ip6 raw', False, '-s')
2:
Adjust arrays nftables_search and nftables6_search to match output of nft which verify_nftables is using:
nftables_search = [ ['ip saddr 192.0.2.1 ip daddr 192.0.2.2 tcp dport 22 counter notrack'], ['ip saddr 192.0.2.1 ip daddr @A_conntracktest counter notrack'] ] nftables6_search = [ ['ip6 saddr fe80::1 ip6 daddr fe80::2 tcp dport 22 counter notrack'], ['ip6 saddr fe80::1 ip6 daddr @A6_conntracktest6 counter notrack'], ['ip6 saddr fe80::1 ip6 daddr != fe80::3 counter notrack'] ]
Unless the comments must be included in the verify aswell?
Because then it should be:
nftables_search = [ ['ip saddr 192.0.2.1 ip daddr 192.0.2.2 tcp dport 22 counter notrack comment "ignore-1"'], ['ip saddr 192.0.2.1 ip daddr @A_conntracktest counter notrack comment "ignore-2"'] ] nftables6_search = [ ['ip6 saddr fe80::1 ip6 daddr fe80::2 tcp dport 22 counter notrack comment "ignore-11"'], ['ip6 saddr fe80::1 ip6 daddr @A6_conntracktest6 counter notrack comment "ignore-12"'], ['ip6 saddr fe80::1 ip6 daddr != fe80::3 counter notrack comment "ignore-13"'] ]
Fix by @sever regarding those failing conntrack smoketest: https://github.com/vyos/vyos-1x/pull/2234
Latest run https://github.com/vyos/vyos-rolling-nightly-builds/actions/runs/6138721359/job/16655876943
Fixed:
1: test_protocols_pim6.py
https://github.com/vyos/vyos-rolling-nightly-builds/actions/runs/6138721359/job/16655876943#step:10:28678
2: test_system_conntrack.py
https://github.com/vyos/vyos-rolling-nightly-builds/actions/runs/6138721359/job/16655876943#step:10:28974
Still fails:
3: test_interfaces_ethernet.py
https://github.com/vyos/vyos-rolling-nightly-builds/actions/runs/6138721359/job/16655876943#step:10:27880
Looks like leftover from dhcp-tests (all the fails within this testsuite is regarding the leftovers aka tearDown fails):
DEBUG - ====================================================================== DEBUG - FAIL: test_dhcp_vrf (__main__.EthernetInterfaceTest.test_dhcp_vrf) DEBUG - ---------------------------------------------------------------------- DEBUG - Traceback (most recent call last): DEBUG - File "/usr/libexec/vyos/tests/smoke/cli/test_interfaces_ethernet.py", line 145, in tearDown DEBUG - self.assertNotIn(AF_INET, ifaddresses(intf)) DEBUG - AssertionError: 2 unexpectedly found in {17: [{'addr': '52:54:00:00:00:00', 'broadcast': 'ff:ff:ff:ff:ff:ff'}], 2: [{'addr': '192.0.2.101', 'netmask': '255.255.255.0', 'broadcast': '192.0.2.255'}], 10: [{'addr': 'fe80::5054:ff:fe00:0%eth0', 'netmask': 'ffff:ffff:ffff:ffff::/64'}]}