Page MenuHomeVyOS Platform

Smoketests: reorganize folderstructure for embedded configttests
In progress, LowPublicFEATURE REQUEST

Description

The config load tests currently provided via https://github.com/vyos/vyos-1x/tree/02034e4fd77e8af64a7f6a66e5da7fefcd4a9e90/smoketest/configs have also an assertion folder to validate the resulting configuration. https://github.com/vyos/vyos-1x/tree/02034e4fd77e8af64a7f6a66e5da7fefcd4a9e90/smoketest/config-tests

Both seem a bit unrelated and this should be re-ogranized for new developers inckl. a proper README how this works.

As explained to @Oleksandr via Slack:

Sure - the files under smoketests/configs/ are real VyOS configurations adopted from e.g. customers or (as I am a lazy person - the networks I operate). The reason is that we have some real router configurations which we can throw at our system to test config migration and also if all those confis still apply after we change something.
Then after some time we came to the idea that it's not 100% correct to only load the configurations but rather also validate the migrators. The content in smoketests/config-tests(not a fan of that location tbh.) is actually the state of the CLI commands that MUST be present after a configuration is migrated.
Example:
make testc in one step will load smoketests/configs/fancy-vyos-config and needs to migrate it to the latest CLI. After migration is done it verifies that the loaded configuration can be translated to the set commands defined in smoketests/config-tests/fancy-vyos-config and if those set commands are present oin the out put of show configuration commands - or an ionternal wrapper to this representation, I do not know off hand, the test passed.
There is always a 1:1 relation between the files. In VyOS 1.5 and current I too the effor to create all the required set test results under smoketests/config-tests but did not on 1.4 branch.
If you have a migration script and the config is migrated - you should always check if the newly migrated path is correct, by changing the line in smoketests/config-tests - or add it if its a new CLI line generated by the migrator.
Note: The migrated configuration must not match 100% - absent lines in smoketests/config-tests are not treated as failure (yet) so it is up to the PR author to manually take care about this. It will only fail if the line content is different after migration (edited)

While working on this and backporting the file movings to 1.4.4 or 1.4.5 we should also increase the test coverage by adding missing assertion files for VyOS 1.4.x

Details

Version
-
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Feature (new functionality)