Page MenuHomeVyOS Platform

Another corruption on upgrade
Closed, ResolvedPublicBUG

Description

While upgrading multiple VyOS instances I hit config file corruption again.

image.png (239×742 px, 89 KB)

I checked if the config was correct before doing a reboot and it was fine.

I have 3 remarks about this:

  1. It seems like you need a little uptime before this happens. Going back and retrying the upgrade gives no problems
  2. Maybe unrelated, but on another lab router I had database corruption in a container after an upgrade. These were also corrupted files on the filesystem.
  3. Also, when rebooting, the /usr filesystem is not unmounted and throws an error.

I have to upgrade another router with a bigger uptime. Before upgrading I will remount the FS in sync mode to see what happens.

Details

Difficulty level
Unknown (require assessment)
Version
1.4-20230510073 1.4-202306080317
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Event Timeline

On another router I've changed the base mounts to sync ie: mount /dev/md127 /boot -o remount,rw,noatime,sync before upgrading, but after a reboot it came back corrupted.

The configfile was complete before I started the upgrade.

Also on this one, booting back in the old version, removing the new version, re-adding the new version and rebooting fixed it.

Just tried to upgrade from April image and its failed with this error :(

The system currently has the following image(s) installed:

   1: 1.4-rolling-202307050317 (default boot)
   2: 1.4-rolling-202304050317 (running image)

EDIT: as roedie pointed out, a reinstall worked to actually get past this but got a migration error on firewall. It would not load at all with some python errors. I tried to manually load config but same thing. No possibility to proceed.

Is there any way to figure out what in the firewall config is broken?

I'm encountering similar issues on upgrade as well: the configuration fails to apply because of a broken config.boot file (incomplete; stops partway through, perhaps a migration issue?)

@blackhole at the Grub menu, if you have the image selected that you wish to boot to, hit e to edit the params for that entry and add vyos-debug and vyos-config-debug to the other parameters there to get more logging output on boot, which would presumably detail any config migration issues. Source: Relevant VyOS Docs link

A developer I work with reports this:

After nearly 30 reinstalls the upgrade finally blew up. The issue is in the initial copying of the config file, between lines 182 and 219 in the install script. That's where it happens... what's causing it / how it happens is still unclear...

So, with that commentary, I'm thinking @roedie that your filesystem suspicions are perhaps a thread worth following. (But I wouldn't know where to start diagnosing that.)

Viacheslav claimed this task.
Viacheslav added a subscriber: Viacheslav.

Reopen it if necessary