Page MenuHomeVyOS Platform

Update system image without enough space for the files can to break the system
Open, NormalPublicBUG

Description

Update the system image without enough space for the files can to break the system
The update was from VyOS-1.4-epa2, and as I understand, it was enough space for the system, but not enough for the files

vyos@r15-left:~$ add system image vyos-1.4-rolling-202403200309-amd64.iso
Validating image checksums
What would you like to name this image? (Default: 1.4-rolling-202403200309) 
Would you like to set the new image as the default one for boot? [Y/n] 
An active configuration was found. Would you like to copy it to the new image? [Y/n] 
Copying configuration directory

Cleaning up
Unmounting target filesystems
Removing temporary files
Error: [('/opt/vyatta/etc/config/support/r15-left.tech-support-archive.2024-03-11-135556.tgz', '/usr/lib/live/mount/persistence/boot/1.4-rolling-202403200309/rw/opt/vyatta/etc/config/support/r15-left.tech-support-archive.2024-03-11-135556.tgz', "[Errno 28] No space left on device: '/opt/vyatta/etc/config/support/r15-left.tech-support-archive.2024-03-11-135556.tgz' -> '/usr/lib/live/mount/persistence/boot/1.4-rolling-202403200309/rw/opt/vyatta/etc/config/support/r15-left.tech-support-archive.2024-03-11-135556.tgz'"), ('/opt/vyatta/etc/config/user-data', '/usr/lib/live/mount/persistence/boot/1.4-rolling-202403200309/rw/opt/vyatta/etc/config/user-data', "[Errno 28] No space left on device: '/usr/lib/live/mount/persistence/boot/1.4-rolling-202403200309/rw/opt/vyatta/etc/config/user-data'"), ('/opt/vyatta/etc/config/archive', '/usr/lib/live/mount/persistence/boot/1.4-rolling-202403200309/rw/opt/vyatta/etc/config/archive', "[Errno 28] No space left on device: '/usr/lib/live/mount/persistence/boot/1.4-rolling-202403200309/rw/opt/vyatta/etc/config/archive'"), ('/opt/vyatta/etc/config/config.boot.2024-02-07-083626.pre-migration', '/usr/lib/live/mount/persistence/boot/1.4-rolling-202403200309/rw/opt/vyatta/etc/config/config.boot.2024-02-07-083626.pre-migration', "[Errno 28] No space left on device: '/opt/vyatta/etc/config/config.boot.2024-02-07-083626.pre-migration' -> '/usr/lib/live/mount/persistence/boot/1.4-rolling-202403200309/rw/opt/vyatta/etc/config/config.boot.2024-02-07-083626.pre-migration'"), ('/opt/vyatta/etc/config/config.boot.bkup', '/usr/lib/live/mount/persistence/boot/1.4-rolling-202403200309/rw/opt/vyatta/etc/config/config.boot.bkup', "[Errno 28] No space left on device: '/opt/vyatta/etc/config/config.boot.bkup' -> '/usr/lib/live/mount/persistence/boot/1.4-rolling-202403200309/rw/opt/vyatta/etc/config/config.boot.bkup'"), ('/opt/vyatta/etc/config/config.boot.2024-03-18-095745.pre-migration', '/usr/lib/live/mount/persistence/boot/1.4-rolling-202403200309/rw/opt/vyatta/etc/config/config.boot.2024-03-18-095745.pre-migration', "[Errno 28] No space left on device: '/opt/vyatta/etc/config/config.boot.2024-03-18-095745.pre-migration' -> '/usr/lib/live/mount/persistence/boot/1.4-rolling-202403200309/rw/opt/vyatta/etc/config/config.boot.2024-03-18-095745.pre-migration'")]
vyos@r15-left:~$ 
vyos@r15-left:~$ 
vyos@r15-left:~$ ls
vyos-1.4-rolling-202403200309-amd64.iso
vyos@r15-left:~$ rm -rf /opt/vyatta/etc/config/support/r15-left.tech-support-archive.2024-03-11-135556.tgz
vyos@r15-left:~$ 
vyos@r15-left:~$ add system image vyos-1.4-rolling-202403200309-amd64.iso
Validating image checksums
What would you like to name this image? (Default: 1.4-rolling-202403200309) 
Would you like to set the new image as the default one for boot? [Y/n] 
An active configuration was found. Would you like to copy it to the new image? [Y/n] 
Copying configuration directory
Cleaning up
Unmounting target filesystems
Removing temporary files
Error: [Errno 17] File exists: '/usr/lib/live/mount/persistence/boot/1.4-rolling-202403200309/rw/opt/vyatta/etc/config'
vyos@r15-left:~$ show system image 
Name        Default boot    Running
----------  --------------  ---------
1.4.0-epa2  Yes             Yes
1.3.6
vyos@r15-left:~$

Try again but with the name 1.4-roll

vyos@r15-left:~$ add system image vyos-1.4-rolling-202403200309-amd64.iso
Validating image checksums
What would you like to name this image? (Default: 1.4-rolling-202403200309) 1.4-roll
Would you like to set the new image as the default one for boot? [Y/n] 
An active configuration was found. Would you like to copy it to the new image? [Y/n] 
Copying configuration directory
Would you like to copy SSH host keys? [Y/n] 
Copying SSH host keys
Copying system image files
Cleaning up
Unmounting target filesystems
Removing temporary files
vyos@r15-left:~$ 
vyos@r15-left:~$ 
vyos@r15-left:~$ show system image 
Name        Default boot    Running
----------  --------------  ---------
1.4-roll    Yes
1.4.0-epa2                  Yes
1.3.6
vyos@r15-left:~$ reboot now

It cannot correctly boot anymore:

[   22.780433] vyos-router[1147]: OSError: [Errno 30] Read-only file system: '/etc/frr/daemons'
[   22.796201] vyos-router[901]: could not reset FRR config ... failed!
[   23.420770] vyos-router[901]: Starting VyOS router: migrate
[   23.421779] vyos-router[1220]: Traceback (most recent call last):
[   23.422391] vyos-router[1220]:   File "/usr/libexec/vyos/run-config-migration.py", line 86, in <module>
[   23.423032] vyos-router[1220]:     main()
[   23.423713] vyos-router[1220]:   File "/usr/libexec/vyos/run-config-migration.py", line 65, in main
[   23.424318] vyos-router[1220]:     cmd(f'cp -p {config_file_name} {backup_file_name}')
[   23.425058] vyos-router[1220]:   File "/usr/lib/python3/dist-packages/vyos/utils/process.py", line 155, in cmd
[   23.429471] vyos-router[1220]:     raise OSError(code, feedback)
[   23.430039] vyos-router[1220]: PermissionError: [Errno 1] failed to run command: cp -p /opt/vyatta/etc/config/config.boot /opt/vyatta/etc/config/config.boot.2024-03-20-125322.pre-migration
[   23.430600] vyos-router[1220]: returned:
[   23.430982] vyos-router[1220]: exit code: 1
[   23.480168] vyos-router[1243]: mkdir: cannot create directory ‘/boot/grub’: No space left on device
[   23.480855] vyos-router[1245]: mount: /boot/grub: mount point does not exist.
[   23.481400] vyos-router[1245]:        dmesg(1) may have more information after failed mount system call.
[   23.481610] vyos-router[901]: Couldn't bind mount /boot/grub
[   24.123011] vyos-router[901]:  configure failed!
[   24.311607] vyos-router[1331]: chmod: changing permissions of '/': Read-only file system
[   24.938529] vyos-config[905]: Configuration error

debian login:

The boot.log file is attached

And cannot boot to the previous 1.4.0-epa2 (I can boot but in read-only mode)
log attached:

[   24.344193] vyos-config[909]: Configuration error
[   24.455598] vyos-router[1311]: chmod: changing permissions of '/': Read-only file system

Welcome to VyOS - r15-left ttyS0

Config.boot (from 1.4.0-epa2) before migration

Details

Version
1.4.0-epa2
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Event Timeline

At least tech-support-archive.tgz MUST not copied between updates

dmbaturin added a project: Restricted Project.Apr 4 2024, 7:48 PM
syncer lowered the priority of this task from High to Normal.
syncer added subscribers: dmbaturin, syncer.

@dmbaturin can you add space check into the upgrade scripts

I suppose we could just add a "guesstimate" check, say, 2GB of free space. At the moment, rolling release images are about 600M, 2GB should be enough for a while, hopefully.