Page MenuHomeVyOS Platform

Update system image without enough space for the files can to break the system
Open, NormalPublicBUG

Assigned To
Authored By
Viacheslav
Mar 20 2024, 1:06 PM
Referenced Files
F4240723: config.boot
Mar 20 2024, 1:06 PM
F4240722: boot-epa2.log
Mar 20 2024, 1:06 PM
F4240721: boot.log
Mar 20 2024, 1:06 PM

Description

Update the system image without enough space for the files can to break the system
The update was from VyOS-1.4-epa2, and as I understand, it was enough space for the system, but not enough for the files

vyos@r15-left:~$ add system image vyos-1.4-rolling-202403200309-amd64.iso
Validating image checksums
What would you like to name this image? (Default: 1.4-rolling-202403200309) 
Would you like to set the new image as the default one for boot? [Y/n] 
An active configuration was found. Would you like to copy it to the new image? [Y/n] 
Copying configuration directory

Cleaning up
Unmounting target filesystems
Removing temporary files
Error: [('/opt/vyatta/etc/config/support/r15-left.tech-support-archive.2024-03-11-135556.tgz', '/usr/lib/live/mount/persistence/boot/1.4-rolling-202403200309/rw/opt/vyatta/etc/config/support/r15-left.tech-support-archive.2024-03-11-135556.tgz', "[Errno 28] No space left on device: '/opt/vyatta/etc/config/support/r15-left.tech-support-archive.2024-03-11-135556.tgz' -> '/usr/lib/live/mount/persistence/boot/1.4-rolling-202403200309/rw/opt/vyatta/etc/config/support/r15-left.tech-support-archive.2024-03-11-135556.tgz'"), ('/opt/vyatta/etc/config/user-data', '/usr/lib/live/mount/persistence/boot/1.4-rolling-202403200309/rw/opt/vyatta/etc/config/user-data', "[Errno 28] No space left on device: '/usr/lib/live/mount/persistence/boot/1.4-rolling-202403200309/rw/opt/vyatta/etc/config/user-data'"), ('/opt/vyatta/etc/config/archive', '/usr/lib/live/mount/persistence/boot/1.4-rolling-202403200309/rw/opt/vyatta/etc/config/archive', "[Errno 28] No space left on device: '/usr/lib/live/mount/persistence/boot/1.4-rolling-202403200309/rw/opt/vyatta/etc/config/archive'"), ('/opt/vyatta/etc/config/config.boot.2024-02-07-083626.pre-migration', '/usr/lib/live/mount/persistence/boot/1.4-rolling-202403200309/rw/opt/vyatta/etc/config/config.boot.2024-02-07-083626.pre-migration', "[Errno 28] No space left on device: '/opt/vyatta/etc/config/config.boot.2024-02-07-083626.pre-migration' -> '/usr/lib/live/mount/persistence/boot/1.4-rolling-202403200309/rw/opt/vyatta/etc/config/config.boot.2024-02-07-083626.pre-migration'"), ('/opt/vyatta/etc/config/config.boot.bkup', '/usr/lib/live/mount/persistence/boot/1.4-rolling-202403200309/rw/opt/vyatta/etc/config/config.boot.bkup', "[Errno 28] No space left on device: '/opt/vyatta/etc/config/config.boot.bkup' -> '/usr/lib/live/mount/persistence/boot/1.4-rolling-202403200309/rw/opt/vyatta/etc/config/config.boot.bkup'"), ('/opt/vyatta/etc/config/config.boot.2024-03-18-095745.pre-migration', '/usr/lib/live/mount/persistence/boot/1.4-rolling-202403200309/rw/opt/vyatta/etc/config/config.boot.2024-03-18-095745.pre-migration', "[Errno 28] No space left on device: '/opt/vyatta/etc/config/config.boot.2024-03-18-095745.pre-migration' -> '/usr/lib/live/mount/persistence/boot/1.4-rolling-202403200309/rw/opt/vyatta/etc/config/config.boot.2024-03-18-095745.pre-migration'")]
vyos@r15-left:~$ 
vyos@r15-left:~$ 
vyos@r15-left:~$ ls
vyos-1.4-rolling-202403200309-amd64.iso
vyos@r15-left:~$ rm -rf /opt/vyatta/etc/config/support/r15-left.tech-support-archive.2024-03-11-135556.tgz
vyos@r15-left:~$ 
vyos@r15-left:~$ add system image vyos-1.4-rolling-202403200309-amd64.iso
Validating image checksums
What would you like to name this image? (Default: 1.4-rolling-202403200309) 
Would you like to set the new image as the default one for boot? [Y/n] 
An active configuration was found. Would you like to copy it to the new image? [Y/n] 
Copying configuration directory
Cleaning up
Unmounting target filesystems
Removing temporary files
Error: [Errno 17] File exists: '/usr/lib/live/mount/persistence/boot/1.4-rolling-202403200309/rw/opt/vyatta/etc/config'
vyos@r15-left:~$ show system image 
Name        Default boot    Running
----------  --------------  ---------
1.4.0-epa2  Yes             Yes
1.3.6
vyos@r15-left:~$

Try again but with the name 1.4-roll

vyos@r15-left:~$ add system image vyos-1.4-rolling-202403200309-amd64.iso
Validating image checksums
What would you like to name this image? (Default: 1.4-rolling-202403200309) 1.4-roll
Would you like to set the new image as the default one for boot? [Y/n] 
An active configuration was found. Would you like to copy it to the new image? [Y/n] 
Copying configuration directory
Would you like to copy SSH host keys? [Y/n] 
Copying SSH host keys
Copying system image files
Cleaning up
Unmounting target filesystems
Removing temporary files
vyos@r15-left:~$ 
vyos@r15-left:~$ 
vyos@r15-left:~$ show system image 
Name        Default boot    Running
----------  --------------  ---------
1.4-roll    Yes
1.4.0-epa2                  Yes
1.3.6
vyos@r15-left:~$ reboot now

It cannot correctly boot anymore:

[   22.780433] vyos-router[1147]: OSError: [Errno 30] Read-only file system: '/etc/frr/daemons'
[   22.796201] vyos-router[901]: could not reset FRR config ... failed!
[   23.420770] vyos-router[901]: Starting VyOS router: migrate
[   23.421779] vyos-router[1220]: Traceback (most recent call last):
[   23.422391] vyos-router[1220]:   File "/usr/libexec/vyos/run-config-migration.py", line 86, in <module>
[   23.423032] vyos-router[1220]:     main()
[   23.423713] vyos-router[1220]:   File "/usr/libexec/vyos/run-config-migration.py", line 65, in main
[   23.424318] vyos-router[1220]:     cmd(f'cp -p {config_file_name} {backup_file_name}')
[   23.425058] vyos-router[1220]:   File "/usr/lib/python3/dist-packages/vyos/utils/process.py", line 155, in cmd
[   23.429471] vyos-router[1220]:     raise OSError(code, feedback)
[   23.430039] vyos-router[1220]: PermissionError: [Errno 1] failed to run command: cp -p /opt/vyatta/etc/config/config.boot /opt/vyatta/etc/config/config.boot.2024-03-20-125322.pre-migration
[   23.430600] vyos-router[1220]: returned:
[   23.430982] vyos-router[1220]: exit code: 1
[   23.480168] vyos-router[1243]: mkdir: cannot create directory ‘/boot/grub’: No space left on device
[   23.480855] vyos-router[1245]: mount: /boot/grub: mount point does not exist.
[   23.481400] vyos-router[1245]:        dmesg(1) may have more information after failed mount system call.
[   23.481610] vyos-router[901]: Couldn't bind mount /boot/grub
[   24.123011] vyos-router[901]:  configure failed!
[   24.311607] vyos-router[1331]: chmod: changing permissions of '/': Read-only file system
[   24.938529] vyos-config[905]: Configuration error

debian login:

The boot.log file is attached

And cannot boot to the previous 1.4.0-epa2 (I can boot but in read-only mode)
log attached:

[   24.344193] vyos-config[909]: Configuration error
[   24.455598] vyos-router[1311]: chmod: changing permissions of '/': Read-only file system

Welcome to VyOS - r15-left ttyS0

Config.boot (from 1.4.0-epa2) before migration

Details

Difficulty level
Unknown (require assessment)
Version
1.4.0-epa2
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)

Event Timeline

At least tech-support-archive.tgz MUST not copied between updates

dmbaturin added a project: Restricted Project.Apr 4 2024, 7:48 PM
syncer lowered the priority of this task from High to Normal.
syncer added subscribers: dmbaturin, syncer.

@dmbaturin can you add space check into the upgrade scripts