Page MenuHomeVyOS Platform
Create Task
Maniphest T5910

Grub problem(?) Serial Console no longer working
Closed, ResolvedPublicBUG
Actions

Description

Installing 1.4.0-rc1 onto PCEngines APU6b4 with onboard drive causes the following messages:

GRUB loading.
Welcome to GRUB!

No further comms on that serial port are observed, even after machine boots.
I can eventually log in via an ssh connection over ethernet, but there is now no path for rescuing this router.

Vyos config-mode displays:

show system console
 device ttyS0 {
     speed 115200
 }

Permissions on ttys:

ls -l /dev/ttyS*
crw--w---- 1 root tty     4, 64 Jan  8 16:46 /dev/ttyS0
crw-rw---- 1 root dialout 4, 65 Jan  8 16:17 /dev/ttyS1
crw-rw---- 1 root dialout 4, 66 Jan  8 16:17 /dev/ttyS2
crw-rw---- 1 root dialout 4, 67 Jan  8 16:17 /dev/ttyS3

The machine has an onboard serial to USB connector physically attached to ttyS0. The serial cable is actually a usb cable plugged into that connector. The setup worked in a previous version of 1.4-rolling.

Details

Difficulty level
Unknown (require assessment)
Version
VyOS 1.4.0-rc1
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Behavior change
Issue type
Bug (incorrect behavior)

Related Objects
Search...

Event Timeline

fghorow created this task.Jan 8 2024, 4:53 PM
fghorow updated the task description. (Show Details)Jan 8 2024, 5:00 PM
pasik added a subscriber: pasik.Jan 8 2024, 5:07 PM
jestabro claimed this task.Jan 8 2024, 6:07 PM
dmbaturin triaged this task as High priority.Jan 8 2024, 7:06 PM
jestabro added a comment.Jan 8 2024, 7:15 PM

Just to clarify the issue: note that one can specify the choice of serial console during the 'install image' process; are you not seeing the boot messages with this setting ? or are you unable to set it during installation ? Thanks.

fghorow added a comment.EditedJan 9 2024, 12:40 AM

I replied to jestabro via email a couple of hours back, but noticed that it didn't show up here. For the benefit of anybody else interested in this bug I'm cutting and pasting the email below:

  • cut here ---

Here is the complete output from install image from a clean install via a usb-key:

install image
Welcome to VyOS installation!
This command will install VyOS to your permanent storage.
Would you like to continue? [y/N] y
What would you like to name this image? (Default: 1.4.0-rc1)
Please enter a password for the "vyos" user (Default: vyos)
What console should be used by default? (K: KVM, S: Serial, U: USB-Serial)? (Default: K) S
Probing disks
2 disk(s) found
Would you like to configure RAID-1 mirroring? [Y/n] n
The following disks were found:
Drive: /dev/sda (111.8 GB)
Drive: /dev/sdb (14.6 GB)
Which one should be used for installation? (Default: /dev/sda)
Installation will delete all data on the drive. Continue? [y/N] y
Searching for data from previous installations
Found data from previous installation:
       1.4.0-rc1 on /dev/sda3
Would you like to copy data to the new image? [Y/n]
Would you like to use all the free space on the drive? [Y/n]
Creating partition table...
Creating temporary directories
Mounting new partitions
Creating a configuration file
Copying system image files
Installing GRUB configuration files
Installing GRUB to the drive
Cleaning up
Unmounting target filesystems
Removing temporary files
The image installed successfully; please reboot now.

And here is the output from booting that installed image:

SeaBIOS (version rel-1.16.0.1-0-g77603a32)

Press F10 key now for boot menu

Booting from Hard Disk...
GRUB loading.
Welcome to GRUB!

That is the same GRUB output as before. The result is that (because I copied my config) I am still able to ssh in on ethernet, but no serial console is available.

Selecting “K” for kvm results in the same output from the bootloader, but the machine actually fails to boot AFAICT. The (2 populated) ethernet port activity lights did not stop like they did in the serial (S) boot attempt, and my laptop did not get a DHCP assignment from the router. That’s after more than 10 minutes of waiting after attempting to reboot from the USB install image procedure.

Selecting U for ‘USB-Serial’ reports:

Searching for data from previous installations
No previous installation found

(This implies I will not have access to the ethernet and ssh. Since I figured out how to recover from this situation earlier today by clean-installing an earlier version of Vyos, I proceeded anyway.) The observed result appears similar to the result from selecting “K” — the ethernet port blinking LEDs do not stop. I only waited 5 minutes before giving up this time.

I suspect that in both the K and the U console situations, the machine never actually makes it out of the BIOS, but the serial console is not working so I can’t verify that.

marc_s added a subscriber: marc_s.Jan 10 2024, 9:23 AM

See also forum thread @ https://forum.vyos.io/t/grub-menu-fails-to-load-on-serial-only-devices-with-no-kvm/

fghorow added a comment.Jan 10 2024, 2:58 PM

OK, the grub serial config described here got me as far as seeing the Grub selection screen at boot time.

The serial console is still inoperative for me after the machine successfully boots.

fghorow added a comment.EditedJan 10 2024, 6:29 PM

OK, a little digging around in the grub configs led me to a "workaround" solution for the serial console problem after the machine is booted.

The workaround is to ignore the line number specified in this link and to actually _read_ the grub stanzas. I needed to set the ,115200 at the end of the line in the if nest that was clearly not booting into password reset or recovery. Setting the other two wouldn't hurt anything either. ;-)

Clearly these hacky solutions need to be encapsulated in the install scripts (somehow; or at least better documented).

I'm a happy (hacky?) camper now.

@jestabro

jestabro added a comment.Jan 11 2024, 3:41 AM

@fghorow Thank you for the details; much appreciated. I've reproduced the issue, and the fix, and a proper update will be added to the install scripts for 1.4/1.5.

fghorow added a comment.Jan 11 2024, 3:35 PM

@jestabro Thanks for that!

jestabro added a project: VyOS 1.5 Circinus.Jan 11 2024, 8:03 PM

PR:
https://github.com/vyos/vyos-1x/pull/2806

jestabro added a subtask: T5923: Config mode system_console.py is not aware of revised GRUB file structure.Jan 11 2024, 8:27 PM
marvin added a subscriber: marvin.Jan 16 2024, 2:06 AM
jestabro closed subtask T5923: Config mode system_console.py is not aware of revised GRUB file structure as Resolved.Jan 17 2024, 5:52 AM
jestabro added a comment.Feb 8 2024, 3:06 PM

This is resolved in T5923; any other related concerns, reopen or open a new task.

jestabro closed this task as Resolved.Feb 8 2024, 3:06 PM
jestabro moved this task from Need Triage to Finished on the VyOS 1.5 Circinus board.
jestabro moved this task from Need Triage to Finished on the VyOS 1.4 Sagitta board.