Page MenuHomeVyOS Platform

ModemManager hijacks serial console
Open, NormalPublicBUG

Description

When a router boots with ModemManager enabled, eg. when using a wwan interface, ModemManager tries to connect to the local serial console to probe for modems.
This makes the local serial console some times un-usable as it changes baudrate etc on the port and it is not always reverted back to the initial serial console settings(115200 8n1).

This is quite a problem as i cannot guarantee to have a working serial console to my device at all times.

these are the settings the serial port is stuck with when the fault happens:

vyos@vyos:~$ sudo stty -F /dev/ttyS0 -a
speed 57600 baud; rows 24; columns 80; line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^A; eol = <undef>; eol2 = <undef>; swtch = <undef>; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R; werase = ^W; lnext = ^V; discard = ^O; min = 1; time = 0;
-parenb -parodd -cmspar cs8 hupcl -cstopb cread clocal -crtscts
-ignbrk -brkint -ignpar -parmrk -inpck -istrip -inlcr -igncr -icrnl -ixon -ixoff -iuclc -ixany -imaxbel iutf8
-opost -olcuc -ocrnl -onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
-isig -icanon -iexten -echo -echoe -echok -echonl -noflsh -xcase -tostop -echoprt -echoctl -echoke -flusho -extproc

interesting configuration:

interfaces {
    wwan wwan0 {
        address dhcp
        apn internet
        authentication {
            password ****************
            username dj
        }
    }
}
}

system {
    console {
        device ttyS0 {
            speed 115200
        }
    }
}

probing log from /var/log/messages

Jun 26 10:22:17 vyos systemd[1]: Starting ModemManager.service - Modem Manager...
Jun 26 10:22:17 vyos ModemManager[2894]: [1719397337.785697] ModemManager (version 1.20.4) starting in system bus...
Jun 26 10:22:17 vyos systemd[1]: Started ModemManager.service - Modem Manager.
Jun 26 10:22:20 vyos ModemManager[2894]: [1719397340.445045] [base-manager] couldn't check support for device '/sys/devices/pci0000:00/0000:00:0b.0/0000:04:00.0': not supported by any plugin
Jun 26 10:22:20 vyos ModemManager[2894]: [1719397340.445244] [base-manager] couldn't check support for device '/sys/devices/pci0000:00/0000:00:0c.0/0000:05:00.0': not supported by any plugin
Jun 26 10:22:20 vyos ModemManager[2894]: [1719397340.445404] [base-manager] couldn't check support for device '/sys/devices/pci0000:00/0000:00:0e.0/0000:06:00.0': not supported by any plugin
Jun 26 10:22:20 vyos ModemManager[2894]: [1719397340.446631] [base-manager] couldn't check support for device '/sys/devices/pci0000:00/0000:00:0f.0/0000:07:00.0': not supported by any plugin
Jun 26 10:22:20 vyos ModemManager[2894]: [1719397340.446814] [base-manager] couldn't check support for device '/sys/devices/pci0000:00/0000:00:10.0/0000:08:00.0': not supported by any plugin
Jun 26 10:22:20 vyos ModemManager[2894]: [1719397340.447026] [base-manager] couldn't check support for device '/sys/devices/pci0000:00/0000:00:16.0/0000:0b:00.0': not supported by any plugin
Jun 26 10:22:20 vyos ModemManager[2894]: [1719397340.447226] [base-manager] couldn't check support for device '/sys/devices/pci0000:00/0000:00:16.0/0000:0b:00.1': not supported by any plugin
Jun 26 10:22:20 vyos ModemManager[2894]: [1719397340.448501] [base-manager] couldn't check support for device '/sys/devices/pci0000:00/0000:00:17.0/0000:0c:00.0': not supported by any plugin
Jun 26 10:22:20 vyos ModemManager[2894]: [1719397340.448688] [base-manager] couldn't check support for device '/sys/devices/pci0000:00/0000:00:17.0/0000:0c:00.1': not supported by any plugin
Jun 26 10:22:44 vyos ModemManager[2894]: [1719397364.319091] [device /sys/devices/pci0000:00/0000:00:1a.0] creating modem with plugin 'generic' and '1' ports
Jun 26 10:22:44 vyos ModemManager[2894]: [1719397364.323159] [plugin/generic] could not grab port ttyS0: Cannot add port 'tty/ttyS0', unhandled port type
Jun 26 10:22:44 vyos ModemManager[2894]: [1719397364.323268] [base-manager] couldn't create modem for device '/sys/devices/pci0000:00/0000:00:1a.0': Failed to find primary AT port
Jun 26 10:22:44 vyos vyos-configd[805]: [Errno 1] failed to run command: mmcli --modem 0 --simple-disconnect
Jun 26 10:22:45 vyos PermissionError: [Errno 1] failed to run command: mmcli --modem 0 --simple-disconnect
Jun 26 10:22:45 vyos cmd 'mmcli --modem 0 --simple-disconnect'
Jun 26 10:22:45 vyos error: couldn't find modem

example output when probing reverts the previous settings:

[   28.257757] vyos-router[1082]: Waiting for NICs to settle down: settled in 0sec..
[   34.634638] vyos-router[1082]: Mounting VyOS Config...done.
�`���`���`���`���`���`��~x�~~x�~[   84.851602] vyos-router[1082]: Starting VyOS router: migrate system configure failed!
          [   85.424457] vyos-config[1088]: Configuration error

Welcome to VyOS - vyos ttyS0

vyos login:

example of a un-usable serial console:

[   28.617425] vyos-router[1108]: Waiting for NICs to settle down: settled in 0sec..
[   35.062536] vyos-router[1108]: Mounting VyOS Config...done.
�`���`���`���`�怞f`�x��xf�~~��f��f~f���f���~�~��~��~���~f��f��f~f��~��f�~��f~f��f��~ff�����xf�~f~fx�f`��`�~~��f���xf�~�f����f~����xf�~f~f~�������f~��fx��f�怘�~f����f~��f��f��~��~`f���xf����~f���f�f�f�f�f怆������������������~��f��~��f�~�~����fx��f���������fx�`�������fx��f��������x�f��fx~������怘��f�f�~x�������f��ff�fx���xfx`�����fx��f�~�~����f��~ff�����`���x���fx�`����������fx����~��f��~��f�~�~����fxf���~fx`~�~`f昆�x����fx�`�������fxf���`f~`f���~`f����怘���~��f��~��f�~�~�������xf�xf~�`��fx�~��x������fx��`��fxf~fx���f����fx�~f����怘���~��f��~��f�~�~����f��~ff�`f�f��xf�`��f�f�~x�������f��ff��~��fx`��f�~�~����fx�`f�f���fx�~��fx�`���x����`����f����f�ff��ff���~��xf��`���怘��f�f�~x�������f��ff��~��fx`��f�~�~����f��~ff�`f�f��xf�`��fx��x��fx`�f�x`f�`f������f����f`��f�~���怘��f�f�~x�������f��ff��~��fx`��f�~�~����f��~ff�`f�f��`���fxf��x�~��fx�`��������x��f��~f�f��`����fxfx��怘��f�f�~x�������f��ff��~��fx`��f�~�~����fx��f���������fx�`�������fx��f��������x�f��fx~�����怘��f�f�~x�������f��ff��~��fx`��f�~�~����f��~ff�`f�fx~ff�f~�x���x��f��~ff�����`�f�f~`�~�������ff�f��怘��f�f�~x�������f��ff��~��fx`��f�~�~��f~����fx�~��fx�`������~����xxf��fx`fx�x��f���fx����`����怘��f�f�~x�������f��ff�fx���xfx`�����fx��f�~�~����f��~ff��~fx`~������fx�`����������fx��f��~ff���~fx`~����怘��f�f�~x�������f��ff�fx���xfx`�����fx��f�~�~����f��~ff��fxf���x��f���������fx��f��~ff���fxf���~fx`~����怘��f�f�~x�������f��ff�fx���xfx`�����fx��f�~�~����f��~ff���������

There seems to be a race condition if the console will be unusable or not as it does not always happen.

Details

Difficulty level
Unknown (require assessment)
Version
1.5-rolling-202406230022
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Bug (incorrect behavior)