Page MenuHomeVyOS Platform

Hyper-V integration services not working on VyOS 1.4 (sagitta/current)
Closed, ResolvedPublicBUG

Description

Hyper-V LIS (Linux Integration Services) is not working on VyOS 1.4-rolling-202103190553 (sagitta/current)

  • Image built using Docker over WSL2
  • Instructions from https://docs.vyos.io/en/latest/contributing/build-vyos.html
  • Used sudo make openstack to build an image with cloud-init (I need cloud-init!)
  • Used qemu-img.exe convert -f raw built.iso -O vhdx -o subformat=dynamic MyVm.vhdx to convert generated .ISO to .VHDX
  • VM Gen2 created with secure boot disabled. Booted correctly from .VHDX

Everything appears to work. Except the Hyper-V integration services.

[From Hyper-V HOST]

C:\> Get-VMIntegrationService -VMName tst

VMName Name                    Enabled PrimaryStatusDescription SecondaryStatusDescription
------ ----                    ------- ------------------------ --------------------------
tst    Guest Service Interface True    No Contact
tst    Heartbeat               True    No Contact
tst    Key-Value Pair Exchange True    No Contact
tst    Shutdown                True    No Contact
tst    Time Synchronization    True    No Contact
tst    VSS                     True    No Contact

[From GUEST VM]

$ cat /etc/os-release

PRETTY_NAME="VyOS 1.4-rolling-202103190553 (sagitta)"
NAME="VyOS"
VERSION_ID="1.4-rolling-202103190553"
VERSION="1.4-rolling-202103190553 (sagitta)"
VERSION_CODENAME=buster
ID=vyos
HOME_URL="https://vyos.io"
SUPPORT_URL="https://support.vyos.io"
BUG_REPORT_URL="https://phabricator.vyos.net"
$ sudo apt update && sudo apt install --only-upgrade hyperv-daemons

Reading package lists... Done
Building dependency tree
Reading state information... Done
All packages are up to date.
Reading package lists... Done
Building dependency tree
Reading state information... Done
hyperv-daemons is already the newest version (4.19.171-2).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

P.S. It works with VyOS 1.3. Using the same build process with an image built from v1.3 (equuleus) branch:

[From Hyper-V HOST]

C:\> Get-VMIntegrationService -VMName tst

VMName Name                    Enabled PrimaryStatusDescription SecondaryStatusDescription
------ ----                    ------- ------------------------ --------------------------
tst    Guest Service Interface True    OK
tst    Heartbeat               True    OK
tst    Key-Value Pair Exchange True    OK                       The protocol version of the component installed in t...
tst    Shutdown                True    OK
tst    Time Synchronization    True    OK
tst    VSS                     True    OK                       The protocol version of the component installed in t...

Discussion:
https://forum.vyos.io/t/upgrade-to-latest-rolling-release-broke-hyper-v-integration-vmbus-unavailable/6792

Details

Difficulty level
Unknown (require assessment)
Version
1.4-rolling-202103190553 (sagitta/current)
Why the issue appeared?
Will be filled on close
Is it a breaking change?
Unspecified (possibly destroys the router)
Issue type
Unspecified (please specify)

Event Timeline

Confirming that this bug is still present on 1.4-rolling-20210505111.

I had a router running 1.3-rolling-202009300117 that was unaffected by this.

I'll switch it back to the 1.3 release to see why it was working there in the first place.

All of the required Hyper-V LIS modules appear to be completely missing from /etc/initramfs-tools/modules:

hv_vmbus
hv_storvsc
hv_blkvsc
hv_netvsc

Not sure if this is intentional as I know there's a custom VyOS build process that might be accounting for this in the initramfs modules in the pre-ISO state.

journalctl output doesn't look promising either:

journalctl -b -o cat --no-pager | egrep "Hyper|hyper|hv"
DMI: Microsoft Corporation Virtual Machine/Virtual Machine, BIOS Hyper-V UEFI Release v4.0 12/17/2019
Hypervisor detected: Microsoft Hyper-V
Hyper-V: features 0xe7f, hints 0x6002c, misc 0x17b2
Hyper-V Host Build:17763-10.0-1-0.1879
Hyper-V: LAPIC Timer Frequency: 0x30d40
tsc: Marking TSC unstable due to running on Hyper-V
Hyper-V: Using hypercall for remote TLB flush
clocksource: hyperv_clocksource_tsc_page: mask: 0xffffffffffffffff max_cycles: 0x24e6a1710, max_idle_ns: 440795202120 ns
Booting paravirtualized kernel on Hyper-V
Hyper-V: PV spinlocks disabled
Hyper-V: Using enlightened APIC (xapic mode)
clocksource: Switched to clocksource hyperv_clocksource_tsc_page
hv_vmbus: Vmbus version:3.0
hv_vmbus: registering driver hv_netvsc
hv_vmbus: registering driver hv_storvsc
hv_vmbus: registering driver hyperv_keyboard
Condition check resulted in Hyper-V key-value pair (KVP) daemon being skipped.
Condition check resulted in Hyper-V volume shadow copy service (VSS) daemon being skipped.
Condition check resulted in Hyper-V file copy service (FCOPY) daemon being skipped.

I just rolled back the router and it appears to have hv_utils module loading:

lsmod | egrep "hyper|hv"
hyperv_keyboard        16384  0
hv_utils               32768  2
hv_netvsc              65536  0
hv_storvsc             20480  1
scsi_transport_fc      65536  1 hv_storvsc
hv_vmbus               61440  4 hv_utils,hv_netvsc,hv_storvsc,hyperv_keyboard
scsi_mod              196608  5 sd_mod,hv_storvsc,scsi_transport_fc,usb_storage,libata

On the 1.4 version, hv_utils is gone:

lsmod | egrep "hyper|hv"
hyperv_keyboard        16384  0
hv_storvsc             24576  1
scsi_transport_fc      65536  1 hv_storvsc
hv_netvsc              77824  0
scsi_mod              208896  4 sd_mod,hv_storvsc,scsi_transport_fc,usb_storage
hv_vmbus               81920  3 hv_netvsc,hv_storvsc,hyperv_keyboard

hv_utils seems to be the missing link since this is what's pulling in some of the required functionality (VM Heartbeat, Shutdown, Timesync, and VSS):

sudo dmesg | egrep "Hyper|hyper|hv"
[    0.000000] Hypervisor detected: Microsoft Hyper-V
[    0.000000] Hyper-V: features 0xe7f, hints 0x2c, misc 0x17b2
[    0.000000] Hyper-V Host Build:9600-6.3-19-0.19101
[    0.000000] Hyper-V: LAPIC Timer Frequency: 0xc3500
[    0.000000] tsc: Marking TSC unstable due to running on Hyper-V
[    0.000000] Hyper-V: Using hypercall for remote TLB flush
[    0.133108] Hyper-V: Using MSR based APIC access
[    0.133115] clocksource: hyperv_clocksource_tsc_page: mask: 0xffffffffffffffff max_cycles: 0x24e6a1710, max_idle_ns: 440795202120 ns
[    0.168114] SRBDS: Unknown: Dependent on hypervisor status
[    0.219683] clocksource: Switched to clocksource hyperv_clocksource_tsc_page
[    1.380097] ata_piix 0000:00:07.1: Hyper-V Virtual Machine detected, ATA device ignore set
[    1.380690] hv_vmbus: Vmbus version:3.0
[    1.399795] hv_vmbus: registering driver hv_storvsc
[    1.400939] hv_vmbus: registering driver hv_netvsc
[   17.278024] hv_utils: Registering HyperV Utility Driver
[   17.278027] hv_vmbus: registering driver hv_util
[   17.278968] hv_utils: FCopy IC version 1.1
[   17.279601] hv_utils: Heartbeat IC version 3.0
[   17.282709] hv_utils: Shutdown IC version 3.0
[   17.283198] hv_utils: TimeSync IC version 3.0
[   17.283564] hv_utils: VSS IC version 5.0
[   17.342916] hv_vmbus: registering driver hyperv_keyboard
[   17.553996] hv_utils: KVP IC version 4.0

And there we go, the kernel module was never included on 1.4:

ls -lhaR /usr/lib/modules/*/kernel/* | grep hv
drwxr-xr-x  2 root root   55 Apr 22 21:21 hv
/usr/lib/modules/5.10.32-amd64-vyos/kernel/drivers/hv:
-rw-r--r--  1 root root  36K Apr 22 02:29 hv_balloon.ko
-rw-r--r--  1 root root 147K Apr 22 02:29 hv_vmbus.ko
-rw-r--r--  1 root root 138K Apr 22 02:29 hv_netvsc.ko
-rw-r--r--  1 root root  41K Apr 22 02:29 hv_storvsc.ko
-rw-r--r--  1 root root  24K Apr 22 02:29 hv_sock.ko

But it is on 1.3:

ls -lhaR /usr/lib/modules/*/kernel/* | grep hv
drwxr-xr-x  2 root root   74 Sep 29  2020 hv
/usr/lib/modules/4.19.148-amd64-vyos/kernel/drivers/hv:
-rw-r--r--  1 root root  34K Sep 29  2020 hv_balloon.ko
-rw-r--r--  1 root root  65K Sep 29  2020 hv_utils.ko
-rw-r--r--  1 root root 127K Sep 29  2020 hv_vmbus.ko
-rw-r--r--  1 root root 113K Sep 29  2020 hv_netvsc.ko
-rw-r--r--  1 root root  35K Sep 29  2020 hv_storvsc.ko
-rw-r--r--  1 root root  23K Sep 29  2020 hv_sock.ko

I'm not sure how/why this disappeared, but that would be your problem. You can't modprobe to add it after-the-fact -- it's just missing period.

I have the same problem. This is a kernel configuration issue. The following settings must be made:

CONFIG_CONNECTOR=y
CONFIG_HYPERV_UTILS=m

Then the problem will be solved.

vyos@vyos-vpn-14:~$ cat /etc/os-release 

    PRETTY_NAME="VyOS 1.4-rolling-202211241917 (current)"
    NAME="VyOS"
    VERSION_ID="1.4-rolling-202211241917"
    VERSION="1.4-rolling-202211241917 (current)"
    VERSION_CODENAME=bullseye
    ID=vyos
    HOME_URL="https://vyos.io"
    SUPPORT_URL="https://support.vyos.io"
    BUG_REPORT_URL="https://phabricator.vyos.net"

vyos@vyos-vpn-14:~$ lsmod | grep ^hv_
hv_balloon             36864  0
hv_utils               49152  3
hv_storvsc             24576  1
hv_netvsc              77824  0
hv_vmbus               86016  5 hv_balloon,hv_utils,hv_netvsc,hv_storvsc,hyperv_keyboard
dmbaturin set Issue type to Unspecified (please specify).
Viacheslav changed the task status from Open to Needs testing.Jan 10 2024, 7:50 PM
dmbaturin changed the task status from Needs testing to Needs reporter action.Jan 11 2024, 3:39 PM
dmbaturin added a subscriber: dmbaturin.

@nitopt Please test and let us know if it works for you.

Fixed! Thanks guys!

C:\> Get-VMIntegrationService -VMName tst-vyos

VMName   Name                    Enabled PrimaryStatusDescription SecondaryStatusDescription
------   ----                    ------- ------------------------ --------------------------
tst-vyos Guest Service Interface True    OK
tst-vyos Heartbeat               True    OK
tst-vyos Key-Value Pair Exchange True    OK                       The protocol version of the component installed in the virtual machine does not match the version expected by the hosting system
tst-vyos Shutdown                True    OK
tst-vyos Time Synchronization    True    OK
tst-vyos VSS                     True    OK                       The protocol version of the component installed in the virtual machine does not match the version expected by the hosting system

Image built from current sagitta branch.

Viacheslav claimed this task.
Viacheslav moved this task from Need Triage to Finished on the VyOS 1.4 Sagitta board.
Viacheslav added a project: VyOS 1.5 Circinus.
Viacheslav moved this task from Need Triage to Finished on the VyOS 1.5 Circinus board.

Also working with 1.5 / circinus:

Get-VMIntegrationService -VMName tst-vyos

VMName   Name                    Enabled PrimaryStatusDescription SecondaryStatusDescription
------   ----                    ------- ------------------------ --------------------------
tst-vyos Guest Service Interface True    OK
tst-vyos Heartbeat               True    OK
tst-vyos Key-Value Pair Exchange True    OK                       The protocol version of the component installed in...
tst-vyos Shutdown                True    OK
tst-vyos Time Synchronization    True    OK
tst-vyos VSS                     True    OK                       The protocol version of the component installed in...

Image built from current current branch.