Page MenuHomeVyOS Platform

VyOS freeze during creating VMware snapshot with quiesce a file system option
Closed, InvalidPublic

Description

If snapshot in VMware (5.5) is requested with "quiesce file system" VyOS freezes (Can't login to console/ssh, openvpn stops working, general routing/forwarding is not affected). This is default behaviour of Veeam 7 during backup jobs.

According to https://github.com/vmware/open-vm-tools/issues/16 , problem is caused by broken open-vm-tools / ESXi / kernel. It was already fixed in open-vm-tools.

open-vm-tools 9.4.* seems to be affected, uprgrading to 9.10 should fix this.

VyOS 1.1.6 and 1.1.7 affected.
VC6.5R1 is not affected (Using open-vm-tools 8.*)

Details

Difficulty level
Normal (likely a few hours)
Version
1.1.7

Event Timeline

syncer triaged this task as Normal priority.
syncer added a project: VyOS 1.1.x (1.1.8).
syncer added subscribers: higebu, dmbaturin, syncer, Unknown Object (User).

@dmbaturin @higebu can we just upgrade tools or it has some dependencies in the kernel?

@brona Do you know exact reproducing procedure?
We noticed this problem with Veeam Backup and Replication tool that uses quiescence, but we could never reproduce it simply by making a snapshot of a test VM.

Also, I can't edit this task sadly, but no way the difficulty is less than an hour... It's pretty involved, open-vm-tools source repo doesn't make it particulary easy to debianize...

In T67#1060, @dmbaturin wrote:

@brona Do you know exact reproducing procedure?
We noticed this problem with Veeam Backup and Replication tool that uses quiescence, but we could never reproduce it simply by making a snapshot of a test VM.

Maybe just bad luck, but 4x create snapshot 4x frozen router (Always on ESXi 5.5)

brona changed Difficulty level from Easy (less than an hour) to Normal (likely a few hours).May 18 2016, 7:19 AM

Maybe just bad luck, but 4x create snapshot 4x frozen router (Always on ESXi 5.5)

Unfortunately we have the same problem on the latest VMware vSphere 6.x platform and the latest VyOS version.
We did some backups with Commvault with consistent snapshots and all VyOS systems (about 14) were affected.

In T67#1060, @dmbaturin wrote:

@brona Do you know exact reproducing procedure?
We noticed this problem with Veeam Backup and Replication tool that uses quiescence, but we could never reproduce it simply by making a snapshot of a test VM.

Maybe try to execute the action on multiple machines in parallel.
We were affected when we did it for the whole datacenter.

@adestis @dmbaturin We had an issue like this, and the workaround is here:

  1. Add these parameters to /etc/vmware-tools/tools.conf
[vmbackup]
enableSyncDriver = false
  1. Reboot the VM.

@higebu: Thank you we will give it a try.
I think that should be the default setting as long as the problem is not fixed.

@dmbaturin @higebu
lets adopt this setting by default
crash consistent backups should be enough for VyOS image level backup use case

Complete solution(open-vm-tools and drivers update) will be added in 1.2.x
for 1.1.x series i going to create separate ticket with workaround provided by @higebu

dmbaturin raised the priority of this task from Normal to High.May 24 2018, 6:23 PM
dmbaturin edited projects, added VyOS 1.2 Crux (VyOS 1.2.0-rc1); removed VyOS 1.2 Crux.
dmbaturin moved this task from Need Triage to In Progress on the VyOS 1.2 Crux board.

Needs to be tested in 1.2.0

syncer lowered the priority of this task from High to Normal.Jun 10 2018, 2:59 AM

it was not included in 1.1.8 iso
and not applies to 1.2
OVA for 1.1.8 contain manual fix