Hyper-V 2012 host crashes on backup or guest migration

SUMMARY

When SMB3 shares are used to store Hyper-V guests and the hosts are running unitrends agent 8.2 or 9.0, it is possible for some actions to cause a host to crash or blue screen when the CBT driver is used.

ISSUE

When SMB3 shares are used to store Hyper-V guests and the hosts are running unitrends agent 8.2 or 9.0, it is possible for some actions to cause a host to crash or blue screen when the CBT driver is used.   This issue has only been seen on 3rd party NAS/SAN hardware.  A native Windows 2012R2 server hosting an SMB3 file share uses a different driver process for creating remote VSS snapshots and we are not aware of any issues with this process at this time.  It is only when leveraging non-Windows SMB3 hosted CSVs, or 3rd party platforms that leverage custom VSS drivers or lack native domain integration as a Windows host server that this issue has been seen.  

RESOLUTION

1.  Ensure Unitrends Appliance is on release 9.2.0 or higher and the current 9.2.0 agents have been deployed to each node in the cluster.  
Note, when updating a Hyper-V client agent, Unitrends recommends guests be suspended or migrated first to other nodes as this agent may require a reboot of the node to complete upgrade of the CBT drivers.  

2.  Remove the Hyper-V CBT driver from each Hyper-V host and replace the CBT tracking functions with the Unitrends hash scanning Hyper-V incremental process following the knowledge article How to uninstall and re-install CBT on a Hyper-V cluster.  Repeat this process for EACH node in cluster.

3.  Edit the master.ini configuration file for the agent running on the Hyper-V host:  These instructions assume command line use for Hyper-V Core.  Alternatively, this may be done from the Hyper-V host's Windows interface. Repeat this process for EACH node in cluster.

a. Log in to the host for which you want to disable the hash-scan method. 
b. Access the command line and enter the following command to access the agent directory: 
cd /PCBP
c.  To view the contents of the master.ini file, run the following command:  
notepad Master.ini
d.  Scroll to the bottom and under [Hyper-V], set the following parameters are set (note these parameters may not be present, this section and values can be manually added following standard INI file formatting):  HashEnabled=True and CBTDriverEnabled=False.  

4.  Upon completing these instructions, the master.ini file should have entry added similar to the snippet below.  Once confirmed save this file.  Do not forget to do this on EACH node in the cluster.  
[BareMetalSection]

[Hyper-V]
HashEnabled=True
CBTDriverEnabled=False

[usnap]
USnapEnable=TRUE

5.  Perform a new full backup of each Hyper-V guest in the cluster.  Once CBT is disabled current schedules will fail until a new full is run. 

CAUSE

SMB3 does not provide committed writes for data tracking.  As the local node that is part of the cluster is actually what is writing to and from the VHD or VHDX disk in the CSV, the CBT driver attempts to monitor that traffic, however without committed writes, the Windows SCSI utilities does not relay completion of that activity.  When a snapshot of the remote disk is taken by VSS, either by a backup, or by a VM migration or other disk shadow processes, the CBT driver can in some cases cause the host to crash because of that uncommitted disk activity.  

The CBT driver is not currently compatible with SMB3 shares and is intended exclusively for DASD, SCSI, iSCSI, or FC connected CSVs.  When using SMB3, disabling the CBT driver is currently recommended.  

 

NOTES

With CBT Disabled, backup processes will use a scanning hash method for backup.   This processes creates block signature files in the guest path in the hyper-V host that are referenced during backups.  During incremental backups, the entire VHD file is scanned and compared to the signature map to determine which blocks have changed. This will mean incremental backups will take significantly longer than backups using CBT technology.  If this time cannot be accounted for in RPO/RTO windows, and the CBT drivers cannot be used in your environment, then the use of agent-based backups may be recommended to provide faster backup operations. Hyper-V guests protected with agent backups can be easily recovered back to hyper-V (or even to dissimilar VM platforms or to physical in some cases) using IBMR recovery technology and/or Windows instant Recovery for hyper-V 2012R2/2016.  

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Contact us