How to troubleshoot virtual machine VSS snapshot and quiesce issues during VMware backups
<meta http-equiv="Pragma" content="no-cache" /> <meta http-equiv="Cache-Control" content="no-cache,no-Store" /> How to troubleshoot virtual machine VSS snapshot and quiesce issues during VMware backups.[if gt IE 8]><!<![endif][if ! lte IE 8]><!<![endif][if (gt IE 6) & (lt IE 9)]> <link rel="stylesheet" href="../assets/css/veritas-ie8.css"> <![endif][if lte IE 9]> <![endif][if lte IE 6]> <link rel="stylesheet" href="//universal-ie6-css.googlecode.com/files/ie6.1.1.css" media="screen, projection"> <![endif]
StartFragment
Problem
How to troubleshoot virtual machine VSS snapshot and quiesce issues during VMware backups.
Error Message
0xe000959a - An attempt to take a snapshot of a virtual machine failed because it was unable to quiesce an application. Final error category: Resource Errors
For additional information regarding this error refer to link V-79-57344-38298
Cause
Quiesce and other snapshot errors during virtual machine backups can occur from:
Multiple VSS providers installed.
An unsupported snapshot provider.
An unsupported operating system.
Overloaded systems.
Custom pre-freeze-script and/or post-thaw-script.
The snapshot file at its maximum size is unable to fit into a datastore
Solution
Snapshot Providers:
VSS (VMware Tools)Symantec VSS Provider
Service: VMware Snapshot ProviderService: Backup Exec VSS Provider
VMware distributes a VSS snapshot provider with VMware ToolsInstalled by pushing RAWS (Remote Agent for Windows Systems) or manual installation
Install: http://kb.vmware.com/kb/1008965Push Install: 000028625
Manual Install: 000026760
Uninstall: TECH129864Uninstall: TECH130940
NOTE: When installing the Backup Exec Remote Agent, there is also an option to install AOFO (Advanced Open File Option). The only time AOFO needs to be pushed to a virtual machine is if the guest operating system is Windows 2000.
The following information determines which snapshot provider should be installed:
ESX version on host.
Operating system on guest.
Backup Exec GRT (Granular Restore Technology).
It is recommended to install and use the Backup Exec VSS Provider if:
Using system or application GRT (Granular Restore Technology) on virtual machine backups.
Backing up guest machines with a Windows 2008, Windows 7 or earlier operating system on an ESX 4.0 or earlier (VMware VSS support for Windows 7 and Server 2008 gains functionality in ESX 4.0 update 1 and later)
Otherwise, the VMware Snapshot Provider is sufficient for backup.
When installing the Backup Exec Remote Agent on a guest machine for GRT enabled virtual machine backups, the VMware Snapshot provider is automatically uninstalled and the BE VSS Provider is installed (this is why a reboot is required). The Backup Exec VSS Provider cannot exist in a guest alongside the VMware Snapshot Provider as with both installed on a guest will cause quiesced snapshots to hang the guest and timeout.
Once the Backup Exec Remote Agent is installed, thus installing the BE VSS Provider, do NOT re-install the VMware Tools VSS Provider as this will re-create the conflict condition and cause quiesce and/or snapshot errors. To see which providers are installed, launch Windows Services on the guest machine and confirm one of services from the table above are listed. If more than one provider is installed, determine which provider is needed and follow the steps below to uninstall the other:
TECH130940 - How perform an automated or manual uninstall of the Backup Exec Remote Agent for Windows Systems (RAWS) - 11d and later.
TECH129864 - How to perform an automated and/or manual uninstall of the VMware Snapshot Provider.
To confirm the VMware Snapshot provider is correctly uninstalled, confirm the following:
Confirm there is no VMware Snapshot Provider service listed in Windows services.
Confirm the "VSS" folder is deleted from <volume>:\Program Files\VMware\VMware Tools\Drivers.
Confirm the VMware Snapshot Provider {564d7761-7265-2056-5353-2050726f7669} is deleted from HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Providers.
Note: If steps 1-3 are not confirmed, please review TECH129864 for the manual uninstall steps.
Note : If the BE VSS Provider is correctly installed on the guest machine, the BeVssProvider service will show in Windows services and BE VSS Provider will be listed under the "vssadmin list providers" command ONLY during a backup.
If either is seen and no backups are running of the guest machine, there could be an issue with one of the following:
The post-thaw script.
The snapshot provider is hung.
VMware Snapshot Provider was re-installed.
The guest machine was never restarted after installing the Backup Exec Remote Agent.
To resolve this, perform the following:
Run the "post-thaw-script.bat" from the VSS Provider directory in the Backup Exec install directory of the guest machine (Default path is C:\Program Files\Symantec\Backup Exec\RAWS\VSS Provider)
Reboot the guest machine.
A test to confirm snapshots are working correctly within the virtual environment is to run a snapshot from inside vCenter/vSphere client.
Remove the check from "Snapshot the Virtual machine's memory"
Check option "Quiesce guest file system."
Review the VSS logs on the guest C:\Program Files\Symantec\Backup Exec\RAWS\VSS Provider\logs. Note: If the logs do not provide enough information, edit the "Pre" and "Post" scripts: a. Edit the "Pre" and "Post" scripts in C:\Program Files\Symantec\Backup Exec\VSS Provider and %systemroot% b. Add -logging to the end of each argument. Example. %Programfiles%\Symantec\Backup Exec\BE VSS Provider\BeVssRequestor.exe" -post2 -logscreen -logging. There are three arguments in each script.
A debug of the virtual machine may provide additional information about the failing snapshot or quiesce errors. Please refer to the following TechNote for more information: TECH63926
If the above configurations are corrected and there are still quiesce errors, Symantec recommends reviewing and decreasing the load on both the guest and host machines to allow the systems enough time to create snapshots. The following links from VMware may also be helpful:
http://kb.vmware.com/kb/1007696 - Troubleshooting Volume Shadow Copy (VSS) quiesce related issues.
http://kb.vmware.com/kb/1015180 - Understanding virtual machine snapshots in VMware ESX.
Snapshot File is unable to fit into the specified Datastore:
Create a quiesced snapshot in the vSphere console for the virtual machine that is failing with the error. If it produces: File <unspecifi-ed filename> is larger than the maximum size supported by datastore '<unspecifi-ed datastore> and the snapshot fails then have him review: http://kb.vmware.com/kb/1012384 If further assistance is required with this issue please contact VMware technical support.
EndFragment