High Utility Queue Depth Issues (post-processing)
The Utility Queue on the PHD VBA's virtual machine console within the hypervisor client displays the number of utility processes waiting to take place. These include post-processes and deletes for retention.
Figure 1: A PHD VBA's virtual machine console with a high Utility Queue Depth
A consistently high Utility Queue Depth can indicate that these processes are not completing in time, and that the PHD VBA may be overloaded. The queue depth may reach zero eventually, but after reboot, you may notice it grow once again. This is because the VBA begins to process the deletes by first working through the queue, but it in some cases, may never able to complete the actual delete. This is related to how the overall backup process is prioritized:
- Backup runs
- Post processing occurs
- Deletes are processed
All processes must complete within a 24 hour timeframe. If there is not enough time to complete all processes, some may fall behind and the utility queue will continue to grow.
(Another indicator that this is happening is visible on startup where many deleting incomplete backup log messages are displayed).
To correct this issue and prevent possible problems, including running out of backup storage because deletes are not completing, or issues with backups not completing correctly due to the incomplete post processing, you can do the following:
- Deploy additional PHD VBAs and load balance your backup jobs to allow all processes to complete properly.
- Allow more time between jobs on an individual PHD VBA to provide more time for all post-processing to complete.
- Disable backups to allow post-processing and deletes to complete.
These issues may also be related to specific post-processing errors. If you are encountering these issues, please contact support:
In version 5.4.1, the email reports were updated to include queue depth information for both retention and post-processing. These can be used as an ongoing VBA health indicator. Consistently high queue depths may indicate an issue.
In version 6.0 the VBA Health Monitor was added to the Tools menu to help monitor Post Processing Queue Depth.