Retention Queue queue was not emptied in a 24-hour period
Retention and Post Processing Queue Depths
In addition to the worker threads which perform the backup, restore, and replication processing on each PHD VBA, utility threads perform additional processing, including removing backups that fall outside the retention period (deletes) and postprocessing tasks.
These post-processes and deletes are queued and processed during the course of the day, with deletes being processed only when no other backup jobs are running. In some instances, if too many deletes or post-processes are pending, they may not finish in a 24 hour time period. When this happens, the utility queue depth continues to grow. Some of the problems that can result from utility queue depths that do not reach zero are:
- Backup data stores can begin to run out of space because the deletes, both manual and retention-based, are not completing and therefore space is not reclaimed on the backup target storage device.
- Backups may not be available for file recovery, Backup Data Connector exports, or replication, if post-processing has not completed for a VM.
How to prevent issues
You can proactively monitor the pending post-process and retention queue depths in one of two ways:
- Viewing the VBA Health Monitor tab in the Tools area of the PHD Console.
- Reviewing the email report that is sent after each job completes (you can enable email reports using the PHD Console's (Configuration area, Email tab). The queue depths are included as part of the Job Summary.
Note: A healthy PHD VBA contains queue depths that are typically no greater than the total number of virtual disks backed up in a single day. The Retention processes will vary based on your defined retention policy and how much your environment changes over time.
If you are experiencing issues related to consistently high queue depths, you can:
- Deploy additional PHD VBAs and load balance your backup jobs to allow all processes to complete.
- Allow more time between jobs on an individual PHD VBA to provide more time for all processing to complete.
- Disable backups temporarily to allow post-processing and retention to complete.
Example Alert Email
The PHD VBA is experiencing problems completing all backup processes in a timely manner. This is typically due to backup job overload. If you encounter this issue frequently, you can reduce the load by splitting larger backup jobs into smaller jobs or by deploying additional PHD VBAs.
Post Processing Queue: (failure type:linkd-neveridle): Post Processing Queue queue was not emptied in a 24-hour period