No More Space on Hot Copy (Replication) Target and copy processes are backlogged

SUMMARY

Discuss solutions to “No More Space” issues when replication or vaulting issues are impacting D2D purging routines.

ISSUE

For more information on "Out of Space" conditions on a unit that processes backups, see Error: No more space on device.

This article discusses issues that may effect 

RESOLUTION

Ensure you are on the latest release.  Prior versions of the Unitrends product may include issues related to purging or purging performance that can lead to a target purging slower than anticipated.  Also ensure your target's physical disk is not less than 1.5X the size of data being replicated for best use, and that legal hod settings if used on a target are attainable.  Customers running CentOS 5 on a replication target may need to redeploy the target to Cent 6 to resolve space issues (this should have been done prior to Aug 2017 per communication to all customers).  

The specific resolution may vary by case, but in most cases, deletion of some older backups using the Recover Tab will relieve enough space temporarily to run new backups.  The best practice would be to delete backups that have already been archived to other media that are listed in the backup copy queue as pending backups but that are not listed as “last” backups in the backups report.  Please note that in most cases due to deduplication processes and compression the amount of space reclaimed by a deleting backup will be vastly smaller than the size of the backup to be deleted, the most efficient way to reclaim large amounts of space is to delete entire protected assets, not individual backups!  

Temporarily suspend replication from sources, and, ensure the target is also not performing backups.  If 24 hours after deleting backups space reclamation is not processing normally, and the appliance is on the latest available release and CentOS 6 operating system, contacting support for assistance may be required to resolve this situation.  

Once the immediate concern of obtaining space for new backups to run is met, resolving the backlog will be critical.  If the copying processes are running smoothly without failures this is likely just a backlog due to network overhead, outage, or large delta change in backups.  Monitor the replication/vaulting dashboard several times a day and ensure it is progressing and that the backlog is being reduced dramatically each day.  If the backlog is not reducing, reduce the level of throttling, increase available network bandwidth, or re-seed data to your target manually using the Rapid Seed process documented in the Technical Documents section of www.unitrends.com.

If replication (hot backup copy) or vaulting is failing, diagnose the issue immediately, and engage Unitrends support as necessary.  It may be necessary to temporarily disable vaulting/replication/hot backup copy so that new backups are not marked for syncing and are purgeable, and use manual archiving processes more aggressively in the interim until normal WAN processes can be resumed.

 

For those using Replication, you can also change the Queue Scheme in Configure > Appliances > Edit > Backup Copy.  By setting the scheme to Recency instead of maximize retention, and then restarting replication, only current backups will be queued, freeing older backups in the backlog. The nature of the backlog still needs to be resolved, but this can speed space availability for replication under 7.0.

CAUSE

Backups pending replication or vaulting are protected from automatic purging by devmonitor.  If the backlog is excessive, causing numerous master or other large backups to still be in pending transfer status when future master backup jobs are to be run for the same client, insufficient space conditions may occur on units with limited retention periods.

After large delta changes, deployment of new servers, extended network outages, bandwidth limiting factors, or failing vault/replication processes, it may be difficult for a system to catch up within a normal backup rotation once a backlog occurs before new master backups are scheduled to occur.

This can usually be corrected before backups begin failing by monitoring the vault/replication dashboard and reports periodically, but if this process has been unmonitored the first indication may be failing backups due to out of space conditions in the D2D device.

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Contact us