Appliance's /usr/bp /var or '/' partitions are full

ISSUE

Use an SSH client such as  PuTTY to access the Unitrends system at the command line level.  Note:  Ensure you have the OS password to access the Unitrends system’s command line.  The OS  password may differ from the password used to access the User Interface.

# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/md3         15G  2.1G   12G  16% /
tmpfs            16G   96K   16G   1% /dev/shm
/dev/md0        243M   63M  169M  27% /boot
/dev/md2         99G  2.8G   91G   3% /usr/bp
/dev/md1        788G   28G  721G   4% /var/opt/unitrends/database
/dev/sda         17T  1.8T   15T  11% /backups
If the % value before /usr/bp indicates 100%, the /usr/bp/ partition is full and needs to be cleaned.

RESOLUTION


1)  From the command line, change directories to /usr/bp/.
 

[[email protected] ~]# cd /usr/bp
[[email protected] bp]#

2)  Within  /usr/bp/, run the command "du -sh *".  This will list all files and directories and their sizes.
 
# du -sh *

3)  Determine which logs are using the most amount of space.
# cd logs.dir/
# ls -alh

4)  Before deleting check to find out why the logs are so large in size by inspecting the log level.  
Warning:  Skipping this step will result in the re-occurrence of this issue.

5)  Determine any process using an elevated log level.  For more information, including which logs should use an elevated log level by default that shold not be lowered, see Log Level Definitions.  
 
# /usr/bp/bin/check_logging
Checking /usr/bp/bpinit/master.ini ...
bpserver=5 # WARNING: DEBUG LOG LEVEL
updatedb=5 # WARNING: DEBUG LOG LEVEL
autosynth=5 # WARNING: DEBUG LOG LEVEL
===ERROR: Log parameters are out of bounds for /usr/bp
Number of Log sources=55
LogRetentionInstances=100
LogsPerInstance=200
MaxLogKB=10240
Max Log size = 11264000000 KB
/usr/bp size = 68849048 KB
===ERROR===

6)  Now, lower the log level from the UI. 

Navigate to Configure > <<Unitrends appliance>> >Edit > Advanced > General Configuration

Search the "Section" column for "debugging".
User-added image

7)  Delete the offending log.  

Warning:  Once removed, this data will not be available for retrieval.  If you are unsure of how to proceed, stop here and ask for assistance.
# cd /usr/bp/logs.dir/ 
# ls -alh
# rm <offending>.log

8) Check to confirm file is not held open and pending deletion
# lsof | grep -i delete | /var/log/messages
# rsyslogd   6347            root    6w      REG               8,23 4714463232      131147 /var/log/messages (deleted)
# in:imjour  6347  6353      root    6w      REG               8,23 4714463232      131147 /var/log/messages (deleted)
# rs:main    6347  6366      root    6w      REG               8,23 4714463232      131147 /var/log/messages (deleted)

9) Kill running PID of process holding logs open or reboot appliance to allow linux to clear the file system and reclaim space. 
 
# kill -9 6347
# lsof | grep -i delete | grep /var/log/messages
#



 

CAUSE

This issue is commonly caused by logging levels at an elevated level.

 

NOTES

For more information, see Log Level Definitions

Was this article helpful?
0 out of 0 found this helpful
Have more questions? Contact us