RMAN operations consume additional process memory when opening a session to an Oracle Database instance. The system user's project must have a high enough shared memory allowance or the operation will fail with:
To verify and increase the user's share memory run the following:
As the system user running the backup/restore operations:
id -p # Note the users numerical project id prctl -n project.max-shm-memory -i project # This will display the current shared memory allowance for the project. The PRIVILEGE VALUE that is set is 'privileged'
prctl -n project.max-shm-memory -r -v -i project # This will set the shared memory allowance for the running environment. G,M suffixes may be used (4.5G). projmod -p -K 'project.max-shm-memory=(priveleged,,deny)' # This will set the share memory allowance to survive a restart. G,M suffixes may be used (4.5G).
Oracle Database 11g Release 2 on Oracle Solaris Release Notes - Section 4.1.1
For more detailed information on backup failures and performance issues see Unitrends KB 5062 - Backup Failures and Performance Issues