Oracle Database causing Linux auditd logs to grow fast

Those days I was having some issues on some of my Oracle Linux 6 servers (VMs in OCI), with some Oracle Databases running still in project phase.

I could notice /var/log/audit folder size growing abnormally (around ~80MBs day)

[root@ocasxxxt00dbre1 audit]# ls -ltrh | tail -n 30
-r--------. 1 root root 6.1M Aug 15 10:24 audit.log.29
-r--------. 1 root root 6.1M Aug 15 14:08 audit.log.28
-r--------. 1 root root 6.1M Aug 15 18:00 audit.log.27
-r--------. 1 root root 6.1M Aug 15 21:48 audit.log.26
-r--------. 1 root root 6.1M Aug 16 01:25 audit.log.25
-r--------. 1 root root 6.1M Aug 16 05:20 audit.log.24
-r--------. 1 root root 6.1M Aug 16 09:15 audit.log.23
-r--------. 1 root root 6.1M Aug 16 11:42 audit.log.22
-r--------. 1 root root 6.1M Aug 16 12:59 audit.log.21
-r--------. 1 root root 6.1M Aug 16 14:15 audit.log.20
-r--------. 1 root root 6.1M Aug 16 15:32 audit.log.19
-r--------. 1 root root 6.0M Aug 16 16:49 audit.log.18
-r--------. 1 root root 6.1M Aug 16 18:03 audit.log.17
-r--------. 1 root root 6.1M Aug 16 19:18 audit.log.16
-r--------. 1 root root 6.1M Aug 16 20:34 audit.log.15
-r--------. 1 root root 6.1M Aug 16 21:51 audit.log.14
-r--------. 1 root root 6.1M Aug 16 23:06 audit.log.13
-r--------. 1 root root 6.1M Aug 17 00:17 audit.log.12
-r--------. 1 root root 6.1M Aug 17 01:32 audit.log.11
-r--------. 1 root root 6.1M Aug 17 02:48 audit.log.10
-r--------. 1 root root 6.1M Aug 17 04:03 audit.log.9
-r--------. 1 root root 6.1M Aug 17 05:16 audit.log.8
-r--------. 1 root root 6.1M Aug 17 06:32 audit.log.7
-r--------. 1 root root 6.1M Aug 17 07:46 audit.log.6
-r--------. 1 root root 6.1M Aug 17 09:03 audit.log.5
-r--------. 1 root root 6.1M Aug 17 10:18 audit.log.4
-r--------. 1 root root 6.1M Aug 17 11:35 audit.log.3
-r--------. 1 root root 6.1M Aug 17 12:53 audit.log.2
-r--------. 1 root root 6.1M Aug 17 14:10 audit.log.1
-rw-------. 1 root root 740K Aug 17 14:20 audit.log
[root@ocasxxxt00dbre1 audit]#

Auditd, responsible for those files, is a important service that keeps track of changes on your OS that could potentially be a security risk. Disabling it was not an option.

This server had multiple databases.. checking the content of those audit files, there were thousands and thousands of entries caused mostly by:

$ORACLE_HOME/bin/tnslsnr
$ORACLE_HOME/bin/oracle

80 MBs/day was not a huge problem. The filesystem had more than 30GBs free space so it could handle those files until some space alert triggered, someone deleted them or the retention cleaned them up.

The major issue I faced was when for some reason the archivelog area run out of space. The /var/log/audit started growing so fast, but so fast, that in only 5 hours my filesystem run out of space. This caused sshd to become inaccessible, boot to fail and I had to clean up my filesystem manually attaching it on a working machine.

The Bombflow was:

Archivelog Full -> Auditd out of control -> "/" out of space -> sshd unavailable -> server unable to boot.

After some investigation, I solved this issue by appending to /etc/audit/audit.rules the following lines:

# Oracle Binaries - Ignore
w never,exclude -F path=/u01/app/oracle/ -k exclude

Where /u01/app/oracle/ is my ORACLE_BASE.

Restarting the service via "service auditd restart" didn't work as the configuration was immutable. So reboot was required to change audit rules (this protection happens when there is a "-e 2" in audit.rules file).

References: