Home

How to tell which processes are going to be killed first by the linux kernel's OOM killer

Wed, 2012.01.18 - 13:37 — müzso
If an OOM (Out-Of-Memory) event occurs (the system runs out of allocatable memory), the linux kernel invokes the oom-killer which uses an algorithm (as described in the kernel docs) to select some processes to be killed and thus free some memory.

The kernel constantly keeps track of a score for each process and if an OOM occurs, it kills the processes with the highest score. This score can be read from the /proc/PID/oom_score files. The following one-liner lists the processes with the highest scores:
for i in /proc/*/oom_score; do pid=$(echo "${i}" | cut -d/ -f3); echo "oom_score=$(cat "${i}"), PID=${pid}, exe=$(readlink -e /proc/${pid}/exe)"; done 2> /dev/null | sort -rn -t, -k 1.11 | head

Here're a few articles on OOM and the OOM-killer:
Here's a sample code to set oom_score to zero (by setting oom_adj to -17) for a few daemons/services so oom-killer will not consider them for a kill:
echo "-17" > /proc/1/oom_adj
pidfiles="/var/run/sshd.pid /var/run/apache2.pid /var/run/mysqld/mysqld.pid"
for pidfile in ${pidfiles}; do
  if [ -f "${pidfile}" ]; then
    pid="$(cat "${pidfile}")"
    [ -n "${pid}" ] && kill -0 "${pid}" > /dev/null 2>&1 && [ -f "/proc/${pid}/oom_adj" ] && echo "-17" > "/proc/${pid}/oom_adj"
  fi
done

Adjust the pidfile list to your liking and potentially add this script to a cronjob (so it's executed on a regular basis).

P.S.: it's quite interesting that on a server I've tested this with the init process was on the top of the list (by far ... it had a score of 3969661, while the next process to kill had a score of 1703158. Shock