Detect Suspicious Linux Processes

Datetime:2016-08-23 04:43:49          Topic: Python           Share

Ever bothered by suspicious processes running on your servers? No doubt how dangerous they might be: valuable data leaked, CPU/memory wasted, or DDoS attack other victims, etc. Want to learn how to easily capture those annoying troublemakers ? Even better, learn how to get alerted without additional human effort.

List all non-kernel processes. Usually, kernel processes are safe and clean. For kernel processes, either PID(process id) is 2 or PPID(parent process id) is 2. Here is how to get all non-kernel processes.

# rss(resident set size): real RAM usage
# -deselect: rule out matched processes
root@denny:~# ps --ppid 2 -p 2 -p 1 \
   --deselect -o uid,pid,rss,%cpu,command
UID   PID   RSS %CPU COMMAND
   0   411  1848  0.0 /lib/systemd/systemd-
   0   572  2904  0.0 dhclient -1 -v -pf /r
 102   902  1244  0.0 dbus-daemon --system
   0   912  1948  0.0 /lib/systemd/systemd-
   0  5869   388  0.0 upstart-socket-bridge
 200  1953   904  0.0 /usr/sbin/apache2 -k
 200  3463  3700  0.0 /usr/sbin/apache2 -k
  ...  ...
  ...  ...
   0  5098  4224  0.0 sshd: ubuntu [priv]
   0  5139  1748  0.0 /usr/bin/python /usr/
 200  5140  3484  0.0 /usr/bin/python /usr/
 200  5176  1904  0.0 sshd: ubuntu@pts/3
 200  5177  3860  0.0 -bash
 200  5193  1200  0.0 tmux attach -t denny
   0  5297  4224  0.0 sshd: ubuntu [priv]
  ...  ...
  ...  ...

Rule out trusted processes. We may have many processes running, which are expected and trusted. e.g apache2, tomcat7, mysqld, etc. To avoid distraction, build a white list especially for your project.

Sort processes by memory and CPU. We're more concerned about suspicious processes using noticeable resources.

# Sort by memory first, then cpu
ps --ppid 2 -p 2 -p 1 --deselect \
  -o uid,pid,rss,%cpu,command, \
  --sort -rss,-cpu

Automate Detection Process and Get Alerts. We hide all the complexities and whitelist configuration in a python script ( detect_suspicious_process.py ). If you issue the python command, you may see output like " Identified processes count: XXX ." Define a scheduled task to run periodical checks and confirm the number.

If the number is not 0 or it changes, send alerts. It might take a while to build a suitable white list. Once it's done, your servers are always more secured and managed!

wget -O /tmp/detect_suspicious_process.py \
https://raw.githubusercontent.com/\
DennyZhang/devops_public/tag_v2/python/\
detect_suspicious_process/\
detect_suspicious_process.py

# Detect suspicious process
python /tmp/detect_suspicious_process.py

# Detect by customized whitelist
python /tmp/detect_suspicious_process.py \
   --whitelist_file /tmp/whitelist.txt

Here is some more reading that you can do: Automate Insecure Ports Check By Nmap .





About List