Identify the Bad Process

CentOS, Hosting, Linux / December 26, 2021 / ashishpsk / 0

I have a process named “stealth” that has infected my server (slamming my CPU)

do a find /tmp | grep -i stealth, people with similar infections found the executables and hackers break-in tools in /tmp

ls -l /proc/pid/exe will tell you where the file is located. Removing it might be a whole other matter though.

examine the proc entry for the process id, look at /proc/[pid]/cwd this gives you the "current working directory" which will tell you where ./stealth is

To get a list of all current active connections, you can use the netstatcommand:

netstat -an

With the -a parameter, we tell the command to show all connections(including the LISTENING ones), instead of only those who are connected.

The -n parameter shows the different port numbers used

Now, to find out which user and which process is making connections on this port, we’ll use the lsof (List Open Files) command.

$ lsof -i tcp:80 -P -R

This command shows us all running processes that are using port 80 for any kind of communication.

• The -i parameter specifies we want to list the processes, by identifying them with IPv4 or IPv6.
• The tcp:80 part means we only want to show TCP connections (and ignore UDP for the time being) using port 80.
• Using -P we specify we want to see the port numbers (80, 21, …) instead of the names (HTTP, FTP) which are shown by default. Much like the -nparameter with netstat.
• With -R we also show the Parent Process ID, to see who initiated this process.

It will show output like this:

$ lsof -i tcp:80 -P -R
COMMAND   PID  PPID   USER   FD   TYPE    DEVICE SIZE NODE NAME
httpd   13538 16324 apache    4u  IPv6 548928212       TCP *:80 (LISTEN)
httpd   14279 16324 apache    4u  IPv6 548928212       TCP *:80 (LISTEN)
httpd   16324     1   root    4u  IPv6 548928212       TCP *:80 (LISTEN)
httpd   28042 16324 apache    4u  IPv6 548928212       TCP *:80 (LISTEN)
httpd   29772 16324 apache    4u  IPv6 548928212       TCP *:80 (LISTEN)
httpd   32424 16324 apache    4u  IPv6 548928212       TCP *:80 (LISTEN)

What can we learn from this? We see 6 instances of httpd, the HTTP daemon that is acting as the webserver. Five of these processes are run by the apache user, and one by root.

All apache httpd processes were started by a main process (PID 16324) which was executed by the root user (PPID column).

The column “command” displays the name of the process. Under “PID” we can see the Process ID, this can be used to easily stop, or kill the process. With “PPID” we can see the process ID that started this specific process.

Beneath “USER” we see the user that is executing the process. We can now see that all httpd processes that are being run as the apache user, were started by Parent Process ID 16324, the httpd process executed as root.

Note; you should be alerted when you see a postgres user running all sorts of programs on all sorts of exotic ports. If that’s the case, the server’s probably hacked, and used for means it wasn’t supposed to.

Now you have a list of all the processes that are using a specific port to communicate. Let’s find out which files these processes use, where they are located, and then stop the process – to prevent further harm. We won’t delete the files that were used just yet, because we might learn something afterwards when we examine them. We will, however, move them outside the user’s directory.

lsof -p 16324

With -p we can specify a Process ID (PID) to filter our results to. This’ll only list open files for the PID 16324.

$ lsof -p 16324
COMMAND   PID USER   FD   TYPE    DEVICE    SIZE       NODE NAME
httpd   16324 root  cwd    DIR      0,61    4096   14624004 /
httpd   16324 root  rtd    DIR      0,61    4096   14624004 /
httpd   16324 root  txt    REG      0,61  312372   19768930 /usr/sbin/httpd
httpd   16324 root  mem    REG      0,61   11704   20455174 /usr/lib/libgpg-error.so.0.3.0
httpd   16324 root  mem    REG      0,61  346148   20455190 /usr/lib/libgcrypt.so.11.2.2
httpd   16324 root  mem    REG      0,61  213116   20455786 /usr/lib/libxslt.so.1.1.17
httpd   16324 root  mem    REG      0,61   67932   20455784 /usr/lib/libexslt.so.0.8.13
httpd   16324 root  mem    REG      0,61   23172   20513414 /usr/lib/php/modules/xsl.so
httpd   16324 root  mem    REG      0,61   32216   20513412 /usr/lib/php/modules/xmlwriter.so</span>
....

You may have ended the program, but you still have to prevent if from being started again. First of all, check to see if the program was filed under /etc/init.d . Every command located in this directory will be executed upon boot – and you don’t want the process to restart if you have to reboot the server.

Next, make sure it’s not created a crontab entry to be executed every X-seconds.