Server Is Compromised: What Do I Do?

In the vast majority of cases, servers tend to get broken into for the following reasons:

  1. Vulnerabilities in the software environment (firewalls and antiviruses misconfigured or software failing to update on time.)
  2. Mistakes in server configuration.
  3. Weak passwords.
  4. Human factors (giving access to a third party, leaked or stolen passwords, inside jobs.)

As a systems administrator, it's your job to keep tabs on your server. To address the first point, keeping up with antivirus definitions updates and making sure you have the latest service packs for your software, get creative with passwords and change them regularly. Do everything you can to address the above concerns before a breach happens.

But what do we do once we know we have a breach?

First course of action should be to see if your reserve copies are still accessible. You will need to figure out when the system was first compromised and if returning to a copy before the breach is possible, and restore the system. However, it's very possible that the reserve copy is too old or inaccessible. So let's explore your options in the case that a reserve copy is inaccessible. 

First, you need to figure out the source of the breach and what's been compromised as that will dictate how you'll be dealing with the problem. If the issue stems from a rogue script or a binary file, see if you can find an unfamiliar program in your list of running processes using the ps auwfx command. The process in question will usually be putting the system under heavier load than usual, and that would be a giveaway. Once found, we need to learn as much as we can about it and what it's doing with your system. The lsof -p PID command will be useful, as it will show you every file used by a process. To follow up, you can use the strace -p PID command to see exactly what the process is doing at any given moment.

You can also run iftop, which is a software that can be used to monitor server traffic in real time, to track processes that might be using your server as a source for DDoS attacks. If you manage to find these processes, you can use the tcpdump utility to see traffic and where it's headed, and you can check connected networks using lsof -i or netstat  -tulpan commands. Once you manage to track down the culprit processes and binaries, you should stop the infected services and kill the problem processes using kill -9 PID command.

Debian and Ubunty distros include a utility called debsums that lets you check MD5 hashes of previously installed packets and configuration files for malware. RedHat and associated distros have a rpm command with -qaV that does the same thing. 

Just as important is checking through your logs of recent access attempts to your server, through SSH, FTP, email., etc. Don't forget to check the files with bash command history, as intruders often forget to clean the log files. It good to pay special attention to /var/log: messages, secure, audit.log, yum.log, apt.log, lastlog, auth.log, and syslog.

The nature of the breach will dictate how you deal with it. For instance, if you found that the intrusion happened through SSH, then your first course of action would be to go into your /root/.ssh/authorized_keys file, change every password and release new keys. On the other hand if the problem is with your email server, the intruder might be using it to send spam. Emails marked as spam usually tend to get blacklisted and accumulate in the server's queue. You can check this queue to see if your server is sending spam using mailq or exim -bpc if you're running and Exim server, or postqueue -p if you're using Postfix.

Sometimes, the goal of the breach is not to carry anything out immediately but to delay tasks to avoid detection. So it's also important to check you cron scheduler to make sure there aren't any processes being run at undesignated times. It's equally important to check any files that were changed recently using the Find utility using find -mtime   command. 

It's a great idea to delete and uninstall any services that your server is no longer using, as dormant services can become points of vulnerability. Check all autorunning services too. You can do this using the following:


systemctl list-unit-files | grep enabled


service --status-all | grep +


initctl list | awk '{print $1}' | xargs initctl show-config

You firewall also needs to be configured to make sure that your server is only using the port you've designated. For instance, if you're running MySQL on just one local server, then there's no need to have port 3306 open as it just creates another window of vulnerability. 

Ultimately, there's no substitute for being prepared. But even after a breach, it still helps to look into security modules like SELinux, AppArmor, chroot, Docker, LXC, or systemd-nspawn. These modules are great because they can stop and intrusion from getting access to the rest of your server from getting through just a single process. Antivirus definition are also key, as mentioned before, and making sure you have something like maldet, clamav, or ai-bolit. If you aren't already, you should be running virus scans regularly; at least a few times a week if not daily along with the rest of your scheduled tasks. Finally, nikto is a great utility to use as it lets you scan through protocols like http/https and ports, proxy servers and SSL.

That basically concludes our article! If you have questions, check with us on Twitter and Facebook

-Till next time!

August 23 2017

Add or review comments

Please leave your comment

Existing comments

Comments 66

Antonio Medina
Dont forget rkhunter and chkrootkit. Great tools when used apropriately
SELinux should be enabled on pmuch all distro's by default. Enabling UFW and setting up iptables is one I see pmuch every admin nowadays forget.... Other useful tools: fail2ban (protect your outward services from bruteforce), DUO mobile (easily integrate two-factor login on basically ANYTHING.... duo is robust as hell). Also doesn't hurt to configure rsyslog for remote logging and setup smtp alerts, as well as private keys for login..... I could go on.....
buy viagra online <a href="">viagra online mastercard</a> where can i buy viagra in los angeles area [url=]purchasing viagra online[/url] ’
viagra 25mg no prescription <a href="">buying viagra online legal</a> can i order viagra online without prescription [url=]cheap brand viagra in usa[/url] ’
cheap canadian drugs <a href=" ">canadian pharmacy viagra</a> pharmacies
order viagra china <a href=" ">free viagra samples canada</a> i want to buy viagra online
us pharmacy no prior prescription <a href=" ">discount canadian drugs</a> online pharmacies
canadian pharmacy kingcanadian viagra <a href=" ">best canadian online pharmacies</a> tadalafil
mail order pharmacy <a href=" ">buy viagra for women germany</a> medication costs
canada drug <a href="">cialis daily</a> canada online pharmacies
cialis black 800 to buy in the uk one pill <a href=" ">cialis us pharmacy</a> cialis overnight delivery
buy brand viagra <a href=" ">canadian drugs real viagra</a> non prescription viagra las vegas nv
buying viagra without a prescription <a href=" ">dapoxetine with viagra</a> viagra gold
side effects of cialis and alcohol <a href=" ">cialis traitement continu</a> cialis tablete forum
buy generic viagra online <a href=" ">viagra auf rechnung bestellen </a> street value for viagra
cash loans kenya <a href=" ">combining payday loans</a> how to cash advance in rcbc credit card
same day cash loans philippines <a href=" ">online instant cash loans in south africa</a> payday loans maine
risque avec cialis <a href=" ">domande su cialis</a> cialis 10 mg einnahme
cialis c800 <a href=" ">buy generic cialis in canada </a> voucher cialis index
halifax payday loans <a href=" ">payday loans in 24 hrs</a> cash call loans california
ordering viagra from india <a href=" ">sildenafil price </a> where to buy viagra in south africa
womans viagra <a href=" ">ingredients in viagra</a> recreational viagra
shelf life of viagra <a href=" "> </a> viagra in stores
[url=]dissertation guidelines[/url] dissertation formatting <a href=" ">editing dissertations</a> umi dissertation
[url=]thesis express[/url] help with writing a thesis <a href=" ">thesis research proposal</a> psychology thesis
[url=]best essay cheap[/url] auto essay writer <a href=" ">do my essay for cheap</a> professional essay writer
[url=]thesis titles[/url] thesis express <a href=" ">psychology thesis topics</a> writing a phd thesis
[url=]writing a phd thesis[/url] thesis express <a href=" ">thesis proposal writing</a> help writing thesis statement
[url=][/url] thesis service <a href=" "></a> phd thesis search
Colospa Meclizine
cialis buy australia [url= ]viagra vs cialis[/url] stockists of cialis
is viagra otc pills like viagra over the counter
levitra dosage reviews buy levitra online from canada
cialis drugs cialis
find tadalafil [url= ]tadalafil daily[/url] amino tadalafil
blood pressure walmart pharmacy online
Tinidazole [url= ]Lincocin[/url] Hytrin
levitra instructions [url= ]levitra strength[/url] п»їlevitra
does viagra expire do you need a prescription for viagra
Norvasc ed meds
cialis nex day delivery uk how long does cialis last
Voveran [url= ]legal canadian prescription drugs online[/url] online pharmacy uk
cialis without prescribtion canada cialis online
shelf life of viagra real viagra for sale
viagra bestellen per rechnung [url= ]viagra dopaje[/url] viagra wit product
buy genereiccialis [url= ]buy cheap cialis[/url] super cialis professional
Zenegra [url= ]legitimate online pharmacies[/url] viagra from canada
fda approved canadian online pharmacies Trental
over the counter viagra substitute walgreens [url= ]viagra 25mg[/url] free sample of viagra
cialis 20 mg tablets [url= ]cialis generic[/url] cialis 20 mg canada
will viagra help with performance anxiety [url= ]venta de viagra en mendoza[/url] why viagra didn work
cialis cheap name brand cialis
prescription drug assistance [url= ]best online international pharmacies[/url] canadian pharmacy sildenafil
canadian mail order pharmacies [url= ]reputable online pharmacy uk[/url] Zyvox
viagra use viagra jelly
where to buy viagra in patong buying viagra in poland
cialis, generic cialis 20mg П„О№ОјО·
med pharmacy Atorlip-5
creighton university pharmacy online [url= ]botox online pharmacy[/url] buying prescription drugs from canada online
united healthcare online pharmacy [url= ]pharmacy store layout design[/url] best canadian pharmacy
registered canadian pharmacies [url= ]online pet pharmacy[/url] humana online pharmacy login
cocaine cialis [url= ]cost of cialis for daily use[/url] dapoxetine and generic cialis
how much is cialis at walmart [url= ]good looking loser cialis[/url] what company makes cialis
importing drugs from canada [url= ]aetna rx pharmacy[/url] 1st rx pharmacy statesville nc
canadian drugstore cialis [url= ]cialis cost walgreens[/url] cialis manufactured in canada
cvs pharmacy cvs store [url= ]blue mountain canadian pharmacy[/url] rx express pharmacy navarre fl