Server Is Compromised: What Do I Do?

In the vast majority of cases, servers tend to get broken into for the following reasons:

  1. Vulnerabilities in the software environment (firewalls and antiviruses misconfigured or software failing to update on time.)
  2. Mistakes in server configuration.
  3. Weak passwords.
  4. Human factors (giving access to a third party, leaked or stolen passwords, inside jobs.)

As a systems administrator, it's your job to keep tabs on your server. To address the first point, keeping up with antivirus definitions updates and making sure you have the latest service packs for your software, get creative with passwords and change them regularly. Do everything you can to address the above concerns before a breach happens.

But what do we do once we know we have a breach?

First course of action should be to see if your reserve copies are still accessible. You will need to figure out when the system was first compromised and if returning to a copy before the breach is possible, and restore the system. However, it's very possible that the reserve copy is too old or inaccessible. So let's explore your options in the case that a reserve copy is inaccessible. 

First, you need to figure out the source of the breach and what's been compromised as that will dictate how you'll be dealing with the problem. If the issue stems from a rogue script or a binary file, see if you can find an unfamiliar program in your list of running processes using the ps auwfx command. The process in question will usually be putting the system under heavier load than usual, and that would be a giveaway. Once found, we need to learn as much as we can about it and what it's doing with your system. The lsof -p PID command will be useful, as it will show you every file used by a process. To follow up, you can use the strace -p PID command to see exactly what the process is doing at any given moment.

You can also run iftop, which is a software that can be used to monitor server traffic in real time, to track processes that might be using your server as a source for DDoS attacks. If you manage to find these processes, you can use the tcpdump utility to see traffic and where it's headed, and you can check connected networks using lsof -i or netstat  -tulpan commands. Once you manage to track down the culprit processes and binaries, you should stop the infected services and kill the problem processes using kill -9 PID command.

Debian and Ubunty distros include a utility called debsums that lets you check MD5 hashes of previously installed packets and configuration files for malware. RedHat and associated distros have a rpm command with -qaV that does the same thing. 

Just as important is checking through your logs of recent access attempts to your server, through SSH, FTP, email., etc. Don't forget to check the files with bash command history, as intruders often forget to clean the log files. It good to pay special attention to /var/log: messages, secure, audit.log, yum.log, apt.log, lastlog, auth.log, and syslog.

The nature of the breach will dictate how you deal with it. For instance, if you found that the intrusion happened through SSH, then your first course of action would be to go into your /root/.ssh/authorized_keys file, change every password and release new keys. On the other hand if the problem is with your email server, the intruder might be using it to send spam. Emails marked as spam usually tend to get blacklisted and accumulate in the server's queue. You can check this queue to see if your server is sending spam using mailq or exim -bpc if you're running and Exim server, or postqueue -p if you're using Postfix.

Sometimes, the goal of the breach is not to carry anything out immediately but to delay tasks to avoid detection. So it's also important to check you cron scheduler to make sure there aren't any processes being run at undesignated times. It's equally important to check any files that were changed recently using the Find utility using find -mtime   command. 

It's a great idea to delete and uninstall any services that your server is no longer using, as dormant services can become points of vulnerability. Check all autorunning services too. You can do this using the following:

systemd:

systemctl list-unit-files | grep enabled

systemv:

service --status-all | grep +

upstart:

initctl list | awk '{print $1}' | xargs initctl show-config

You firewall also needs to be configured to make sure that your server is only using the port you've designated. For instance, if you're running MySQL on just one local server, then there's no need to have port 3306 open as it just creates another window of vulnerability. 

Ultimately, there's no substitute for being prepared. But even after a breach, it still helps to look into security modules like SELinux, AppArmor, chroot, Docker, LXC, or systemd-nspawn. These modules are great because they can stop and intrusion from getting access to the rest of your server from getting through just a single process. Antivirus definition are also key, as mentioned before, and making sure you have something like maldet, clamav, or ai-bolit. If you aren't already, you should be running virus scans regularly; at least a few times a week if not daily along with the rest of your scheduled tasks. Finally, nikto is a great utility to use as it lets you scan through protocols like http/https and ports, proxy servers and SSL.

That basically concludes our article! If you have questions, check with us on Twitter and Facebook

-Till next time!
 

August 23 2017

Add or review comments

Please leave your comment

Existing comments

Comments 66


Antonio Medina
Dont forget rkhunter and chkrootkit. Great tools when used apropriately
nkallechy
SELinux should be enabled on pmuch all distro's by default. Enabling UFW and setting up iptables is one I see pmuch every admin nowadays forget.... Other useful tools: fail2ban (protect your outward services from bruteforce), DUO mobile (easily integrate two-factor login on basically ANYTHING.... duo is robust as hell). Also doesn't hurt to configure rsyslog for remote logging and setup smtp alerts, as well as private keys for login..... I could go on.....
Lbgsesse
buy viagra online <a href="http://genqpviag.com/#">viagra online mastercard</a> where can i buy viagra in los angeles area [url=http://genqpviag.com/]purchasing viagra online[/url] ’
Jdbxsesse
viagra 25mg no prescription <a href="http://llviabest.com/#">buying viagra online legal</a> can i order viagra online without prescription [url=http://llviabest.com/]cheap brand viagra in usa[/url] ’
Jbbvsesse
cheap canadian drugs <a href="http://canadianpharmnorx.com/ ">canadian pharmacy viagra</a> pharmacies
Kuikinend
order viagra china <a href="http://viagerkr.com/ ">free viagra samples canada</a> i want to buy viagra online
Ahkdsesse
us pharmacy no prior prescription <a href="http://storerxpharmcanada.com/ ">discount canadian drugs</a> online pharmacies
JbnbUript
canadian pharmacy kingcanadian viagra <a href="http://online21rxon.com/ ">best canadian online pharmacies</a> tadalafil
Lbsxsesse
mail order pharmacy <a href="https://ssviagriia.com/ ">buy viagra for women germany</a> medication costs
Kbcxhifs
canada drug <a href="http://xuypharmacyonline.com/">cialis daily</a> canada online pharmacies
AqcfHemyinend
cialis black 800 to buy in the uk one pill <a href="http://cialisvja.com/ ">cialis us pharmacy</a> cialis overnight delivery
FvfcExole
buy brand viagra <a href="http://hopeviagrin.com/ ">canadian drugs real viagra</a> non prescription viagra las vegas nv
NncsExole
buying viagra without a prescription <a href="http://lightvigra.com/ ">dapoxetine with viagra</a> viagra gold
Jbbnsesse
side effects of cialis and alcohol <a href="http://ljcialishe.com/ ">cialis traitement continu</a> cialis tablete forum
Kbbfinend
buy generic viagra online <a href="http://kloviagrli.com/ ">viagra auf rechnung bestellen </a> street value for viagra
Ahbzsesse
cash loans kenya <a href="http://daymoneygo.com/ ">combining payday loans</a> how to cash advance in rcbc credit card
JbnvUript
same day cash loans philippines <a href="http://samcash21.com/ ">online instant cash loans in south africa</a> payday loans maine
Labxsesse
risque avec cialis <a href="https://jecialisbn.com/ ">domande su cialis</a> cialis 10 mg einnahme
FbsgExole
cialis c800 <a href="http://llecialisjaw.com/ ">buy generic cialis in canada </a> voucher cialis index
Fqbbsesse
halifax payday loans <a href="http://ascashapply.com/ ">payday loans in 24 hrs</a> cash call loans california
NbnhExole
ordering viagra from india <a href="http://vigedon.com/ ">sildenafil price </a> where to buy viagra in south africa
Kndnhifs
womans viagra <a href="http://viagratx.com/ ">ingredients in viagra</a> recreational viagra
FjjuExole
shelf life of viagra <a href="https://viagraonlinejc.com/ ">http://viagraonlinejc.com/ </a> viagra in stores
Jvqqsesse
[url=https://dissertationahelp.com/]dissertation guidelines[/url] dissertation formatting <a href="http://dissertationahelp.com/ ">editing dissertations</a> umi dissertation
Abgcsesse
[url=https://thesiswritingtob.com/]thesis express[/url] help with writing a thesis <a href="http://thesiswritingtob.com/ ">thesis research proposal</a> psychology thesis
Lmoppsesse
[url=http://essaywriteris.com/]best essay cheap[/url] auto essay writer <a href="http://essaywriteris.com/ ">do my essay for cheap</a> professional essay writer
Abgcsesse
[url=https://thesiswritingtob.com/]thesis titles[/url] thesis express <a href="http://thesiswritingtob.com/ ">psychology thesis topics</a> writing a phd thesis
Abgcsesse
[url=https://thesiswritingtob.com/]writing a phd thesis[/url] thesis express <a href="http://thesiswritingtob.com/ ">thesis proposal writing</a> help writing thesis statement
Abgcsesse
[url=https://thesiswritingtob.com/]thesiswritingtob.com[/url] thesis service <a href="http://thesiswritingtob.com/ ">thesiswritingtob.com</a> phd thesis search
Abdgsesse
Colospa https://canadianeve21.com/ Meclizine
RfvbExole
cialis buy australia [url=https://cileve.com/ ]viagra vs cialis[/url] stockists of cialis
Bbdfhifs
is viagra otc https://gensitecil.com/ pills like viagra over the counter
Lrbssesse
levitra dosage reviews https://uslevitraanna.com/ buy levitra online from canada
LhdvExole
cialis drugs https://asciled.com/ cialis
GvdbUript
find tadalafil [url=http://boxtadafil.com/ ]tadalafil daily[/url] amino tadalafil
Abdgsesse
blood pressure https://canadianeve21.com/ walmart pharmacy online
Jbsdsesse
Tinidazole [url=http://onlinecanda21.com/ ]Lincocin[/url] Hytrin
Lrbssesse
levitra instructions [url=https://uslevitraanna.com/ ]levitra strength[/url] п»їlevitra
Bbshhifs
does viagra expire http://loxviagra.com/ do you need a prescription for viagra
Jebgsesse
Norvasc http://pharmacylo.com/ ed meds
RebfExole
cialis nex day delivery uk https://rcialisgl.com/ how long does cialis last
Lebnsesse
Voveran [url=https://xlnpharmacy.com/ ]legal canadian prescription drugs online[/url] online pharmacy uk
LbsoExole
cialis without prescribtion http://ucialisdas.com/ canada cialis online
GtnbUript
shelf life of viagra http://jokviagra.com/ real viagra for sale
Anoosesse
viagra bestellen per rechnung [url=http://llviagra.com/ ]viagra dopaje[/url] viagra wit product
RebfExole
buy genereiccialis [url=https://rcialisgl.com/ ]buy cheap cialis[/url] super cialis professional
Jebgsesse
Zenegra [url=http://pharmacylo.com/ ]legitimate online pharmacies[/url] viagra from canada
Lebnsesse
fda approved canadian online pharmacies https://xlnpharmacy.com/ Trental
GtnbUript
over the counter viagra substitute walgreens [url=http://jokviagra.com/ ]viagra 25mg[/url] free sample of viagra
Fmrfsesse
cialis 20 mg tablets [url=https://cialisee.com/ ]cialis generic[/url] cialis 20 mg canada
Anoosesse
will viagra help with performance anxiety [url=https://llviagra.com/ ]venta de viagra en mendoza[/url] why viagra didn work
RebfExole
cialis http://rcialisgl.com/ cheap name brand cialis
Jebgsesse
prescription drug assistance [url=http://pharmacylo.com/ ]best online international pharmacies[/url] canadian pharmacy sildenafil
Lebnsesse
canadian mail order pharmacies [url=http://xlnpharmacy.com/ ]reputable online pharmacy uk[/url] Zyvox
Bbshhifs
viagra use https://loxviagra.com/ viagra jelly
Anoosesse
where to buy viagra in patong http://llviagra.com/ buying viagra in poland
RebfExole
cialis, generic http://rcialisgl.com/ cialis 20mg П„О№ОјО·
Jebgsesse
med pharmacy https://pharmacylo.com/ Atorlip-5
sesseAni
creighton university pharmacy online [url=https://pharmacyken.com/ ]botox online pharmacy[/url] buying prescription drugs from canada online
sesseGtv
united healthcare online pharmacy [url=https://pharmacyhrn.com/ ]pharmacy store layout design[/url] best canadian pharmacy
sesseDev
registered canadian pharmacies [url=https://cjepharmacy.com/ ]online pet pharmacy[/url] humana online pharmacy login
Exoleloh
cocaine cialis [url=https://rcialisgl.com/ ]cost of cialis for daily use[/url] dapoxetine and generic cialis
ExoleVed
how much is cialis at walmart [url=https://krocialis.com/ ]good looking loser cialis[/url] what company makes cialis
sesseAni
importing drugs from canada [url=https://pharmacyken.com/ ]aetna rx pharmacy[/url] 1st rx pharmacy statesville nc
sesseHtf
canadian drugstore cialis [url=https://cialishav.com/ ]cialis cost walgreens[/url] cialis manufactured in canada
sesseAni
cvs pharmacy cvs store [url=https://pharmacyken.com/ ]blue mountain canadian pharmacy[/url] rx express pharmacy navarre fl