Dealing With A Compromised Server

First things first, there are no "quick fixes" other than restoring your system from a backup taken prior to the intrusion, and this has at least two problems.

1. It's difficult to pinpoint when the intrusion happened.

2. It doesn't help you close the "hole" that allowed them to break in last time, nor deal with the consequences of any "data theft" that may also have taken place.

This question keeps being asked repeatedly by the victims of hackers breaking into their web server. The answers very rarely change, but people keep asking the question. I'm not sure why. Perhaps people just don't like the answers they've seen when searching for help, or they can't find someone they trust to give them advice. Or perhaps people read an answer to this question and focus too much on the 5% of why their case is special and different from the answers they can find online and miss the 95% of the question and answer where their case is near enough the same as the one they read online.

That brings me to the first important nugget of information. I really do appreciate that you are special and unique. I appreciate that your website is too, as it's a reflection of you and your business or at the very least, your hard work on behalf of an employer. But to someone on the outside looking in, whether a computer security person looking at the problem to try and help you or even the attacker himself, it is very likely that your problem will be at least 95% identical to every other case they've ever looked at.

Don't take the attack personally, and don't take the recommendations that follow here or that you get from other people personally. If you are reading this after just becoming the victim of a website hack then I really am sorry, and I really hope you can find something helpful here, but this is not the time to let your ego get in the way of what you need to do.

Don't Panic

Do not panic. Absolutely do not act in haste, and absolutely do not try and pretend things never happened and not act at all.

First: understand that the disaster has already happened. This is not the time for denial; it is the time to accept what has happened, to be realistic about it, and to take steps to manage the consequences of the impact.

Some of these steps are going to hurt, and (unless your website holds a copy of my details) I really don't care if you ignore all or some of these steps, that's up to you. But following them properly will make things better in the end. The medicine might taste awful but sometimes you have to overlook that if you really want the cure to work.

Stop the problem from becoming worse than it already is:

1. The first thing you should do is disconnect the affected systems from the Internet. Whatever other problems you have, leaving the system connected to the web will only allow the attack to continue. I mean this quite literally; get someone to physically visit the server and unplug network cables if that is what it takes, but disconnect the victim from its muggers before you try to do anything else.

2. Change all your passwords for all accounts on all computers that are on the same network as the compromised systems. No really. All accounts. All computers. Yes, you're right, this might be overkill; on the other hand, it might not. You don't know either way, do you?

3. Check your other systems. Pay special attention to other Internet facing services, and to those that hold financial or other commercially sensitive data.

4. If the system holds anyone's personal data, immediately inform the person responsible for data protection (if that's not you) and URGE a full disclosure. I know this one is tough. I know this one is going to hurt. I know that many businesses want to sweep this kind of problem under the carpet but the business is going to have to deal with it - and needs to do so with an eye on any and all relevant privacy laws.

However annoyed your customers might be to have you tell them about a problem, they'll be far more annoyed if you don't tell them, and they only find out for themselves after someone charges $8,000 worth of goods using the credit card details they stole from your site.

Remember what I said previously? The bad thing has already happened. The only question now is how well you deal with it.

Understanding the problem:

1. Do NOT put the affected systems back online until this stage is fully complete, unless you want to be the person whose post was the tipping point for me actually deciding to write this article. I'm not going to link to that post so that people can get a cheap laugh, but the real tragedy is when people fail to learn from their mistakes.

2. Examine the 'attacked' systems to understand how the attacks succeeded in compromising your security. Make every effort to find out where the attacks "came from", so that you understand what problems you have and need to address to make your system safe in the future.

3. Examine the 'attacked' systems again, this time to understand where the attacks went, so that you understand what systems were compromised in the attack. Ensure you follow up any pointers that suggest compromised systems could become a springboard to attack your systems further.

4. Ensure the "gateways" used in any and all attacks are fully understood, so that you may begin to close them properly. (e.g. if your systems were compromised by a SQL injection attack, then not only do you need to close the particular flawed line of code that they broke in by, you would want to audit all of your code to see if the same type of mistake was made elsewhere).

5. Understand that attacks might succeed because of more than one flaw. Often, attacks succeed not through finding one major bug in a system but by stringing together several issues (sometimes minor and trivial by themselves) to compromise a system. For example, using SQL injection attacks to send commands to a database server, discovering the website/application you're attacking is running in the context of an administrative user and using the rights of that account as a stepping-stone to compromise other parts of a system. Or as hackers like to call it: "another day in the office taking advantage of common mistakes people make".

Can't I just repair the exploit or rootkit you've detected and put the system back online?

In situations like this the problem is that you don't have control of that system any more. It's not your computer any more.

The only way to be certain that you've got control of the system is to rebuild the system. While there's a lot of value in finding and fixing the exploit used to break into the system, you can't be sure about what else has been done to the system once the intruders gained control (indeed, its not unheard of for hackers that recruit systems into a botnet to patch the exploits they used themselves, to safeguard "their" new computer from other hackers, as well as installing their rootkit).

Make a recovery plan and stick to it:

Nobody wants to be offline for longer than they have to be. That's a given. If this website is a revenue generating mechanism then the pressure to bring it back online quickly will be intense. Even if the only thing at stake is your / your company's reputation, this is still going generate a lot of pressure to put things back up quickly.

However, don't give in to the temptation to go back online too quickly. Instead move with as fast as possible to understand what caused the problem and to solve it before you go back online or else you will almost certainly fall victim to an intrusion once again, and remember, "to get hacked once can be classed as misfortune; to get hacked again straight afterward looks like carelessness" (with apologies to Oscar Wilde).

1. I'm assuming you've understood all the issues that led to the successful intrusion in the first place before you even start this section. I don't want to overstate the case but if you haven't done that first then you really do need to. Sorry.

2. Never pay blackmail / protection money. This is the sign of an easy mark and you don't want that phrase ever used to describe you.

3. Don't be tempted to put the same server(s) back online without a full rebuild. It should be far quicker to build a new box or "nuke the server from orbit and do a clean install" on the old hardware than it would be to audit every single corner of the old system to make sure it is clean before putting it back online again. If you disagree with that then you probably don't know what it really means to ensure a system is fully cleaned, or your website deployment procedures are an unholy mess. You presumably have backups and test deployments of your site that you can just use to build the live site, and if you don't then being hacked is not your biggest problem.

4. Be very careful about re-using data that was "live" on the system at the time of the hack. I won't say "never ever do it" because you'll just ignore me, but frankly I think you do need to consider the consequences of keeping data around when you know you cannot guarantee its integrity. Ideally, you should restore this from a backup made prior to the intrusion. If you cannot or will not do that, you should be very careful with that data because it's tainted. You should especially be aware of the consequences to others if this data belongs to customers or site visitors rather than directly to you.

5. Monitor the system(s) carefully. You should resolve to do this as an ongoing process in the future (more below) but you take extra pains to be vigilant during the period immediately following your site coming back online. The intruders will almost certainly be back, and if you can spot them trying to break in again you will certainly be able to see quickly if you really have closed all the holes they used before plus any they made for themselves, and you might gather useful information you can pass on to your local law enforcement.

Reacting to a problem is always harder than working to prevent it, so I think I will split this article into two and cover some preventative measures in the next one. For now, if you have any comments, feel free to post them below or find us on Twitter and Facebook!

- Until next time!


January 13 2021

Add or review comments

Please leave your comment

Existing comments

Comments 71

viagra usa <a href="">viagra price in pakistan</a> byy viagra online [url=]where to buy viagra online[/url] ’
viagra 25 mg online <a href="">genaric viagra</a> generic viagra 100 [url=]how much does viagra cost with a prescription[/url] ’
drugs from canada with prescription <a href=" ">canadian drug stores online</a> discount prescription drug
pharmacy online store <a href=" ">buy viagra no prescription by cipro</a> cialis from canada
viagra super force reviews <a href=" ">viagra austrailia buy</a> where can i get viagra cheap
Viagra Soft Tabs <a href=" ">canada pharmacies</a> canadian pharmacy
no prior prescription required pharmacy <a href=" ">prescription online</a> pharmacie canadienne
pharmacies near me <a href="">drugs without prescription</a> medication costs
viagra without prescription in australia <a href=" ">free viagra samples boynton beach</a> viagra online overnight shipping
sex pills cialis <a href=" ">cialis, 20 mg, canadian</a> cialis south africa
navarro pharmacy miami <a href=" ">canadian pharmacy viagra brand</a> cheap drugs online
installment payday loans columbus ohio <a href=" ">payday loans direct lender uk no credit check</a> cash loans west auckland
cuanto vale el medicamento cialis <a href=" ">crestor cialis interactions</a> viagra kombination cialis
viagra 100 <a href=" ">viagra online</a> pfizer viagra 100mg price
ez money loan pflugerville tx <a href=" ">payday loan post falls idaho</a> cash advance chase checking account
cialis generico prospecto <a href=" ">cialis tutti giorni</a> cialis jelly review
cialis cheap over night <a href=" ">mastercard cialis </a> original cialis online
bradenton payday loans <a href=" ">loan money new delhi delhi</a> native american payday loan lenders
payday loan on 99th western <a href=" ">what's a payday loan</a> cash advance wisconsin dells
cheapest viagra prices <a href=" "> </a> viagra lowers blood pressure
cialis 20mg П„О№ОјО· <a href=" "> </a> cheep cialis
[url=]thesis binding service[/url] phd thesis writing help <a href=" ">phd thesis paper</a> what is thesis writing
[url=]best essay websites[/url] college essay writers <a href=" ">essay writers online cheap</a> buy essay papers cheap
[url=]uk dissertation help[/url] uk dissertation <a href=" ">dissertations help</a> help writing dissertation
[url=]what are the best essay writing services[/url] services essay <a href=" ">medical school essay writing service</a> best essay services
[url=]revatio vs viagra[/url] [url=]viagra without presc[/url] [url=]cialis for bph[/url] [url=]cialis 2.5 mg[/url] [url=]buying cheap cialis online[/url]
[url=]nursing thesis[/url] thesis data analysis <a href=" ">help in writing thesis</a> thesis writing services uk
[url=]thesis online[/url] proposal for thesis <a href=" ">buy a thesis</a> thesis help online
[url=][/url] thesis proposal writing service <a href=" "></a> thesis help free
cialis a domicilio new jersey [url= ]free cialis in canada[/url] no rx cialis
list of legitimate canadian pharmacies [url= ]best canadian online pharmacy[/url] Endep
levitra purchase walmart pharmacy price levitra
Reforge pain meds online without doctor prescription
viagra vs cialis [url= ]viagra alternatives[/url] viagra triangle
Periactin prescription drug
cialis 10mg ireland bye cialis online from usa
buy cialis online in australia cialis buy info/
levitra vs viagra vs cialis reviews compare price viagra cialis levitra
canadian pharmacy uk delivery [url= ]Prozac[/url] Flagyl
eli lilly tadalafil [url= ]combitic global tadalafil[/url] combitic global caplet tadalafil
cialis 20mg price where to buy cialis online
erectile dysfunction drug best rogue online pharmacy
buy generic cialis online cheap generic cialis
generic viagra from canada female viagra
viagra kД±sД±rlД±k yaparmД± how long until viagra wears off
cialis on line sell [url= ]cialis 5mg australia[/url] cialis pills overnight
cialis med dapoxetine cheap cialis uk
where can i buy cialis online really really cheap cialis
Medex cross border pharmacy canada
drugs from canada [url= ]best australian online pharmacy[/url] canadian pharmacy discount coupon
how well does viagra work viagra canada free sample
ce il viagra per le donne [url= ]viagra za i przeciw[/url] consecuencias negativas del viagra
order cialis india [url= ]can i take cialis with daxpoteine[/url] cialis pills canada
cialis dapoxetine pictures of penis on cialis
canadian online pharmacy reviews buy prescription drugs online
Flexeril online pharmacy meds
discount cialis [url= ]levitra or cialis which is better[/url] herbal cialis
buy cialis online in austalia generic cialis price
viagra wiki [url= ]viagra ingredients[/url] itsoktocry viagra
no prior prescription required pharmacy navarro pharmacy miami
Pamelor rx pharmacy coupons
what does cialis do for a woman [url= ]cialis without ed[/url] how can i get cialis
canadian pharmacy non prescription drugs [url= ]cialis from india online pharmacy[/url] can i be denied employment due to prescription drugs
best drugstore concealer [url= ]vipps certified online pharmacy list[/url] canada pharmacies/account
cialis [url= ]original cialis product dapoxetine[/url] cialis paypal accepted
silkroad online pharmacy review [url= ]online pharmacy degree programs[/url] buy prescription drugs online without
generic cipla cialis [url= ]do you need to take cialis everyday[/url] cialis shop online
cialis dosage frequency [url= ]cialis 20mg[/url] canada pharmacy generic cialis
buy viagra from canadian pharmacy [url= ]recommended canadian online pharmacies[/url] canadian pharmacy legitimate
cialis pills overnight [url= ]cialis tabs[/url] cost of daily cialis
cialis 80mg [url= ]blak cialis[/url] cialis and food