I wrote about this on my professional blog the other day but sometimes it makes sense to cross-post as some of my more technically-minded pals and colleagues read stuff here. Hopefully, this sort of thing will be helpful to someone at the right time when they need it the most. If your the intrepid type, not-so-technical but who is curious to learn something new, read on! It is an empowering experience to learn about this stuff. It’s not rocket science!
Get Rid of the Malware
NOTE: This guide presumes the hack took place on a Linux-based server. If you have been hacked on a Windows Server system, the damage is likely more widespread than we are going to have time and interest to get into in this short and sweet post. There are, however, some great guides on cleaning up extensive Windows Server attacks.
However the hacker(s) got into your server, once we discover the hack the first thing we want to do is get rid of the tool they most likely used to get in. We want to make sure it isn’t as easy for them to get in again.
To do this, we will go through every file and folder (sigh) associated with the compromised user account and get rid of everything that doesn’t belong there.
From the command line (CLI) do:
This will show all files within a directory, including hidden ones. Hackers often hide the files they use as tools using a
. in front to hide the file from less experienced users. It is far easier to detect this via the CLI than the desktop interface (GUI). Learning to get comfortable on the CLI may be worthwhile for you at this point if you wish to proceed! For an great into on using a command line environment like bash, there are some friendly tutorials out there to help you get started.
If you’re cool moving forward, the first thing we will look for is files with timestamps that show changes have been made recently or around the time the hack took place. Once we identify a file or files modified in the hack (such as a messed up home or index page), we can identify files used to modify the hacked file by searching the HTTP logs using those timestamps. Timestamps are key to helping us understand exactly when the hack took place.
Here’s one way to do this:
[yourservername]$ find /yourhomedirectory/yourpublicwebsitefolder/ ! -name "log" -mtime -3
The “-3” equals the number of days back to look for files modified within this time frame. This command lists all files under /yourhomedirectory/yourpublicwebsitefolder that have been modified in the past 3 days. We can certainly change this to 4 or 5 or 6 or 7 but be aware that this will greatly increase the amount of files and folders that show up in our search. Getting as close to the time as possible will make our work easier!
Website access logs (HTTP logs) are awesome for helping us isolate this information. They are typically located outside the public website directory, in a place like this:
Is the Database hacked?
SQL injection attacks are especially effective against vulnerable Joomla! sites. These attacks often result in altering databases with malicious code. Such modifications allow hackers back in even after identifying and removing affected files in the steps above. This is why it’s essential fter a hack to verify the database like we checked our files above to see if anything has changed that shouldn’t have.
Figuring out when the hack occurred is important for this reason. When we know when the hack occurred, we can even quickly revert the database back to a prior time using backups. That’s ideal.
Restoring Order to the Universe
If important files were modified or deleted in the hack, this can be much more involved. Most hosting providers provide tools we can use to restore from their dashboard panels. Check your hosting provider for specific details on available tools and steps.
Once we’ve restored working files, folders and domains, we can begin from the top of this guide to make sure everything is in order. There are no shortcuts so be prepared to do some work (ugh) but also be prepared to celebrate that we understand our systems better and therefore can then defend them better and respond faster next time something unplanned (and unwanted) happens.
Oh, no you don’t!
Hackers hack to gain and maintain access to systems for their use today and/or tomorrow based on their needs and intent. We can always count on at least one backdoors having been installed and hidden so that the hacker can bend the system to their will later on when needed. As long as these backdoors remain on our system, lying in wait, our site or service will remain vulnerable.
Where do we look? What can we do about it?
- rogue users – Check and make sure there are no unauthorized users. These could be web access, shell or database users. Verify all users and delete any that aren’t authorized or required. Keep in mind some hosting providers create users with weird usernames for backups, etc. Don’t delete one that does something important that you may not be aware of!
- authenticated cookies – Websites remember users by storing a small text file (cookie) on the user’s computer. Hackers often create and/or modify a valid user account and then force an authenticated cookie to their browser. That cookie can be reused until it expires even if the password for that user is changed. This is another reason why it’s important to modify HTTP headers to include expiration values for cookies. These cookies can be reset, forcing every user to log in again. There are many ways to do this, depending on your hosting provider and your web server’s configuration.
- installing remote shell – It’s trivial for a hacker to install a file that gives them complete access anytime they want. Files like these can be written in any object-oriented programming language, such as PHP or HTML. They are often given legitimate names to look like necessary files and sometimes contain obscured code to appear authentic to less-discriminating eyes. They can also be appended to legitimate files. The extra code enables functionality that’s only available when an extra parameter or two is passed to the file during the request that only the hacker who put it there knows about.
- changing permissions on stuff – This is the oldest trick in the book but still works quite well. Hackers will often set permissions on a publicly-addressable directory or folder so that they can read and write to it. This means they can copy files to the folder, delete files, whatever they want. Not awesome. We will want to be sure and do a permissions audit to make sure no other folders are still accessible to unauthorized users.
- only using SSH – Sure it’s true that many average users out there are still using FTP to transfer files back and forth. Truth is, someone should have told them long ago that, while this might seem easy to them and not worth changing, the level of vulnerability they are accepting in using FTP puts their entire system at risk. FTP sends usernames and passwords “in plain text” which means the connection strings (handshakes between computers) include this information and it’s really, really easy for others to obtain it. Using even SSH/SFTP is exponentially more secure. Most up-to-date FTP clients support these protocols and in most cases doing so doesn’t change the workflow in any way. Users won’t even know the difference, however, using a secure and encrypted method to move files back and forth is a solid way to help prevent any breaches going forward.
Wrap Up – CELEBRATE (then document!)
Obviously, there is no single-click, easy peasy tool or method for finding this nasty stuff, removing backdoors and cleaning up after a site has been hacked. It definitely takes time, experience and knowledge to investigate and isolate how hackers got in, what they added, deleted or changed and the what exactly occurred. There are many stories on the web of different types of hacks and the steps that others took to clean them up.
The single best thing we can do after a hack has been remediated is to document it in as much detail and accuracy as we are able. Share this internally so everyone has a chance to better understand what happened and what can be done to help prevent it from happening again. Add any important insights to your organization’s incident response plan.
Last but not least: CELEBRATE. This victory may seem small but it’s no small thing. While it sucks to get hacked, the one upside is how much stronger our cyber resiliency strategies grow from each one. Through exercise like these we not only grow our overall understanding about how our technology works and the hidden machinery that powers it, we also greatly improve our resiliency strategy and response methods. No easy way to do that but through experience.
I often think of this stuff like playing hide and seek. Hackers may find a great hiding spot once in a while but, once we discover it, they have to work really hard to find another one.
So go celebrate with your team. You’re all a whole lot smarter than you were before. Good luck out there!