Code Red! SMTP brute force hack (coded by darker) on my code server

[rainless]

While going through his bash history one of my programmers noticed that we had been hit with an SMTP brute force attack.

I've changed every password on the system.

But basically they used the www-data password www-data to get into our system... which is really just WONDERFUL...

I'd like some advice on how I could rig it so that only my live server can access the code server. And also I'm curious as to how this happened in the first place.

We only just opened one port for ssh access from the live server to the code server (in order to update the site using git.)

Any advice would be appreciated.

 

[gangaskan]

While going through his bash history one of my programmers noticed that we had been hit with an SMTP brute force attack.

I've changed every password on the system.

But basically they used the www-data password www-data to get into our system... which is really just WONDERFUL...

I'd like some advice on how I could rig it so that only my live server can access the code server. And also I'm curious as to how this happened in the first place.

We only just opened one port for ssh access from the live server to the code server (in order to update the site using git.)

Any advice would be appreciated.

my guess is they used your live server as the bridge head, are the root / super user passwords the same on each box? do you have any holes in your live box code that could be used for exploits? another possibility is someone on your network got bugged and its being used as the entry point, and finally, it could be someone gaining physical entry to the building and re-conning, or perhaps using wifi in the parking lot.


how does your IP Tables look? you should only really open up just what you need on those boxes and ONLY those, regardless of development box / live. also change the SSH port from default 22 and make it random.


you will have to set a rule from the live box, saying only XXXXXX ip or XXXX mac (i think you can do mac based rules?) can SSH into that box, also, only have possibly one or two boxes be allowed to SSH into that machine. i would also do the same on the code box reversing the route (cant be too secure)


if need be, to make it more secure, only allow the test box to SSH and just use the test box as a piggy back. another thing to use is Fail2ban to prevent brute force attacks. fail2ban will disco and lock the account after x minutes, i think it can send emails too. it sounds like you just need some general lockdown is all



i would also go through your logs i know some people will mask theirs and remove traces, but novices do forget things, even experts too, in your case, you saw the bash history.

 

[dropadrop]

Few ideas on how you can make the box more secure, with varying difficulty, but also a recommendation that if somebody has accessed the box then you need to wipe it and make a clean install... Unless you are installing all security updates the day they are coming out, some local binary is bound to have had a vulnerability allowing gaining root access and you could easily have a backdoor or rootkit that's difficult to find.

- Run the application in chroot
- Mount /tmp and other locations writable by everyone with a noexec flag, so if somebody manages to download a binary with an applications privileges they can't execute it
- Limit outgoing connections with IP-tables so only the necessary are open, log any attempts that get blocked. One of the first things they'll do is connect to an IRC channel or start up a ftp server, you want to block that
- Limit access to any service that does not need to be accessible to all using iptables and tcp-wrappers, or the services own configuration
- Ensure you are on the mailinglist of any service open to the internet to know of vulnerabilities, this also includes any libraries or modules used by those services (often overlooked, but can be as serious as a vulnerability in the service that loaded the library / module)
- For SSH you could also consider using SSH-keys, especially if it's only used for a set connection between two computers, still I'd use tcp-wrappers too as it will also protect against an unknown vulnerability in the service.
- On our ssh-server I also block computers who have opened too many connections in the last minute, this has worked well for brute force attacks that have passed our (pretty relaxed) restrictions in tcp-wrappers, and I have not heard of one problem in the last 4 years.

http://www.ducea.com/2006/06/28/using-iptables-to-block-brute-force-attacks/

- Also SELinux is great for hardening a service, but quite difficult to really get running with a good custom set up.

 

[gangaskan]

Few ideas on how you can make the box more secure, with varying difficulty, but also a recommendation that if somebody has accessed the box then you need to wipe it and make a clean install... Unless you are installing all security updates the day they are coming out, some local binary is bound to have had a vulnerability allowing gaining root access and you could easily have a backdoor or rootkit that's difficult to find.

- Run the application in chroot
- Mount /tmp and other locations writable by everyone with a noexec flag, so if somebody manages to download a binary with an applications privileges they can't execute it
- Limit outgoing connections with IP-tables so only the necessary are open, log any attempts that get blocked. One of the first things they'll do is connect to an IRC channel or start up a ftp server, you want to block that
- Limit access to any service that does not need to be accessible to all using iptables and tcp-wrappers, or the services own configuration
- Ensure you are on the mailinglist of any service open to the internet to know of vulnerabilities, this also includes any libraries or modules used by those services (often overlooked, but can be as serious as a vulnerability in the service that loaded the library / module)
- For SSH you could also consider using SSH-keys, especially if it's only used for a set connection between two computers, still I'd use tcp-wrappers too as it will also protect against an unknown vulnerability in the service.
- On our ssh-server I also block computers who have opened too many connections in the last minute, this has worked well for brute force attacks that have passed our (pretty relaxed) restrictions in tcp-wrappers, and I have not heard of one problem in the last 4 years.

http://www.ducea.com/2006/06/28/using-iptables-to-block-brute-force-attacks/

- Also SELinux is great for hardening a service, but quite difficult to really get running with a good custom set up.

i thought about selinux too, but it might be a bit overwhelming.


another thing you can maybe do is set it up in a private vlan and isolate it.

 

[dark_15]

I'm not quite following the issue here - was someone using your server a spam relay, or were they able to send messages through a web form to other sites? Or did someone remotely login to your server via SSH?

 

[rainless]

I'm not quite following the issue here - was someone using your server a spam relay, or were they able to send messages through a web form to other sites? Or did someone remotely login to your server via SSH?

Someone remotely logged in through SSH.

Our logs show a few login attempts from a few different ip addresses... all different countries. So I'm assuming they were using an onion proxy.

As far as I know they tried to run this SMTP code and left because we don't even have an SMTP server... but that's just as far as I know.

 

[dark_15]

Gotcha - sounds like they wanted to use your system as a SPAM relay or potentially hosting malware. dropadrop really hit a lot of the things I'd recommend for an internet-facing server:

• SSH keys instead of passwords
• Disabling root logins
• Restricting in your ssh/iptables conf file what IP ranges are allowed to SSH into the system
• Restricting ssh logins for administrators only instead of service accounts
• If you cannot restrict which IP's can ssh into the system, then run fail2ban

 

[dropadrop]

Oh yeah, on our SSH gateway I also configured it so that you have to belong to a specific group to be able to su to root... So even if somebody had a weak password and it was used to gain access, it would be impossible to brute force themselves to gain root access unless they had the specific group membership. However it would then be more likely that they would find some installed binary that has a privilege escalation vulnerability and gain root access via that (though having a really minimal install might help a bit).

another thing you can maybe do is set it up in a private vlan and isolate it.

Well yes, if you can isolate it from the internet then great, but I understood the issue here is that one server is open to the internet, and that server needs access to a server which is otherwise isolated. In this case having the second server in an isolated vlan or behind another firewall won't really help (though of course it is an easy way to ensure only the intended service is available with a central place to configure it and thus good practice).

But here's the thing... Any service visible to the internet will have automated scans run against it. This has to be considered while deploying it, and it's really nothing to be worried about IF you have taken appropriate measures.

- Have a baseline configuration for ALL your servers with basic security improvements (out of the box configuration is not sufficient as you have things like SSH login for root enabled, SSH v1 allowed etc)
- If you have a lot of servers then make a central syslog database (even syslog-ng is sufficient unless you want to go with splunk or hadoop etc), this way an attacker can't hide his tracks as easily
- Ensure you know of any vulnerabilities in services visible to the internet, try to patch these as soon as possible (preferably the same day a vulnerability allowing remote access is announced)
- If possible also keep the rest of the system up to date, even if the vulnerabilities are not remotely exploitable (though I've never seen any place be very good with this...)
- Remember the weakest link is finally the user. If you have a service visible to the internet and somebody from the personal is using the same password in that service and anything else, then all an attacker needs to do is get the password from outside your organization and then use it to access your services. There are plenty of examples even in close history how this has resulted in pretty high profile attacks...
- Also remember that one key to limiting how successful an attack is, is how quickly you notice it happened. If somebody got root access on the server you would not be able to find it in the servers logs, at least if they knew what they where doing.

 

[su root]

But basically they used the www-data password www-data to get into our system... which is really just WONDERFUL...

Unless your web server was very, very badly installed, the www-data account cannot log in because it's UID is low, and the account has no password set. I suspect that there was no actual login to that account, or if so, then this account is not a legitimate account.

Typically, in situations where www-data is found to be doing things, the cause is that files you are hosting on your web server are vulnerable to injection of sorts (command injection, SMTP Header injection, etc). This usually results in remote users commanding your server to download a file to /tmp and execute it. -- However, in your case, you are seeing odd things in .bash_history, so this is not necessarily the case. (What user had an odd .bash_history? was it the www-data user?)

Try the command "last" -- if you see the www-data account listed there, you will be able to see the source of the login, however, if you do not see anything for the www-data user, that tells me that the user accessed the server via command injection. If your system is susceptible to command injection, your server could be made to do any user-level task as the www-data user.

I'd like some advice on how I could rig it so that only my live server can access the code server. And also I'm curious as to how this happened in the first place.

We only just opened one port for ssh access from the live server to the code server (in order to update the site using git.)

Any advice would be appreciated.

My advice at this point would be to first do a clean install or restore the server to an earlier, known-good backup, then audit your code for potential command injection issues.

If you are looking to have your production server connect to something else, it does not need to allow incoming SSH connections from the Internet.

As far as I know they tried to run this SMTP code and left because we don't even have an SMTP server... but that's just as far as I know.

Even if they have lost interest, it's possible that they have left backdoors for themselves, so it's important to do a clean install or restore to a known-good configuration, and try to find the source of the intrusion.

You may also consider using something like AppArmour or SELinux or apache mod_security to assist in preventing on injection attempts.