Postfix mail filter

From BcWiki
Jump to: navigation, search

This is my log of setting up Postfix as a mail filter (not hosting any email). We will do: recipient verification, virus/spam filtering, RBL checks, greylisting, and numerous other measures to protect email.

Contents

Linux setup

Installed CentOS 4.4, chose basic components, including devel libraries, devel tools, web server, and mail server. Ran yum update. I've used Red Hat, then Fedora, for years. I switched to CentOS for this build. I like it. My only complaint so far is that ncftp is gone. Odd.

Postfix

  • installed RPMS to satisfy dependencies for building and installing Postfix:

pcre-devel

  • installed Simon Mudd's Postfix SRPM (v2.3.7-1) available at his site - http://postfix.wl0.org/en/ (also available via mirrors listed on his site)
  • edited /usr/src/redhat/SOURCES/make-postfix.spec, added these lines after comments at top of file: 
/usr/src/redhat/SOURCES/make-postfix.spec:
...
POSTFIX_LDAP=0
POSTFIX_PCRE=1
...
  • changed dir to /usr/src/redhat/SOURCES/ and ran sh make-postfix.spec
  • changed dir to /usr/src/redhat/SPECS/ and ran rpmbuild –bb postfix.spec
  • installed the Postfix rpm and ran system-switch-mail to change the mailer to Postfix; if 'system-switch-mail' is not available to you, install said package from yum
  • noted that if I install Postfix during Fedora setup (and thus upgrade the RPM after I build it) the postfix user gets uid 89, but if I hold off and install Postfix after setup it gets user 100; not a big deal, but note the Postfix uid for later. Also note the gid for Postfix and Postdrop. For my setup here, Postfix has uid/gid 100/102 and Postdrop has gid 101. This really only matters for a backend setup, but it's worth noting.

/etc/postfix/main.cf

  • I like to shorten Postfix's primary configuration file by removing comments and some of the empty lines like this: 
cd /etc/postfix
cp main.cf main.cf.comments
egrep -v '^#|^$' main.cf.comments > main.cf
  • I then added this line to main.cf: 
/etc/postfix/main.cf:
 ...
myhostname = pftest.example.com

Relaying: setting up domains, recipients, and transports

This particular mail filter is the primary MX for several domains. In order to reduce backscatter I wanted to do recipient verification for each domain. This requires that I obtain a valid recipient list from each backend mail server whose email I protect. As of this writing, each of those servers runs MySQL, so it is rather straightforward to generate a list of valid email addresses. These will eventually make up relay_recipient_maps. Since I know the domains for which we will receive email, I can maintain that list manually. But I can also obtain those lists from these backend servers. Those domains will comprise relay_domains. Finally, I will use this information to create transport tables which tell this filter which email goes to which backend server.

Recipient Maps

For each customer (not invididual email addresses, but each domain or group of domains, which also typically correspond to a particular backend server) I maintain a separate list of recipients. Thus, in main.cf we have: 

/etc/postfix/main.cf:
 ...
relay_recipient_maps = hash:/etc/postfix/foo-relay-recipients
       hash:/etc/postfix/bar-relay-recipients
...

And so on.

Relay Domains

Relay domains are done similarly. 

/etc/postfix/main.cf:
 ...
relay_domains = hash:/etc/postfix/foo-relay-domains
       hash:/etc/postfix/bar-relay-domains
...
Transport Maps

And finally, we do transport maps. 

/etc/postfix/main.cf:
 ...
transport_maps = hash:/etc/postfix/foo-transport
       hash:/etc/postfix/bar-transport
...

Getting the relay recipient information

On the backend server

On the backend server I occasionally dump out a list of valid recipients. The script to do this is straightforward. 

 #!/bin/bash
 
 mysql -u postfix -ppassword postfix -e 'select address from alias where active='1'' | egrep '@'  > /home/usersync/relay-recipients.txt
 mysql -u postfix -ppassword postfix -e 'select domain from domain where active='1'' | egrep '\.' > /home/usersync/relay-domains.txt

Yes, there are more bulletproof ways to do this.

On the filter

To get the data from the remote servers, I chose SSH. You could also use rsync, which I've done on other filters that I maintain. Below is a sample script. This script should be set to run as a non-privileged user. And you should attach to the backend server as a non-privileged user as well. Make sure you set up public key authentication from the filter to the backend server.

As I said, rsync works very well for this, too, and is probably simpler. Not only so, but you could wind up using make, which would be efficient since postmap would not be run against files that did not change. A thread on the Postfix list regarding syncing the whole /etc/postfix folder between systems has some good discussion on using make. The start of that thread is at http://archives.neohapsis.com/archives/postfix/2007-02/0979.html. 

 #!/bin/bash 

# process domains, create recipient domain map and transport map
/usr/bin/scp backend.example.net:/home/usersync/relay-domains.txt /home/usersync/foo-domains.txt
errcode=$?
if [ $errcode -eq 0 ]
then
  sort -u /home/usersync/foo-domains.txt > /home/usersync/foo-domains.tmp
  sed -e "s/$/ anything/g" /home/usersync/foo-domains.tmp > /home/usersync/foo-relay-domains
  /bin/cp /home/usersync/foo-relay-domains /etc/postfix/foo-relay-domains
  /bin/cp /etc/postfix/transport /home/usersync/foo-transport.bak
  cat /dev/null > /etc/postfix/foo-transport
  for domain in `awk '{print $1}' /etc/postfix/foo-relay-domains`
  do
        echo $domain smtp:[backend.example.net] >> /etc/postfix/foo-transport
  done
fi
# end of domain processing

# process email addresses, create recipient map
/usr/bin/scp backend.example.net:/home/usersync/relay-recipients.txt /home/usersync/foo-userlist.tmp
errcode=$?
if [ $errcode -eq 0 ]
then
  sort -u /home/usersync/foo-userlist.tmp > /home/usersync/foo-userlist.tmp2
  sed -e "/^ /d" /home/usersync/foo-userlist.tmp2 > /home/usersync/foo-userlist.tmp3
  egrep -vi ' .+@' /home/usersync/foo-userlist.tmp3 > /home/usersync/foo-userlist.txt
  sed -e "s/$/ OK/g" /home/usersync/foo-userlist.txt > /etc/postfix/foo-relay-recipients
fi
# end of email address processing 

# create the db files
cd /etc/postfix
/usr/sbin/postmap foo-relay-domains
/usr/sbin/postmap foo-transport
/usr/sbin/postmap foo-relay-recipients
# all done

Postfix wrap-up

  • edited /etc/postfix/aliases and set root/postmaster mail to go to postman@example.com. Don't forget to run newaliases
  • there are other options in /etc/postfix/main.cf that will help in spam prevention. Since this is a mail filter, it is the first line of defense for incoming email. We should turn away as much mail as possible here so we create less work for the backend server and so the user does not see as much undesirable email. 
/etc/postfix/main.cf:
 ...
smtpd_recipient_restrictions = reject_non_fqdn_sender,
 reject_non_fqdn_recipient,
 reject_unknown_sender_domain,
 reject_unknown_recipient_domain,
 permit_mynetworks,
 check_helo_access hash:/etc/postfix/helo_access,
 reject_unauth_pipelining,
 reject_invalid_hostname,
 reject_unauth_destination,
 check_recipient_access hash:/etc/postfix/recipient_checks,
 reject_rbl_client zen.spamhaus.org,
 reject_rbl_client list.dsbl.org
...

I use recipient checks to pass all email to abuse & postmaster addresses with no filtering. 

/etc/postfix/recipient_checks:
postmaster@     OK
abuse@          OK
postman@        OK

I use helo checks to prevent servers from using my hostname(s) and IP(s) in their HELO/EHLO. Assume my mail server's name is example.com and also hosts example.net. Also assume the server's IP is 192.168.1.78. The reject message is configurable and really doesn't matter. It's unlikely anyone who cares will ever see it. 

/etc/postfix/helo_access:
example.net   REJECT No.
example.com	REJECT No.
192.168.1.78  REJECT No.
localhost       REJECT No.
127.0.0.1       REJECT No.

SQLgrey

One of the simpler things to install is sqlgrey. I documented below both a fresh installation and a move from an existing server.

fresh install

'note: need editing from here down'

  • The rpm takes care of adding a system user
  • Created new mysql user, sqlgrey (in MySQL < 5, you do this by granting permissions; in MySQL 5 you can create the user with create user sqlgrey)
  • created the database and assigned permissions: 
 CREATE DATABASE sqlgrey;
 GRANT ALL ON sqlgrey.* TO sqlgrey@localhost;
  • added these 2 lines to sqlgrey's config 
/etc/sqlgrey/sqlgrey.conf:
...
db_type = mysql
db_name = sqlgrey
...
  • made sure sqlgrey was set to run at boot time
  • added a line to call sqlgrey 
/etc/postfix/main.cf:
...
smtpd_recipient_restrictions =
 reject_non_fqdn_sender,
 reject_non_fqdn_recipient,
 reject_unknown_sender_domain,
 reject_unknown_recipient_domain,
 permit_mynetworks,
 check_helo_access hash:/etc/postfix/helo_access,
 reject_unauth_destination,
 check_policy_service inet:127.0.0.1:2501
 reject_unauth_pipelining,
 reject_invalid_hostname,
 reject_rbl_client sbl-xbl.spamhaus.org,
 reject_rbl_client list.dsbl.org
...

I added it to smtpd_recipient_restrictions. Where you place it in there depends on what you want. For example, you may want to run it before RBL checks, to save DNS queries. Or you may want to put it after them, to save CPU. It's your call.

  • started sqlgrey and checked /var/log/maillog for errors (if it gripes about clients_ip_whitelist.local and clients_fqdn_whitelist.local not existing in /etc/sqlgrey/, just create them)
  • I had a problem where sqlgrey would randomly just quit working, so I created sqlgreynanny.sh to run as a cron job keep an eye on it: 

/usr/local/sbin/sqlgreynanny.sh:
 #!/bin/bash
 
/sbin/service sqlgrey status > /tmp/sqlgreystat.txt
 
sqlgreystat=`awk '{print $5}' /tmp/sqlgreystat.txt`
case "$sqlgreystat" in
   running...)
        exit 0
        ;;
   *)
        /sbin/service sqlgrey restart
        exit 0
        ;;
esac

importing from an existing SQLgrey database

  • On the old server, I exported the database like so:

mysqldump sqlgrey > sqlgrey_db.sql

  • Then, I copied sqlgrey_db.sql to the new server and imported it like this:

mysql sqlgrey < sqlgrey_db.sql

ClamAV+Amavisd-new+SpamAssassin

Both clamav and amavisd-new are available via rpmforge. You can add the repo for the RHEL 4 version to your /etc/yum.repos.d. It's available from Dag Wieers' site. http://dag.wieers.com/rpm/FAQ.php#B2

The packages I ended up with are: amavisd-new clamav clamav-db clamd

/etc/amavisd.conf

I set local_domains_maps to reflect the domains for which we protect email. 

@local_domains_maps = (
    read_hash("/etc/postfix/foo_relay_domains"),
    read_hash("/etc/postfix/bar_relay_domains"),
    [".$mydomain"] );

I also set it to let spam through since I filter it in my email client. And I adjusted the minimum score for what SpamAssassin will call spam. 

$final_spam_destiny = D_PASS;
$sa_tag2_level_deflt = 5.0;

In the @av_scanners section I disabled every virus scanner except ClamAV.

Also, the Fedora maintainers did a great job making sure that Amavis and ClamAV work well together. But since I'm not using Fedora for this install I had to do a bit of tweaking. By default Amavis refers to Clamav by a socket named /var/run/clamav/clamd. Per the instructions in /etc/amavisd.conf, you want to match that name to the socket name LocalSocket in clamd.conf. In /etc/clamd.conf, this directive is commented out by default. So I un-commented it and then changed amavisd.conf to use /var/run/clamav/clamd.sock.

/etc/clamd.conf

I set two directives in this file. 

User amavis
LocalSocket /var/run/clamav/clamd.sock

file and folder ownership

Before starting either service I made amavis the owner of /var/log/clamav & /var/run/clamav. Then I did: 

 chkconfig clamd on
 chkconfig amavisd on

And started both services. One other note. Because we are running clamav as amavis, I changed /etc/logrotate.d/freshclam and /etc/logrotate.d/clamav so that they make amavis the owner of /var/log/clamav/freshclam.log after a log rotation, and set the group write bit.

Calling amavisd from Postfix

I added a line to call amavis: 

/etc/postfix/main.cf:
 ...
content_filter = amavis:[127.0.0.1]:10024
 ...

And I modified /etc/postfix/master.cf for amavis: 

/etc/postfix/master.cf:
 ...
amavis unix - - n - 2 smtp
        -o smtp_data_done_timeout=1200
        -o smtp_send_xforward_command=yes
 
127.0.0.1:10025 inet n - n - - smtpd
        -o content_filter=
        -o local_recipient_maps=
        -o relay_recipient_maps=
        -o smtpd_restriction_classes=
        -o smtpd_client_restrictions=
        -o smtpd_helo_restrictions=
        -o smtpd_sender_restrictions=
        -o smtpd_recipient_restrictions=permit_mynetworks,reject
        -o mynetworks=127.0.0.0/8
        -o strict_rfc821_envelopes=yes
        -o receive_override_options=no_unknown_recipient_checks,no_header_body_checks
        -o smtpd_bind_address=127.0.0.1
 ...

Then I reloaded Postfix (if you don't reload Postfix, it won't know about the amavis transport). Also set those services to run at boot time.

FuzzyOCR in SpamAssassin

FuzzyOCR is good for blocking image spam. Sure, spammers find ways to get around it, but FuzzyOCR does a good job. More here: http://wiki.apache.org/spamassassin/FuzzyOcrPlugin Packages you will need are: 

gocr
netpbm
netpbm-progs
libungif
libungif-progs
perl-String-Approx
ImageMagick
ImageMagick-perl

If you have the rpmforge repo, you will be able to yum install all of these. Some are available as part of CentOS, others as part of rpmforge.

  • I got the latest FuzzyOCR plugin from http://http://fuzzyocr.own-hero.net/wiki/Downloads and unzipped it to /etc/mail/spamassassin (it made its own subfolder there)
  • copied FuzzyOcr.pm & FuzzyOcr.cf to /etc/mail/spamassassin
  • did the same for FuzzyOcr.words.sample, as FuzzyOcr.words
  • edited FuzzyOcr.cf, changed the log location to /var/log/fuzzyocr.log
  • then created /var/log/fuzzyocr.log and made amavis the owner of it

-these 2 lines will activate the plugin 

/etc/mail/spamassassin/v310.pre:
...
loadplugin FuzzyOcr /etc/mail/spamassassin/FuzzyOcr.pm
loadplugin Mail::SpamAssassin::Timeout
...

(the 2nd I added after reading a Google thread about timeout issues)

Mailgraph

  • grabbed the tarball from http://mailgraph.schweikert.ch/
  • unzipped it in /usr/src (made its own subdir)
  • got the supporting cast (all from rpmforge):
    • rrdtool
    • perl-rrdtool
    • perl-File-Tail
  • README said it needs perl-Time-HiRes as well (and prior to FC5 it did). On my CentOS 4.4 system it was already installed.
  • copied mailgraph-init to /etc/rc.d/init.d/mailgraph, made it executable, and modified it:
    • changed MAILGRAPH_PL to point to /usr/local/sbin/mailgraph.pl (where I put all my local scripts)
    • changed MAIL_LOG to /var/log/maillog (for obvious reasons)
    • changed RRD_DIR to /var/log since that's where I want to keep the rrd files
    • added >--rbl-is-spam --ignore-localhost after the daemon-pid line so that RBL blocks count as spam and so the numbers don't get artificially inflated by localhost transactions. The line that starts mailgraph now looks like this: 
/etc/rc.d/init.d/mailgraph:
...
        nice -19 $MAILGRAPH_PL -l $MAIL_LOG -d \
                --daemon-pid=$PID_FILE --daemon-rrd=$RRD_DIR \
                --rbl-is-spam --ignore-localhost 
...
  • added mailgraph to chkconfig (chkconfig --add mailgraph)
  • copied mailgraph.pl to /usr/local/sbin/ and edited it
    • I changed my $rrd & my $rrd_virus to the explicit path /var/log/mailgraph.rrd and /var/log/mailgraph_virus.rrd respectively)
  • copied mailgraph.cgi to /var/www/html, made it executable, and modified it
    • again made my $rrd & my $rrd_virus explicit paths, matching what is in mailgraph.pl
  • that should be all the changes needed to the files
  • ran mailgraph manually the first time - this will create the rrd files (this should be a single line): 
 mailgraph.pl -v -c  -l /var/log/maillog --daemon-pid=/var/run/mailgraph.pid --daemon-rrd=/var/log
  • started mailgraph

Retrieved from "http://wiki.rbcollins.net/index.php?title=Postfix_mail_filter&oldid=1744"