Root » FreeBSD » Retrieve e-mail

Retrieve e-mail

Howto retrieve mail from your provider off a pop3 server, scan to identify if the mail is spam or contains a virus and configure some filters.

Intruduction

E-mail is one of the largest and popular applications of the internet. The primairy support channel of FreeBSD exists of a large nummer of mailling lists. I found the support I got from other users very helpfull when I was new to this OS. On this page I'll talk about how to retrieve mail off a pop3 server and filter spam and virusses out.

At this point I like to point you to two related articles incase you where looking for those. The first all about howto read e-mail using mutt and the second about howto transfer e-mail using sendmail.

Summary

The following set of commands need to be execute as root. These will install getmail, SpamAssassin, ClamAV, procmail and archivemail. It also start the requered deamons afterwards.

Installing the software

cd /usr/ports/mail/getmail
make config && make install && make clean
cd /usr/ports/mail/p5-Mail-SpamAssassin/
make config && make install && make clean
cd /usr/ports/security/clamav
make config && make install && make clean
cd /usr/ports/mail/procmail
make config && make install && make clean
cd /usr/ports/mail/archivemail
make config && make install && make clean
rehash

Configuring anti-virus and anti-spam

echo 'clamav_clamd_enable="YES"' >> /etc/rc.conf
echo 'clamav_freshclam_enable="YES"' >> /etc/rc.conf
echo 'spamd_enable="YES"' >> /etc/rc.conf
echo 'report_safe 0' >> /usr/local/etc/mail/spamassassin/local.cf
clamd
spamd -c -d -r /var/run/spamd/spamd.pid

Configuring getmail and procmail

A number of commands need to be run for configuration purpeses as the user who will retrieve the mails.

mkdir -m 700 ~/.getmail/
mkdir -m 770 ~/Mail/
mkdir -m 770 ~/Mail/Lists/

cd ~/.getmail/
fetch http://alex.kruijff.org/files/freebsd/getmailrc
fetch http://alex.kruijff.org/files/freebsd/getmail.sh
chmod 600 getmailrc
chmod 700 getmail.sh

cd ~/
http://alex.kruijff.org/files/freebsd/.procmailrc

Schedulize

Now execute 'crontab -e as the user and add the following to retrieve mail every day and prune the mailinglists every day at 3 am and at boot.

@reboot ~/.getmail/getmail.sh archive
4 3 * * * ~/.getmail/getmail.sh archive
*/5 * * * * ~/.getmail/getmail.sh

Retrieve mail

There are two ways you can read your mail. The first option is to read it directly off the mail server. The second option is to retrieve your mail to your computer though protocols like pop3 and imap4. Getmail is a unix tool that does this for you. This tool also has the ability to leave the mails a number of days on the server. This is the reason I use getmail. With the following command you can install getmail as root.

cd /usr/ports/mail/getmail
make config && make install && make clean
rehash

Getmail requires a .getmail directory with a configuration called getmailrc. The configuration file will contain your username and password of you pop3 account. Therefor you need to make sure its only readable by you! This all can be done with the following command as the user who retrieves the mails.

mkdir -m 700 ~/.getmail/
touch ~/.getmail/getmailrc
chmod 600 ~/.getmail/getmailrc

The following section contains configuration for getmail to retrieve mail from your provider. Copy and past this with your editor in to the file ~/.getmail/getmailrc.

[retriever]
type = SimplePOP3Retriever
server = popmail.isp.example.net
username = account_name
password = my_mail_password

[destination]
type = Maildir
path = ~/Mail/maildir/

This will set getmail up to retrieve mail from you provider and put it in a large single inbox file capable of containing multiple mails. Getmail will not create the inbox for you, so you have to do this manualy before you can test this by running the command getmail as user.

mkdir -m 770 ~/Mail/
mkdir -m 770 ~/Mail/maildir/
mkdir -m 770 ~/Mail/maildir/cur
mkdir -m 770 ~/Mail/maildir/new
mkdir -m 770 ~/Mail/maildir/tmp
getmail

Getmail will by default retieve every mails over and over again so its time to tweak getmail operation a bit. Open the configuration file again. By adding the following tell getmail to be only print out warnings and errors, read only mails not previously retrieved and keep the mails at your provider for 7 days.

[options]
verbode = 0
read_all = FALSE
delivered_to = FALSE
received = FALSE
delete_after = 7

If you want getmail to delete every retrieved mail directly and retrieve only 100 mails per session you can add the next two lines.

delete = TRUE
max_messages_per_session = 50

Scan for spam

Spam is unsolicited mail send in mass quantities and is a common problem these days. SpamAssassin is a very accurate tool to identify mails that are spam. In all the years I've learned to trust this tool blind. The very few cases where it misidentified mail where caused because the mail server where used to send spam. The following commands will install this tool.

cd /usr/ports/mail/p5-Mail-SpamAssassin/
make config && make install && make clean
rehash

When SpamAssassin indentifies a mail as spam it, by default, creates a new mail containing a report and the orginal mail as an attachement. You can change this by setting report_safe to 0 or 2. The first will change the headers of orginal mail instead and the later will instruct SpamAssassin do this for all mails. Perform as root the following.

echo 'report_safe 0' >> /usr/local/etc/mail/spamassassin/local.cf

Next you need to make sure spamd is started during boot. The following commands will do that and need to be run as root.

echo 'spamd_enable="YES"' >> /etc/rc.conf
spamd -c -d -r /var/run/spamd/spamd.pid

Then you need to tell getmail to call spamassassin by modifing ~/.getmail/getmailrc and add the following to the bottom.

[filter-spam]
type = Filter_external
path = /usr/local/bin/spamc

A scan take at the very least 6 seconds. It might therefor be a good idea to use the max_messages_per_session variable that getmail offers to prevent retrieving the mails dubble due to any glitch.

Scan for viruses

Another problem you may face is recieving virusses though you mail. This is mostly a problem for Windows computer. Still you might retrieve mails that are read by Windows users. I've installed a virus scanner just to be on the safe side. Just so I don't forward virussus to my Windows friends.

cd /usr/ports/security/clamav
make config && make install && make clean
rehash

Again you need to make sure clamd is started during boot. The following commands will do that and need to be run as root.

echo 'clamav_clamd_enable="YES"' >> /etc/rc.conf
echo 'clamav_freshclam_enable="YES"' >> /etc/rc.conf
clamd

And you also need to tell getmail to call clamav again by modifing ~/.getmail/getmailrc and add the following to the bottom.

[filter-virus]
type = Filter_classifier
path = /usr/local/bin/clamdscan
arguments = ("--stdout", "--no-summary", "-")
exitcodes_drop = (1, 0)

At this point ClamAV will scan all mails for virusses that are retrieved with getmail. This includes the contence of zip, rar (version 2 not 3) and bz2 archives.

As an alternative you might use clamscan. This does have the ability to scan wile using extenal unpackers. This is beyond the scope of this tutorial, so I will not go in to that.

You can test if your clamav installation works by executing the following commands. There need to be executed as root:

cd /usr/ports/security/clamav/
make extract
clamscan -r work/test/
make clean

Filter mail

Now where going to extend the number mailboxes: one for each freebsd & sourceforge maillinglist, one for spam and optionaly one for virusses. For this purpose I use the program procmail. Install it as root.

cd /usr/ports/mail/procmail
make config && make install && make clean
rehash
mkdir ~/Mail/Lists/

Procmail determens where a mail should go by rule that are written down in ~/.procmailrc. Open the file and fill it with the following rules.

MAILDIR=$HOME/Mail
DEFAULT=$MAILDIR/inbox
VERBOSE=no

:0
* X-Spam-Flag: YES
inbox-spam

:0
* X-ClamAV:.*FOUND
inbox-virus

:0
* ^List-Post:[^:]+:[^@]+@(freebsd\.org|lists\.sourceforge\.net)
* ^List-Post:[^:]+:\/[^@]+
Lists/$MATCH

Now open the file ~/.getmail/getmailrc again and change the destination section in to the following.

[destination]
type = MDA_external
path = /usr/local/bin/procmail
unixfrom = true

Currently getmail deletes mails with virusses. If you like to retrieve these then you need to change 'exitcodes_drop = (1, 0)' with 'exitcodes_keep = (0, )' in getmailrc. In that case you also need to add the following section directly under the filter-virus section.

[filter-virus2]
type = Filter_external
path = /usr/local/bin/formail
arguments = ("-R", "X-getmail-filter-classifier:", "X-ClamAV:")

Schedulize

The retrieval is schedulized in this chapter. This is done by having crom call getmail. A problem accours when cron calls getmail but the previous getmail process is still running. This causes the mail to be retrieved multiple times. To prevent this I wrote a script.

In addation to that the mailinglist mailboxes are pruned to prevent them becomming very lare. I use the application archivemail to prune mails older than 30 days. Install this application as root.

cd /usr/ports/mail/archivemail
make config && make install && make clean
rehash

Now download my script as the user.

cd ~/.getmail/
fetch http://alex.kruijff.org/files/freebsd/getmail.sh
chmod 700 getmail.sh

And then execute 'crontab -e as the user and add the following to retrieve mail every day and prune the mailinglists every day at 3 am and at boot.

@reboot ~/.getmail/getmail.sh archive
4 3 * * * ~/.getmail/getmail.sh archive
*/5 * * * * ~/.getmail/getmail.sh

Final notes

I like to encourage you to contact me, if you have feedback about this howto. I can be contact my though the feedback link on top.

© Copyright 2000-2006
terms of use / algemene voorwaarden