Set your GMail contacts to be whitelisted in Postfix

[RandomUsername]

N.b. Due to dependency issues, this will currently only work on a B3 but should work on B2 once the upgrade to Debian squeeze has been released.

This might be a bit of a niche use case but it's useful to me so may be to others.

I run my B3 as a mail server and use Google Mail to sync my contacts between Thunderbird, iPhone, iPad and roundcube web mail on the B3.

The default spam blocking on the B3 is a little oversensitive and I found that some emails from trusted contacts were being blocked* so I've written a script that will add all the email addresses in my Google Contacts to a whitelist for Postfix. I can then run this script once a week so it picks up any new contacts I've added.

First, you need to install the package python-gdata. You will need to enable the squeeze repositories if you haven't already by adding these lines to /etc/apt/sources.list:

Code: Select all

deb http://ftp.se.debian.org/debian squeeze main
deb http://ftp.se.debian.org/debian squeeze contrib
deb http://ftp.se.debian.org/debian squeeze non-free

then run:

Code: Select all

aptitude update
aptitude install python-gdata

Now you need to download the most recent deb package of GoogleCL from here: http://code.google.com/p/googlecl/downloads/list

If you're doing this directly from your B3 then do this:

Code: Select all

cd /home/admin/downloads
wget http://googlecl.googlecode.com/files/googlecl_0.9.13-1_all.deb

Then as root (type su and enter the root password):

Code: Select all

dpkg -i googlecl_0.9.13-1_all.deb

Assuming the packages all installed correctly you now need to authorise GoogleCL against your Google account.

Enter this command:

Code: Select all

google contacts list --title=

You will be prompted for your Google username and then given a URL. Browse to this URL and sign in. You will then be given a verification code to enter into the GoogleCL prompt.

Once you've entered your verification code your contacts should all be listed.

Run the command again and this time it should work straight away.

Assuming the above worked you now need to get the list of email addresses formatted in such as way that Postfix will accept it.

We can do this using sed to manipulate the output of the GoogleCL command.

Enter this command:

Code: Select all

google contacts list --title= --fields=email | \
grep -i @ | \
sed -e 's/;/\n/' | \
sed -e 's/;/\n/' | \
sed -e 's/home//' | \
sed -e 's/work//' | \
sed -e 's/other//' | \
sed 's/^[ \t]*//' | \
sed 's/[ \t]*$//' | \
sed 's/$/ OK/g'

This should give you a single column of email addresses with "OK" at the end of each. If you've added any custom labels to a contact's email address (i.e. not "home", "work" or "other")you will need to add an extra line for each one. Something like:

Code: Select all

sed -e 's/custom//' | \

after the line for the "other" label.

You now need to add this to a script.

In a text editor (.e.g. nano) - nano /path/to/script/googlewhitelist.sh - enter the following (remembering to include your custom lines if necessary):

Code: Select all

#!/bin/bash
google contacts list --title= --fields=email | \
grep -i @ | \
sed -e 's/;/\n/' | \
sed -e 's/;/\n/' | \
sed -e 's/home//' | \
sed -e 's/work//' | \
sed -e 's/other//' | \
sed 's/^[ \t]*//' | \
sed 's/[ \t]*$//' | \
sed 's/$/ OK/g' > /etc/postfix/sender_access
for EMAIL in `cat /home/admin/scripts/rejectedemails`
do
echo $EMAIL |  sed 's/$/ REJECT/g' >> /etc/postfix/sender_access
done;
for EMAIL in `cat /home/admin/scripts/acceptedemails`
do
echo $EMAIL |  sed 's/$/ OK/g' >> /etc/postfix/sender_access
done;
postmap hash:/etc/postfix/sender_access 
/etc/init.d/postfix restart

If you understand the code, you will see I've also added the ability to whitelist or blacklist domains or email addresses that aren't in Google contacts by listing them in files called "rejectedemails" or "acceptedemails".

You can add specific email addresses to these files or whole domains by just adding the domain part of the email address. E.g:

Code: Select all

dave@gmail.com
yahoo.com
hotmail.com

Finally, you need to tell Postfix to look at the database you'll be generating (sender_access.db). Edit the file /etc/postfix/main.cf. After the line that says " reject_unknown_recipient_domain" add this:

Code: Select all

 check_sender_access
        hash:/etc/postfix/sender_access

Save the file and exit.

If you run the script you've just written (sh googlewhitelist.sh) it should run through without any hiccups.

Assuming this all works, you can add it to cron.weekly, cron.monthly or create a more specific schedule using a crontab file.

Code: Select all

chmod +x googlewhitelist.sh
ln -s /home/admin/scripts/googlewhitelist.sh /etc/cron.weekly/googleswhitelist

*There is also a post in the forum about changing the DNS blacklists. I suggest everyone at least removes SORBS - http://forum.excito.net/viewtopic.php?f ... rbs#p14546

[Ubi]

Thanks, well done!
I'm testing this and found the following issues:

dpkg -i http://googlecl.googlecode.com/files/go ... -1_all.deb

I think should be

dpkg -i googlecl_0.9.13-1_all.deb

Second, I should not that I could not run the google command as root, only as normal user. Still, you must be root to add the addresses to sender_access

Third, maybe your sed script could be simplified by using:

Code: Select all

google contacts list --title= --fields=email | \
sed -e "s/[; ]/\n/g" -e "s/$/ OK/g"| grep "@"

also, I'd do a ">> /etc/postfix/sender_access" (double >> so not to destroy the original content)

One point: I don't think I fully understand the OK for sender access. Unless otherwise restricted, this allows anyone merely claiming to be a mail address (i.e. everyone!!) from your list to send mail via your server. In the way you've written your recipe, this could very well form an open relay. Could you please address this issue? In short, you must first put a line in main.cf saying that you reject all mail not meant for your local system, and only then put the whitelist

hope this helps

[RandomUsername]

Thanks, well done!
I'm testing this and found the following issues:

dpkg -i http://googlecl.googlecode.com/files/go ... -1_all.deb

I think should be

dpkg -i googlecl_0.9.13-1_all.deb

Yes, that's my clipboard playing tricks on me. I've fixed the post.

Second, I should not that I could not run the google command as root, only as normal user. Still, you must be root to add the addresses to sender_access

Strange. I did the whole thing as root with no issues. Are you using "su" or "su -"?

Third, maybe your sed script could be simplified by using:

Code: Select all

google contacts list --title= --fields=email | \
sed -e "s/[; ]/\n/g" -e "s/$/ OK/g"| grep "@"

I have absolutely no doubt it could be made more efficient. I came up with my recipe by doing multiple iterations and working out what needed to be removed on each run. I'm sure some of them are redundant.

also, I'd do a ">> /etc/postfix/sender_access" (double >> so not to destroy the original content)

But then sender_access will end up with multiple copies of your entire contact list and will eventually grow to ridiculous sizes, or am I missing something? If you're worried about preserving it you could put in a line that backs up the original file first I suppose.

One point: I don't think I fully understand the OK for sender access. Unless otherwise restricted, this allows anyone merely claiming to be a mail address (i.e. everyone!!) from your list to send mail via your server. In the way you've written your recipe, this could very well form an open relay. Could you please address this issue? In short, you must first put a line in main.cf saying that you reject all mail not meant for your local system, and only then put the whitelist

OK, I'm still trying to digest this. Are you saying you don't need to add the OK if you just want to whitelist incoming emails? That's isn't what my own research suggests but you're far more experienced with mail servers so I'll take your word for it.

[EDIT]Just ran my mail server through this - http://www.mxtoolbox.com/SuperTool.aspx - and it doesn't see it as an open relay. Which is nice.

[Ubi]

The point is that sender_address is supplied by whoever sent the mail, and thus cannot be trusted. Once postfix encounters a single OK in whatever check, no further checks are done and the mail is accepted for relay. Therefore, whitelisting based on sender address is extremely unsafe if there are no further safeguards.

One such safeguard would be reject_unauthorised_destination PRIOR to check_sender_address. This is common practice, but your recipe omitted the reject_unauthorised_destination and thus created a semi-open relay. The only benefit of your proposed form of whitelisting is that you may skip stringent spam-checks on the mail, at the risk that the sender address may be bogus and you get some spam.

The only reliable whitelisting is based on verifiable and reliable parameters, such as private keys or passwords. SPF comes close, but doesn't work in reality

[Ubi]

[EDIT]Just ran my mail server through this - http://www.mxtoolbox.com/SuperTool.aspx - and it doesn't see it as an open relay. Which is nice.

Yes, but did they try a sender address from your whitelist?
It's not uncommon for spammers to find multiple mail addresses on a blog post or website and then use these as spoofed FROM: addresses to fake reliabilty. In fact, if your own mail address is in the list, and you host your b3 on it's own domain, it's quite easy for a spammer to "guess" his way into your free mail service!

So, indeed you did not create a fully open relay (I thought i consistently mentioned semi-open in previous posts), but you did create a unnecessary security risk that is easily circumvented by proper placement of the order of checks in main.cf. It's just to avoid the chance that people implement your recipe poorly, then get bitten by a spammer and claim it's Excito's fault...

[RandomUsername]

OK, I see what you're saying (I think).

To address your concerns; reject_unauthorised_destination is already in main.cf - possibly added by Excito if it's not a default for Postfix.

Secondly, in my case we're talking about a few dozen email addresses in my contacts list. I've never received spam from any of those addresses, only legitimate email. However, I found that SORBS in particular has a habit of temporarily blocking all mail from specific mail relays of big mail providers like Yahoo and Hotmail. Therefore, legitimate email was getting blocked simply because it came via the same relay as some spam that ended up on SORBS' blacklist. I partially addressed this by removing SORBS from my main.cf, but to be sure I whitelisted the few email addresses I actually do want email from.

I can see why this approach might be more trouble than it's worth for bigger operations or people with large numbers of people in their GMail contact list.

[EDIT]Brain-dead addition also removed.

[Ubi]

Yes, I'm sure for you It'll work out great, but I thought the point was that this recipe was to be implemented by others as well. I regularly see spam claiming to be one of my domains trying to relay crap. Again, in the case when people put themselves (or info@mydomain.com) in the OK list and host the machine on "mydomain.com"I can pretty much guarantee you'll start seeing your network traffic going up on port 25 (in combination with misconfig in main.cf due to poor reading). This example may not be applicable to your setup, but maybe it is for others and I think they should be aware of the risk. All I'm suggesting is a line in the recipe to make users aware that the whitelist should be placed AFTER the relay checks. That's all.

[EDIT]
** braindead addition removed **

[RandomUsername]

Yes, I'm sure for you It'll work out great, but I thought the point was that this recipe was to be implemented by others as well. I regularly see spam claiming to be one of my domains trying to relay crap. Again, in the case when people put themselves (or info@mydomain.com) in the OK list and host the machine on "mydomain.com"I can pretty much guarantee you'll start seeing your network traffic going up on port 25. This example may not be applicable to your setup, but maybe it is for others and I think they should be aware of the risk.

Of course, there's nothing wrong with pointing out any potential security issues. I did say in the OP this might be a niche use case.

I think I must not be understanding something fundamental to your argument though.

You're saying, if I whitelist email from john@hotmail.com a spammer could spoof the sender in his spam to appear to be from john@hotmail.com and relay it through my server. But reject_unauth_destination is enabled so that threat is minimised. Am I correct?

In your latest post you're saying...actually, I'm not sure what you're saying. The case you mention seems to be you're not actually in control of the machine being hosted on your domain :S Why would you allow that?
But, to follow your line of argument: I add my own email address to the whitelist on my own server on my own domain. Then, anyone spoofing spam to be from my own email address could then get relay access to my server. But again, wouldn't reject_unauth_destination prevent that?

[Ubi]

You're saying, if I whitelist email from john@hotmail.com a spammer could spoof the sender in his spam to appear to be from john@hotmail.com and relay it through my server. But reject_unauth_destination is enabled so that threat is minimised. Am I correct?

Yes. It's not very difficult. Just be aware that many people do not understand that the sender_address is something that can be spoofed. For you and me this is obvious but for many it is not. Thus a warning label is in order.