If only there was a way for a mail server to detect when a Windows XP box is the source of email.

Well, of course there is. Michael Zalewski’s passive fingerprinting tool p0f can detect various operating systems with reasonable accuracy. Unlike active tools like NMap that spray a host with packets to determine open ports and what OS it’s running, passive fingerprinters just sniff network traffic as it goes by. Tell p0f to listen to TCP port 25 on your inbound mail server and you’ll get a real-time guess of which OS is being used on systems as they connect to you.

A few months ago I set up a p0f daemon to collect OS information on a client’s mail servers, then matched that with email identified as spam with SpamAssassin. It turned out that around 70% of spam that made it though the blacklists and other spam defences was being blasted from Windows XP home computers. I was going to try writing a SpamAssassin plugin to use that info, but fortunately someone beat me to it.

The SpamAssassin P0f plugin created by Mingchun (Vincent) Li uses p0f to add OS detection and scoring to SpamAssassin . It’s easy to set up and has SpamAssassin rules for increasing the score of mail originating from Windows boxes, and also decreasing the score for mail from Linux and Unix.

The plugin has three components:

1. A perl daemon that runs p0F and sends OS guesses out a UDP port on localhost
2. A SpamAssassin plugin that gathers the guess and increases or decreases the SpamAssassin score of the corresponding message.
3. A configuration file containing system parameters, rules and scores.

Installing on Ubuntu or Debian

As far as I know, there is no packaged version of this plugin for Ubuntu or Debian. There is a packaged version of p0f in Ubuntu Feisty and older releases, but it seems to be version 2.0.5. You can use that but for greatest accuracy you might want to install p0f from source.

Before compiling p0f, you’ll need libpcap and libpcap-dev packages installed. After that compiling p0f is as simple extracting the tarball to a temporary directory then doing the usual “make; make install”. The p0f binary will be placed in directory /usr/local/sbin.

To install the SpamAssassin p0f plugin, follow the instructions in the INSTALL file. For no particularly good reason I used the UDP method rather than the Unix socket method:

1. Download file p0f-analyzer.pl to directory /usr/local/bin or /usr/local/sbin and make it executable.
2. Download files p0f-analyzer.pm and p0f-analyzer.cf to directory /etc/mail/spamassassin and make sure they are readable.
3. Start p0f-analyzer.pl running with “/path/to/script/p0f-analyzer.pl 2345″
4. Restart spamd. You might want to start spamd in debug mode with the “-D” parameter just to make sure the plugin loaded and is working.

If you want to use this plugin permanently, you’ll need to create an init script to run p0f-analyzer.pl at system startup.

Scoring

By default the plugin increases the SpamAssassin score of messages originating from a Windows XP box by 1.0. Other Windows versions increase the score by 0.1, and unrecognized OSs increase the score by 0.8. Mail from Unix systems ( *BSD, Solaris, HP/UX and Tru64) decrease the SpamAssassin score by 1.0.

There is no score for Linux servers in the default scoring, so on our installation we modified file p0f-analyser.cf to add Linux to the regexp for Unix systems:

header L_P0F_Unix  X-P0f-OS-Fingerprint =~ /^((Free|Open|Net)BSD)|Linux|Solaris|HP-UX|Tru64/

Security

One additional change we made was to force p0f drop root privileges and chroot itself. Protocol analyzers (e.g. Ethereal / Wireshark) have a history of buffer overflows that have allowed attackers to send malformed packets to gain root on the host system. I haven’t seen any published vulnerabilities for p0f, but it’s always a good idea to run all network software with limited privileges.

The file p0f-analyzer.pl also trusts the system PATH to find the p0f binary. That’s not so bad since the script does the right thing by running in Perl taint mode and explicitly setting the PATH, but it always makes me less nervous to use full paths in scripts run by root.

To change file p0f-analyzer.pl to use a full path and run p0f as an unprivileged user, change the statement around line 104 to read:

open($p0f, "/usr/sbin/p0f -u username -l 'tcp dst port 25' 2>&1 | ") or die "Can't fork: $!";

…where “username” is an unprivileged user ID on your system, ideally one dedicated for use by p0f that has no login shell and a locked password.

Accuracy

OS fingerprinting is never completely accurate, but p0f seems to do well identifying most Windows XP systems. NAT routers and certain firewalls can make some systems unidentifiable, however. Also, spam originating from a Windows bot but sent through an open SMTP relay would of course take on the TCP/IP characteristics of the relay, but bot herders don’t usually go to that trouble.

A really determined spammer could change the characteristics of the TCP stack similar to what IP Personality does in Linux… but it’s highly unlikely any bot herder would bother. Of course if enough mail servers and firewalls start using OS detection to thwart bots, they may start doing that someday.

We’ve only just started using the SpamAssassin p0f plugin on one of our mailers as a test so it’s too soon to say how effective this approach really is. The idea is extremely cool though… hats off to the author for developing the plugin. I’ll keep an eye on our spam stats and report back when there’s enough real-world data.

Related posts:

• Blocking image spam with FuzzyOCR
• Spam more profitable than extortion?
• A simple tool to track and control spammers

It turns out that if you prepend unencrypted text to a GPG-signed (or signed and encrypted) file, when the file is decrypted by GPG the prepended text is spit out immediately followed by the decrypted plaintext. There is with no visual indication where one block of text ends and the other begins. An attacker who can modify a GPG-encrypted file (such as an email message or file on disk) can exploit this behavior to turn a signed message like

Purchase 40 shares of Acme Widgets

into a message that reads

Please sell all my shares and deposit the proceeds into account 123456 of Offshore Criminals Savings and Loan. Later this week I will then
Purchase 40 shares of Acme Widgets

This flaw is most likely to be a problem with email clients that use GnuPG, such as Mozilla Thunderbird with the Enigmail extension. The problem isn’t in the cryptography… it’s that by default GPG displays no separation between extraneous output and decrypted plaintext. Read the full announcement for complete details.

Yet another example of how the strongest cryptography algorithms in the world can be defeated by unanticipated real world implementation flaws.

Related posts:

• Little known features: Symmetric encryption with PGP/GPG
• Dear developers: sign your code!
• The state of code signing in Open Source

To reduce it for one of our clients, we added the FuzzyOCR plugin to SpamAssassin on their mail servers. The servers were already well defended against spam using the usual mix of SMTP sanity checks, blocklists and SpamAssassin rules, but too much image spam was still getting in.

When a message contains just an image file, the FuzzyOCR plugin runs the image through the open source GoCR optical character recognition utility, then uses fuzzy string matching techniques on any words that pop out.

Although legitimate mail will occasionally contain only images (such as when someone emails photos), it’s rare for those images to contain lots of text. When the words are "mortgage", "invest", and "enhancement", it’s likely that it’s spam.

GoCR is an impressive open-source project but isn’t the world’s most accurate OCR. The words it finds usually come out mangled to a certain degree. Spammers also purposely obfuscate words. Fuzzy string algorithms attempt to take care of that so for example what comes out as "en-hnEmnt" still has a chance of matching spam keyword "enhancement".

The FuzzyOCR plugin adds new scores to messages based on whether words in images match a list of common spam words. Here’s an original spam image and here are the results from the plugin:

  *  6.0 FUZZY_OCR BODY: Mail contains an image with common spam text inside
  *    Words found:
  *    "news" in 7 lines
  *    "breaking" in 1 lines
  *    "symbol" in 1 lines
  *    "investor" in 1 lines
  *    "company" in 2 lines
  *    "money" in 1 lines
  *    "thousand" in 1 lines
  *    "buy" in 1 lines
  *    "trade" in 1 lines
  *    "target" in 1 lines
  *    "banking" in 1 lines
  *    (18 word occurrences found)


Most of the words in the above are present in the image spam, though a few like "money" and "thousand" are not. For this particular message, FuzzyOCR added 6.0 to the overall spam score while other SpamAssassin rules only gave it a score 3.6. The mail server  is configured using our "Fighting malware and spam with Postfix" setup with the sideline filter script… it labels but delivers messages when total score is 4.0 or higher, and quarantines messages scoring 6.0 or higher. So in this case the additional scoring provided by FuzzyOCR successfully prevented the end-users from receiving the spam.

Taking a quick look at the quarantine on the server, we find that 28% of the image spam received exceeded the quarantine limit just from the normal SpamAssassin rules alone, and 72% were quarantined because of the additional scoring from FuzzyOCR. There were no false positive image spams at all in the quarantine.
Very good results so far.

Of course, spam defense is an arms race… every countermeasure is attacked immediately by those diligent little scumbag spammers. In the spam image above the spammer has added noise to the image and anti-aliased some words to reduce the accuracy of OCR. Others use animated GIFs to further confuse OCR software. Two messages received were in PNG format, which seems to be a new tactic. The effectiveness of OCR will decrease, but for right now it seems to be worth having.

Recently Google released Tesseract OCR as open source. Originally developed as a commercial product by Hewlett-Packard, a decade ago it was considered one of the most accurate OCR products. If it’s better than GoCR, hopefully soon it can be used as an alternative OCR engine for SpamAssassin plugins and boost accuracy for obfuscated image spam even more.

Keeping up with spammer tactics is a never-ending chore. OCR is just one of many identification techniques. Interestingly, while every single image-based pump-and-dump spam has been caught by this new filter, traditional misspelled character spam have been scoring low enough to get through. Oh well… time to adjust the SpamAssassin scores again.

Related posts:

• SpamAssassin p0f plugin catches bot spam

Personally I’m conflicted about this. On the positive side, many email security add-ons that only worked with Sendmail can now also be used by Postfix. On the negative, there’s now further incentive for developers to keep writing for the milter API, keeping the crawling horror that is Sendmail alive for even more years.

We replaced Sendmail with Postfix on our mail servers and started installing it for clients back in 2001, and what a joy the past five years have been. Finally… a mailer with a secure architecture that’s fast and easy to configure. No more of the intentionally obtuse "explosion in a punctuation factory" syntax that Sendmail forces admins to endure and with an outstanding architecture, far less chance of root compromise than with Sendmail’s monolithic "runs as root" design.

As a bonus, Postfix is sponsored by IBM, making it easier to sell to management types, and it’s written by security pioneer Weitse Venema, author of TCP wrappers and co-author of The Coroner’s Toolkit (forensics tool) and SATAN (one of the first vulnerability scanners).

One small problem with ditching Sendmail was giving up add-ons like MIMEDefang that use the Sendmail-only "milter" API. Alternatives existed, and of course it was always an option to run Sendmail behind the protection of Postfix as a sort of massively clunky inline filter, but when you’re building email firewalls no one wants software with the security history of Sendmail anywhere near it.

As you might expect, the milter feature of Postfix comes with a list of caveats and gotchas. Not every milter in the world will work (including, I suspect, MIMEDefang). Still, popular ones like domainkeys-milter which adds Yahoo’s DomainKeys anti-spam capabilities apparently work. Some developers like Snertsoft have even starting writing milters with Postfix compatibility in mind.

Perhaps rather than extend the life of Sendmail, this new feature will have the opposite effect and make it easier for admins to finally upgrade to Postfix. If all that’s stopping them are things like lack of DomainKeys support, maybe someday the need to absorb 1232-page books to figure out syntax like "R$* < @ $* .$m. > $*" just to get email working will become a thing of the past.

Related posts:

• A simple tool to track and control spammers

You trustingly sign up using your valuable main e-mail address, get what you need then forget about it.

Weeks later, spam starts arriving with details you provided in that registration. The web site lied! They did sell your registration info to some scumbag or are spamming you themselves.

With the damage done, your main e-mail address that once got very little spam is suddenly inundated with wonderful opportunities for mortgage refinancing, genital enhancement and vacation specials. Once your personal e-mail address starts circulating, it’s impossible to put the genie back in the bottle.

Some people in this situation abandon their once private e-mail address for a new one, then a year later ditch that one when it winds up on spam lists. Others never reveal their “real” e-mail address in favor of using throw-away addresses such as free web mail accounts like Yahoo or disposable mail services like Mailinator.

It’s impossible to tell which web sites can be trusted to keep your information private. Some will never abuse your e-mail address, but for many the demands from the marketing division or temptation to make a quick buck selling the data to “opt-in marketers” is overwhelming.

Further, in some jurisdictions including the U.S., customer lists including e-mail addresses are considered corporate assets. Shareholders can demand selling the lists if a profit can be made. Insolvent companies have also been forced to sell their lists despite privacy policies and their own wishes to protect customers, such as during the infamous eToys debacle of 2001.

Also, it’s often impossible to conclusively prove a specific organization has violated your trust. Anyone you’ve given your one e-mail address to could be guilty.

There is a feature supported by almost all mail servers that lets you give out a unique e-mail address to anyone who asks, yet have messages all wind up in your regular e-mail box. No more checking web mail accounts! Further, if your mail administrator allows blocking, if the address is abused you can forever reject mail to that address before it gets in.

The feature is called “address extensions” and has been available in Sendmail, Postfix and almost every other mail server for years.

A typical address extension looks like this:

A symbol, usually a plus sign, separates your regular e-mail name from a throwaway portion. The throwaway portion is ignored by the mailer when delivering the message… all mail is delivered as if the extension was not there. However, with most mailers the full address with extension still shows up in your mail client, and can be rejected at the mail server.

You can test your own mail server now to see if it accepts this form of addressing. Send an extended address message to yourself using a web mail service or other external mail system and see if it shows up in your inbox.

We’ve been using mail extensions for years to provide unique e-mail addresses to questionable organizations, web site registration forms and the like. For example, when registering on the New York Times to be able to read an article, we might provide the address “jsmith+nytimes@advosys.ca”. If that venerable institution starts spamming the address, we can block it at the mail server. We also have proof who it was who abused our trust and can take action.

Blocking at the mail server is best. If the mail server rejects the message during the SMTP “envelope” phase (i.e. the “RCPT TO:” command), the spam is rejected before the message body is even sent. The sender gets a delivery failure error, even if they’ve forged the sender address (like almost all spammers do).

If you can’t block at the mail server, most mail clients allow creation of rules to automatically delete messages containing specific headers. This prevents you from seeing the spam, but your mail server and network resources are still abused. Also, the spammer will not see a rejection message and will keep sending.

Note that hardcore spam operations get paid per message accepted by the destination, so deleting spam in your mail client may result in rewarding the spammer with money. Blocking at the mail server is preferable.

Spammers are unethical scum, but they are very devoted to circumventing spam blocks. Some spammers are aware of the “plus sign” form of address extension and strip that portion from addresses they collect.

You can reduce the chance of that by using a less common character. For example, many organizations use periods, dashes or underscores in e-mail addresses (“john.smith@example.com”, “smith_j@example.org”). If possible, change your mail server to use one of those characters for address extensions instead of the plus sign. Since so many legitimate e-mail addresses use those characters, spammers are far less likely to filter them from their spam lists.

In the Postfix MTA, the extension character can be specified using the recipient_delimiter setting in the main.cf file. Similar settings are available in Qmail and other popular SMTP servers, though apparently in Sendmail the plus character is hard coded.

Downside

One potential downside to address extensions is it could open your mailer to certain forms of abuse. If a spammer determines you are using extensions they may abuse the feature to send you huge volumes of mail. Some spam reduction filters block senders automatically when they exceed a threshold of messages to the same address. Unless such a filter is also aware of the extensions, it may allow allow the spammer to use random extension addresses to flood recipients.

However, most mail filters block senders that exceed a threshold regardless of recipient address. Also, spammers rarely craft exploits targeted at specific organizations, except for obvious huge targets such as Hotmail and AOL. Unless your organization has millions of e-mail users or a spammer has a vendetta specifically against you, a targeted attack is unlikely.

Regardless, it’s wise to test the behaviour of your specific mailer’s filtering functions before implementing address extensions, just in case.

Summary

Using addresses extensions to provide unique e-mail addresses can help identify who abused the address provided and block the address once it starts to be spammed.

Almost all mailers support address extensions, usually using the form “username+extension@domain.com”

Blocking an abused address is best done at the server by rejecting the message during the SMTP conversation. If that’s not possible, most email client software have filters that can delete mail containing specific headers on arrival.

Since spammers know about address extensions using the plus sign, your mailer should be configured to use something common in other organization’s e-mail addresses, such as a dash, underscore or period.

Conclusion

This technique is a simple and widely available way to give out unique aliases for your regular email address that you can track and block if abused. It’s more convenient than setting up disposable web mail accounts, and more permanent that using disposable e-mail services such as Mailinator.

Used in conjunction with other spam reduction techniques, address extensions can significantly reduce the spam you receive. It gives you the recipient more control over your inbox and an ability to determine for certain who has abused your private information.

Related posts:

• ICANN speaks up on Spamhaus litigation
• SpamAssassin p0f plugin catches bot spam
• Court declines to suspend Spamhaus domain