Thanks to our friends at F-Secure, we are now running the FreeBSD version of F-Secure's fsav on homer, the RocketUK mail server.

fsav is in addition to our standard unix virus checker ClamAV - all messages are scanned by both. ClamAV also scans the stored mail nightly and fsav may be added to this in the future. Mail is rescanned nightly because sometimes malicious email does leak through before virus analysers can get signatures out to block them.

One of our spam preventing measures, called Greylisting, has also proved to be extremely effective at removing viruses. Around 90% of viruses are eliminated before any direct checks and analysis are even needed.

On the spam problem, we continue to improve our filtering. We use two seperate engines which are constantly tuned to provide the best results for a day - SpamAssassin and DSpam. Over the last two years, these have been carefully trained and adjusted on a daily basis.

In average over a month 88% of mail is rejected outright as spam or viruses (this information is a little outdated, and doesn't cover the problems caused by the recent Stration/Warezof virus).

2% is passed on as "possible spam" (when you see "*** SPAM ***" in the subject line).

The remaining 10% is the email we receive.

I've been asked to post a little about what steps we take to prevent spam at RocketUK for our customers - it's a little rushed and doesn't cover everything but helps explain what we do.

What we do daily to help:

1. Ensure the latest stable versions of the software we use are installed.


2. Research new ways to combat the problem. New algorithms, techniques and technology are being produced all the time.


3. Train the bayesian filters: these are systems which learn what to recogise as spam, a form of artificial intelligence. In order to maintain them in good working order, we need to give examples of what is and isn't spam. These messages are analysed and learned from. We do this using messages from two sources: my own personal emails, messages for generic email addresses at Midland such our sales and technical mailboxes (which attract a lot of spam), and a variety of domains we are holding but which aren't in any use - yes, even domains never used get a lot of spam..


4. Fine tune scoring filters - i.e. last week, stock market spams were common so the filters and rules matching text to do with 'stock' and 'shares' were made slightly more aggressive. This week the problem has subsided and the rules relaxed again.


5. Implement new SpamAssassin rules from 3rd parties. Published rules are of variable quality and this requires a lot of care.

What we have done recently:

Over the last few months a number of significant improvements have been made:

1. dspam bayesian filter. Though not quite as impressive as the marketing makes out, this has helped provide assistance to the SpamAssassin engine by providing a second opinion which is fed back in both directions - a bit like the Space Shuttle computers where there are four computers for the same task, each one running software written by a different team and double checking the results of each other.


2. Greylisting: a hugely effective method to filter spam which we upgraded to a more sophisticated version with great success about a month ago.


3. Implemented several SpamAssassin plugins:
   - DCC
   - Pyzor
   - Razor2
   - SpamCop
   (The four above are community efforts, where common spam messages are submitted for others to block without ever needing to see them).
   - CountryRelay detection
   - more blocklists
   - Switched 'Habeas' from being a whitelist to a blacklist with 100% success...

What we've been doing in general:

We run two virus checkers in addition to spam filters - F-Secure and ClamAV. One of these virus checkers, ClamAV, has an additional benefit in that it detects and removes most 'phishing' emails - from a technical viewpoint these are something like a cross between spam and viruses.
The reason for running two is that in the first minutes and hours of a new virus being unleashed, we have effectively two teams of analysts working on the problem of decoding and blocking it and no matter which is quickest we'll see the benefit.

We're also looking at some future enhancements, particularlly to combat spam with images - you might have noticed these, where the text isn't in the message but a picture attached to the email. Work is going on with detecting these using OCR, but this is very complex and very CPU intensive to do. Another problem is that spammers have already developed defenses against it, image spam now has background interference patterns and is quite often animated. A nightware to detect but work continues.

We maintain graphs of spam analysis performance here:
http://mail.rocketuk.net/virus-stats/

Basically, the blue is email received, the red is spam. As you can see a huge amount is blocked! You can also make out the current 'outbreak' near the end of September 2006 where there is a significant upturn (the decrease at the start of September coincides with improvements we made here, including the new Greylisting).

Spam has increased recently due to a number of new viruses (particularly Stration/Warezof). These viruses infect computers and start sending out email. There are probably many thousands of these machines in the world. One of the tricks these are using is to send out spam using random headers created from domain names picked at random - the idea being that if a domain really exists, it's far more likely to make it through a spam filter. By chance, a lot are going to use your domain for either the To or From - it's likely there is more spam sent in a day than domains exist.

This does *not* necessarily mean that the spam is coming from your computer. Of course, ensure your anti-virus is up to date at all times.

Welcome to the Internet departments first blog entry. This is where we intend to keep our customers informed of all the goings on within the Internet department.