Backscatter Spam

One of the areas where spam detection and filtering has always been difficult is with bounce backs - AKA backscatter. It's very difficult to scan these as they sometimes don't include all of the original message rendering heuristics ineffective (where AI reads and understands the message), and always include a header specific to the mail server it was sent from so that signature based scanning doesn't work. Every type and version of mail server attaches a different header making it difficult to detect what is and isn't a bounce.

Fundamentally, this is only a problem because some mail servers are badly configured and accept mail when they shouldn't. I can't change that.

Our solution:
The RocketUK mail servers now have the ability to detect and score and bounces. For the reasons above this is not infallable but has proved effective in testing.

378 out of 439 bounces received between midnight and 9:30am were marked or blocked as spam.
Previously, most of those would have been passed as clean.

The benefits:
This will markedly reduce the amount of junk bounce backs our customers receive.
I am unaware of any other ISPs having implemented this type of scanning.

Mail Server Quotas

We've always said that we have a mail quota but this has never been enforced.
We're now introducing quotas on mailboxes of 100MB. This affects almost no customers - 95% use POP3 and never leave more an a couple of MB on the server before downloading.

What is does mean is that we're able to provide more IMAP/WebMail services - where the users leave their mail on our servers rather than downloading to their own computers.
This is ideal for people who are more often mobile and don't have an office server looking after email.

Another mail filter

To reduce spam: in testing is a new filter ('policyd') which will analyse the patterns of other computers on the net sending us email. If the patterns match spam behavoir or they are caught sending spam we can automatically blacklist them for long periods of time.
Currently, hosts can't be automatically blacklisted.

Microsoft is getting tougher on UK businesses and threatening legal action using its agents the Business Software Alliance (BSA) for companies that do not comply with requests to keep their software licenses up to date.

Under the new program, if Microsoft doesn't receive a response after 14 days, the company will send a succession of three "escalation" letters over three weeks. The last two letters warn that the case could be turned over the BSA, which could pursue legal action.

If you're unsure of your current licensing status give us a call on 01952 588688 for advice.
Midland Computers are Microsoft Partners supplying official UK license products.

Thanks to our friends at F-Secure, we are now running the FreeBSD version of F-Secure's fsav on homer, the RocketUK mail server.

fsav is in addition to our standard unix virus checker ClamAV - all messages are scanned by both. ClamAV also scans the stored mail nightly and fsav may be added to this in the future. Mail is rescanned nightly because sometimes malicious email does leak through before virus analysers can get signatures out to block them.

One of our spam preventing measures, called Greylisting, has also proved to be extremely effective at removing viruses. Around 90% of viruses are eliminated before any direct checks and analysis are even needed.

On the spam problem, we continue to improve our filtering. We use two seperate engines which are constantly tuned to provide the best results for a day - SpamAssassin and DSpam. Over the last two years, these have been carefully trained and adjusted on a daily basis.

In average over a month 88% of mail is rejected outright as spam or viruses (this information is a little outdated, and doesn't cover the problems caused by the recent Stration/Warezof virus).

2% is passed on as "possible spam" (when you see "*** SPAM ***" in the subject line).

The remaining 10% is the email we receive.

I've been asked to post a little about what steps we take to prevent spam at RocketUK for our customers - it's a little rushed and doesn't cover everything but helps explain what we do.

What we do daily to help:

1. Ensure the latest stable versions of the software we use are installed.


2. Research new ways to combat the problem. New algorithms, techniques and technology are being produced all the time.


3. Train the bayesian filters: these are systems which learn what to recogise as spam, a form of artificial intelligence. In order to maintain them in good working order, we need to give examples of what is and isn't spam. These messages are analysed and learned from. We do this using messages from two sources: my own personal emails, messages for generic email addresses at Midland such our sales and technical mailboxes (which attract a lot of spam), and a variety of domains we are holding but which aren't in any use - yes, even domains never used get a lot of spam..


4. Fine tune scoring filters - i.e. last week, stock market spams were common so the filters and rules matching text to do with 'stock' and 'shares' were made slightly more aggressive. This week the problem has subsided and the rules relaxed again.


5. Implement new SpamAssassin rules from 3rd parties. Published rules are of variable quality and this requires a lot of care.

What we have done recently:

Over the last few months a number of significant improvements have been made:

1. dspam bayesian filter. Though not quite as impressive as the marketing makes out, this has helped provide assistance to the SpamAssassin engine by providing a second opinion which is fed back in both directions - a bit like the Space Shuttle computers where there are four computers for the same task, each one running software written by a different team and double checking the results of each other.


2. Greylisting: a hugely effective method to filter spam which we upgraded to a more sophisticated version with great success about a month ago.


3. Implemented several SpamAssassin plugins:
   - DCC
   - Pyzor
   - Razor2
   - SpamCop
   (The four above are community efforts, where common spam messages are submitted for others to block without ever needing to see them).
   - CountryRelay detection
   - more blocklists
   - Switched 'Habeas' from being a whitelist to a blacklist with 100% success...

What we've been doing in general:

We run two virus checkers in addition to spam filters - F-Secure and ClamAV. One of these virus checkers, ClamAV, has an additional benefit in that it detects and removes most 'phishing' emails - from a technical viewpoint these are something like a cross between spam and viruses.
The reason for running two is that in the first minutes and hours of a new virus being unleashed, we have effectively two teams of analysts working on the problem of decoding and blocking it and no matter which is quickest we'll see the benefit.

We're also looking at some future enhancements, particularlly to combat spam with images - you might have noticed these, where the text isn't in the message but a picture attached to the email. Work is going on with detecting these using OCR, but this is very complex and very CPU intensive to do. Another problem is that spammers have already developed defenses against it, image spam now has background interference patterns and is quite often animated. A nightware to detect but work continues.

We maintain graphs of spam analysis performance here:
http://mail.rocketuk.net/virus-stats/

Basically, the blue is email received, the red is spam. As you can see a huge amount is blocked! You can also make out the current 'outbreak' near the end of September 2006 where there is a significant upturn (the decrease at the start of September coincides with improvements we made here, including the new Greylisting).

Spam has increased recently due to a number of new viruses (particularly Stration/Warezof). These viruses infect computers and start sending out email. There are probably many thousands of these machines in the world. One of the tricks these are using is to send out spam using random headers created from domain names picked at random - the idea being that if a domain really exists, it's far more likely to make it through a spam filter. By chance, a lot are going to use your domain for either the To or From - it's likely there is more spam sent in a day than domains exist.

This does *not* necessarily mean that the spam is coming from your computer. Of course, ensure your anti-virus is up to date at all times.

Welcome to the Internet departments first blog entry. This is where we intend to keep our customers informed of all the goings on within the Internet department.