Spam Filtering

Computing Services Information Document - Email

Beginning September 1, 2003, Camden Computing Services will have available a spam filtering service for people with clam and crab accounts. Spam is unwanted and unsolicited email, also referred to as junk email and formally known as Unsolicited Commercial Email (UCE). It typically takes the form of solicitations for products, services, or web sites and often has a forged sender address. Due to the increasing volume of such unwanted email messages, software called SpamAssassin has been installed to help detect them. To enable the spam filtering service for your email you will need to visit one of the web pages listed below.

SpamAssassin works by analyzing the header, body, and attachments of each email message using a set of heuristics and external programs. Then it calculates the probability that a given email is indeed spam. Once a probability threshold has been reached or exceeded, the message is flagged as spam and then can be automatically removed from your Inbox and placed into a special Spam-Messages email folder, have the subject line tagged indicating spam and delivered to your normal inbox, or automatically deleted. Unfortunately it is impossible to eliminate 100% of spam, but a large percentage should be removed by this filter. Some have claimed that the heuristics used by SpamAssassin catch as much as 80-90% of spam.

Our servers will be analyzing all incoming messages using SpamAssassin. You will be able to enable or disable the filter, adjust the threshold of your personal filter, and control the action to take on spam using a web-based interface. This interface will also allow you to view the size of and empty the contents of the Spam-Messages folder if you are using it. You can access the SpamAssassin interface by going to the URL(s) below:

Students:	https://clam.rutgers.edu/cgi-bin/usertools/webspamassassin
Faculty & Staff:	https://crab.rutgers.edu/cgi-bin/usertools/webspamassassin

As of January 5, 2004 all student accounts have had spam filtering enabled. All NEW crab accounts created on or after May 15, 2006 will have spam filtering enabled. The default behavior will be to place all messages marked as spam (with a threshold of 5) into the Spam-Messages folder.

Some IMAP email clients will require you to subscribe to the Spam-Messages folder in order to view its contents. Other email clients (such as WebMail and pine) should display this folder without any IMAP subscription. It is important to note that if your email client uses POP, you will not be able to view the Spam-Messages folder at all. POP email clients can not see any folders on the server other than Inbox. POP users may instead select the button that enables subject line tagging. This will add {Spam?} to the subject line of any spam and deliver it to your usual inbox. Camden Computing Services recommends the use of IMAP enabled email clients. If you are not certain whether your email client is using POP or IMAP, please contact your departmental support staff or help@camden.rutgers.edu. More information about this can be found below.

Spam Filtering Issues:

False Positives/False Negatives:

False positives are legitimate email messages that have been incorrectly marked as spam. This can and will happen with any spam filtering software. Therefore, it is important to periodically examine the contents of your Spam-Messages folder or email with spam-tagged subject lines. If you find that the spam filter is too sensitive you can use the web interface mentioned above to increase the threshold of spam sensitivity. False positives create a danger for losing valid email if you select the action to delete spam.

False negatives are spam messages that have not been correctly flagged and moved into your Spam-Messages folder, tagged on the subject line, or deleted. Like false positives, these are also impossible to avoid. To reduce the occurence of false negatives, the spam sensitivity threshold can be lowered from the web interface mentioned above.

There may be instances when valid email that you commonly receive has the appearance of spam and you cannot raise your threshold to compensate for this without the expense of many real spam messages making it through. Conversely, there may be times when spam appears in your mail that did not resemble spam strongly enough to be caught, and lowering the sensitivty threshold would cause too many valid messages to be flagged. In these cases you may wish to take advantage of the advanced features of the Spam Filtering service, whitelisting and blacklisting. These are detailed under the Advanced Features section below.

Spam folder:

Messages tagged as spam can be moved into a folder named Spam-Messages if you select that spam handling action. This folder exists in your account under your Maildir/ directory, and therefore it does count against your disk space quota. The web interface will allow you to examine the size of this folder, and expunge all messages from it. All email clients that allow you to view this folder will be able to expunge messages from it as well. Every message that has been in your Spam-Messages folder for over 14 days will be removed automatically at the end of each night. This automated process will keep your Spam-Messages folder from growing too large, while allowing you the chance to go back and look at filtered messages from the current month. This folder may not exist if you have chosen to either tag subject lines or delete spam. (This feature is currently not available for faculty/staff accounts until the Fall Semester)

Email Forwarding and Vacation Message:

If you are currently forwarding your email from the Rutger's email servers to another host, and have not also explicitly setup delivery to your Rutger's account as well, the spam filtering service will not affect your email. Forwarding occurs prior to our spam filters, so any spam email will be forwarded along with the valid email. As mentioned below in the Technical Details section, the email header will still be altered by our spam filter. This data may possibly be used with spam filters on the host where your email is being forwarded, or with the client that is configured to read email from that host.

If you are using the Vacation feature of the email system then replies to incoming email will take place before spam filtering. This means that a response will be sent to each unique spam message, and most likely (since spammers forge their email addresses) the remote email server will "bounce" a message back to you about the email address not being valid. If this is undesirable then some of the Advanced Features such as subject line filtering may help.

Email Clients:

While some IMAP email clients will automatically display your Spam-Messages folder, others may require you to manually subscribe to it. Consult the documentation for your particular email client (most have some form of built-in documentation) to see how you go about subscribing to a new IMAP folder. If you are using a POP email client, spam filtering will still work but you will need to use a non-POP email client (like WebMail) to view and manage your Spam-Messages folder or simply select the option to tag the subject line which will leave the spam in your inbox or select the option to automatically delete spam.

Webmail, Pine, Macintosh Mail (using IMAP), Microsoft Outlook (using IMAP), Netscape Mail (using IMAP), and Mozilla Mail (using IMAP) support automatic IMAP folder subscribing, while Eudora, Microsoft Outlook Express, and Microsoft Entourage do not. These non-autosubscribing clients can however be configured to view the Spam-Messages folder by manually subscribing to it.

If you are using either Webmail or Mozilla and you do not see the Spam-Messages folder you can subscribe to it by doing the following:

Webmail
1.) Login to Webmail
2.) Click the "Options" icon
3.) Click "Server Information"
4.) Check "Use IMAP folder subscriptions" and click "Save Options"
5.) Click the "Folders" icon
6.) If you do not see the Spam-Messages folder in the list click "Show Unsibscribed"
7.) When you see the Spam-Messages folder check the box next to it
8.) Select "Subscribe to Folder(s)" from the drop down box

Mozilla
1.) Launch the Mozilla "Mail & Nesgroups"
2.) Select "Edit" from the toolbar
3.) Select "Mail & Newgroups Account Settings..."
4.) Select "Server Settings"
5.) Click the "Advanced..." button
6.) Enter your mail directory (Maildir/) in the "IMAP server directory" box
7.) Enter your password if you are prompted

It is also important to note that some email clients (such as Netscape 7 and Mozilla) include their own spam filtering tools which may be enabled by default. These may cause email that our systems believes to be valid to be moved to a seperate spam folder, possibly Spam-Messages if the client is configured to use that email folder. This may be mail that is scored lower than your set threshold, or has a whitelisted sender address.

Advanced Features:

Further spam filtering features can be accessed from the web tool by clicking on the button labeled "Display Advanced Options" next to the heading "Advanced Filtering Option". Clicking on this button will expand the page and make available four new features. Be very cautious when changing these options as they are very powerful and misconfigurations can result in lost email.

If you wish to remove these options from the page, click on the "Hide Advanced Options" button which has replaced the "Display Advanced Options" button.

Blacklisting allows you to create lists of either email addresses or subject lines that you never wish to receive email containing. This is useful for particular email addresses that commonly escape the spam filter or are sending offensive email to you and subject lines that are common to email that you never wish to read. This email is immediately deleted and will not be tagged in the subject line or moved to the Spam-Messages folder.

Whitelisting allows you to create lists of email addresses that you never wish to block regardless of the message contents. This is useful for those users that send you valid email appearing to be spam by our filters.

To add whitelisting or blacklisting enter either an entire email address or subject line or a part of either into the appropriate box and click the "Add" button. You can enter multiple entries for each, but you can only do so one at a time. To delete a blacklisted or whitelisted item, highlight it and click on the appropriate "Delete" buttton.

The following are several examples that can be used with explanations (Note: All of these are case insensitive and the quotation marks should be excluded, they are here only to show spacing):

Blacklisting Addresses:
"spammer@spam.com" - email addresses containing 'spammer@spam.com' will be blocked. (Note: This means also that 'iamaspammer@spam.com', for instance, or any other email address containg the string 'spammer@spam.com' anywhere in it would also be blocked)

" spammer@spam.com " - 'spammer@spam.com' only will be blocked.(Note the space before and after 'spammer@spam.com')

"spam.com " = Any email address from the domain 'spam.com' (ie. iamaspammer@spam.com, spammer@spam.com, validemail@spam.com, and any other spam.com user) will be blocked.

Blacklisting Subject Lines:
"free" - email containing the word 'free' anywhere in the subject line will be blocked (Note: This would also block any email with 'freedom' in the subject line, for instance.)

" free " - email containing the word 'free' by itself in the subject line will be blocked. (Note the space before and after 'free')

Whitelisting Addresses:
These work in the same fashion as Blacklisting Addresses, but with the oposite result, the email addresses are never blocked.

If you have any problems or questions, please contact us at help@camden.rutgers.edu

Technical Details:

Email Client Spam Filtering:

Some email clients can perform filtering based on various criteria. SpamAssassin adds information to the message header of each email message that your client may be able to use for its own internal filter. What this means, is that even if you do not choose to opt-in to the server-side filtering offered by Camden Computing Services, you might still be able to perform client-side filtering with your email application. The following lines are an example of the type of information that is added to the message header by SpamAssassin:

X-Spam-Level: **********

X-Spam-Status: Yes, hits=10.9 tagged_above=1.0 required=2.0 tests=DRASTIC_REDUCED, MORTGAGE_PITCH

The name of these header lines are X-Spam-Level and X-Spam-Status, and following them are fields separated by whitespaces. The X-Spam-Level line shows the integer score (10) as a series of asterisks. The X-Spam-Status line shows the exact decimal score (10.9) assigned to this particular message. The higher the score, the higher the probability that the message is spam. Any message that is scored at 5 asterisks or above, or a decimal value of 5 or above, is very likely spam. The score range is from 1 to 40. The final field of X-Spam-Status is a comma-separated list of rules that SpamAssassin matched against when weighing the probability that a message was spam. If your email client supports filtering based on custom message headers, the two important parts for filtering purposes are the number of asterisks or the decimal value that follows.

UNIX Files:

The following files will be placed into your UNIX (clam or crab) account when you enable spam filtering. Removing or editing these files manually, instead of using the web page, may break your filter. Manually removing or editing these files could also lead to loss of incoming email. Email messages lost as a result of incorrect changes may not be recoverable.

Maildir/mailfilter.spam, Maildir/mailfilter.blacklist, Maildir/mailfilter.whitelist : Maildrop (an email local delivery agent that our Spam Filtering service utilizes) can be used for purposes other than spam filtering, but very few people actually know how to write their own Maildrop "recipes". If you intend to use Maildrop for anything in addition to spam filtering, please do not modify any of these files.

Maildir/.Spam-Messages: This is the directory where incoming spam messages will be stored if that action is selected.

Questions/comments regarding Camden Computing Services or this web site
can be directed to: help@camden.rutgers.edu.
Last updated: March 15, 2007, 22:03 EDT.
© 2005-2008 Rutgers, The State University of New Jersey. All rights reserved.