Appies
[ Table of Contents ]

Appies Spam Filters

Appies Spam Filters

According to a recent survey, the average business person spends about 15 minutes a day dealing with spam.  This includes setting spam filters, downloading and reading spam, reporting and responding to spam, and removing viruses due to spam.  Fifteen minutes a day equals almost 8 hours a month -- that is a full workday of productivity lost.  The actual time lost is even more, however.  Imagine trying to get a sound sleep and being awakened dozens of times throughout the night.  The effects of the interruption of spam throughout the workday are not all that different.  There is also an emotional side to the effects of spam.  Spam often angers, and even infuriates, those who receive it.  This feeling of anger distracts from the focus of the productive work itself.  Thanks to Appies Spam Filters, spam no longer needs to be an issue.

 Awesome Feature...

The NEW Appies Spam Filter application is without a doubt a masterful combination between the most complex regular expression (REGEX) based filtering system and the easiest-to-use spam prevention tool on the market today.  Just a few clicks, and you will reduce your spam significantly no matter what system you were using before.

How Spam Filters Work

Let's start with the basics.  There are essentially two general categories of spam filters: 1) client-based filters and 2) server-based filters. With client-based filters, the user is forced to download the spam wasting time and risking infections from viruses.  Server-based filters, such as the ones used by Appies, filter the spam before it is even delivered to the users POP account.

 Remember...

A POP account is a file on the server where mail is actually stored.  The e-mail client reads the mail from this file. Spam filters are applied to POP accounts -- not aliases or forwarding accounts such as lists and autoresponders.

The spam filters then search the message for patterns indicating that it is spam.  Once the filter finds the pattern, it takes a specified action on the message -- usually by a) deleting it b) rejecting it (sending an e-mail back to the sender) c) forwarding it to another e-mail address and/or d) modifying the message.  The end result: less spam and more productive time.

 Technical Note...

Appies spam filters use the UNIX utilities "formail" and "procmail" for its filtering.

The Spam Dilemma

Although the above description of how spam filters work is accurate, it is simplified dramatically.  Fighting spam is both a science and an art that takes time and knowledge. 

Most spam fighting software requires users to create own their rules.  Very few people know how to create rules that effectively block spam.  What ends up happening is that they create rules that block too much legitimate e-mail, or they create simplified rules like blocking the term "Viagra" -- which even the most rookie spammers would never use without carefully munging the term.  And that is if they bother to even try keeping up with creating rules, which can be a full time job in itself.  Appies spam rules are created by spam fighting experts with over 11 years experience, and updated several times a week.  Users have the ability to choose which rules they want to leave out.  This means, Appies spam filters provide all the protection of a complex, customized spam filter system with the simplicity of the most basic system.

Appies Spam Filters Features

  • setup whitelists that allow e-mails to bypass the filters based on To address, From address, Subject, IP address/network
  • software based rules updated several times per week
  • ability to add your own rules as well
  • full regular expression (REGEX) matching allowed for most blocks -- not limited to just the wildcard
  • Three levels of spam protection: 1) spam to delete 2) spam to reject with a custom message defined by you 3) spam to deliver with appended headers to address of your choice
  • choose to disable any level of protection
  • designed for one administrator to easily update and maintain filters on multiple POP accounts
  • level 3 forwarding address can forward level 3 spam to any valid e-mail address on a per POP basis
  • level 1 & 2 spam are logged with detail, including exact rule AND term matched, to spam log, which can be e-mailed to any address at the end of each day
  • level 3 spam is logged as well as modified to include the headers "X-Spam:" and "X-Match:" which will identify the message as possible spam and show the exact term matched, respectively
  • 100% web-based interface

 Technical Note...

What are regular expressions (REGEX)? See Perlcode.org for a detailed description of what they are and how to implement them.  Users are NOT required to know how to use REGEX -- this is only for advanced users who wish to get the most from Appies spam filters.

Your Spam Fighting Strategy

First, make sure you have your POP accounts setup on your server.  This is done under "E-mail Management" >> "POP Accounts".  Remember that spam filters are for POP accounts on your server only. Now you are ready to activate your new spam filters based on your spam fighting strategy.

 Be Careful...

If you have been using the old Appies spam filters, these will continue to be active until you start using the new filters.  Once you use the new filters, you cannot go back to the old filters.

Your role as "The Spam Administrator".  Those with administration access to Appies can play the part of the spam administrator for all the POP users.  In a business environment, it is best not to leave this up to the individual, since we already discussed many reasons how dealing with spam can take away from productivity.  In addition, we cannot rely on each and every account user to have the technical know-how to activate these filters (no matter how simple they may be) and the wisdom not to create a bad rule that will cause them to lose mail both valuable to them and the organization.

The three levels of spam.  To simplify the whole concept of spam filtering, we have put all spam into 3 groups or levels: level 1, level 2, and level 3 spam.

  • Level 1: These are messages that have a 99.9% chance of being spam.  The filters look for viruses, purposely misspelled words and other common spammer tricks (did any one say V1@gr@?)  All level 1 spam is sent immediatly to the trash (deleted) and not delivered to the recipient.  However, these messages are logged in the spam log (see below).  In that rare case the email is legitmate, you can easliy email the sender asking to resend.
  • Level 2: These are messages that have about a 99% chance of being spam.  For most people, having to download and being forced to view some of these emails is less than desireable.  At the same time, we do not want to forget about that 1 message out of 100 that we actually may want.  All level 2 spam is rejected, which means it is sent back to the "sender" with a message that you can customize, usually asking the sender to resend if it is a legitimate e-mail. Note: the chances are about 99 in 100 that the reply-to address is fake or invalid, and your response will not be read by a human.  Only if the e-mail is legitimate, will a human actually see your response and resend the message to the whitelisted address you specify in the autoresponse.
  • Level 3: These are messages that have less than a 99% chance of being spam.  Level 3 messages are delivered to an e-mail address of your choice (forwarded) -- and this could be done on a per POP basis, meaning each POP account can forward level 3 spam to its own forwarding e-mail address.  Before the e-mail is forwarded, it is logged and the headers "X-Spam" and "X-Match" are added to the e-mail.  This will allow the owners of the POP accounts to setup their e-mail client to manage their own spam.


A snapshot of the Appies Spam Report that can be viewed in real time through Appies, or e-mailed at the end of each day.  Notice how the report tells not only the rule, but which term was matched as well.  This makes it very easy to exclude terms that catch legitimate e-mail.

As the spam administrator, you can chose to a) use spam filters on none, some, or all accounts and b) apply level 1, level 2, and/or level 3 filters to those accounts on which you have chosen to use spam filters.

Choosing which POPs to apply the spam filters.  On the main screen, there will be a multiple select form field containing all the POP accounts for that Appies account.  Using the control key and the mouse, click on the POP accounts you wish to have filters on.  Now, click the "update" button to the right.  By default, all rules will be applied to all selected accounts.  You can stop there, or continue to customize your spam protecting strategy.

 Technical Note...

When you add new POP accounts, they are NOT setup with spam protection.  You must revisit this area and select the accounts, then click the "update" button again.

Continuing to customize the spam protection based on strategy.  Now that you have full protection on the chosen accounts, it is time to make some customizations based on your organization's specific needs.  Which strategy do you think would work best for your organization?

  1. Delete, reject, and forward spam. This would be the most common strategy used by most organizations.  It will block the most spam by far, but will require careful monitoring.
  2. Delete and reject spam. Those organizations that want to eliminate most of the spam, but don't want to be involved in forwarding "possible" spam would choose this strategy.  To implement this, deactivate level 3 spam by unchecking the checkbox after opening clicking the "Customize level 3 spam" link.
  3. Reject and forward spam.  If the thought of just deleting e-mails that have even a 1 out of 1000 chance of being spam is too much to handle, you can deactivate level 1 spam by unchecking the checkbox after opening clicking the "Customize level 1 spam" link.  Once you do this, all the default subjects and other terms found anywhere in the message for level 1 will be "rolled up" into the level 2 spam terms, so you will still block those terms, but under the level 2 reject rule as opposed to just trash.
  4. Delete and forward spam.  If you do not like the idea of rejecting spam and sending an autoreply, you can deactivate level 2 spam by unchecking the checkbox after opening clicking the "Customize level 2 spam" link.  Once you do this, all the default subjects and other terms found anywhere in the message for level 2 will be "rolled up" into the level 3 spam terms, so you will still block those terms, but under the level 3 forward rule as opposed to just rejecting the message.
  5. Just delete spam.  If your main concern is protection from viruses and the reduction of the majority of spam, you can deactivate level 2 spam by unchecking the checkbox after opening clicking the "Customize level 2 spam" link and deactivate level 3 spam by unchecking the checkbox after opening clicking the "Customize level 3 spam" link.  This strategy is the most "hands off" for the spam administrator.
  6. Just reject spam.  You can deactivate level 1 spam by unchecking the checkbox after opening clicking the "Customize level 1 spam" link and deactivate level 3 spam by unchecking the checkbox after opening clicking the "Customize level 3 spam" link. Once you do this, all the default subjects and other terms found anywhere in the message for level 1 will be "rolled up" into the level 2 spam terms, so you will still block those terms, but under the level 2 reject rule as opposed to just trash.
  7. Just forward spam.  If your POP account users are savvy or insist that they manage their own spam at a client level, you can deactivate level 1 spam by unchecking the checkbox after opening clicking the "Customize level 1 spam" link and deactivate level 2 spam by unchecking the checkbox after opening clicking the "Customize level 2 spam" link.  With this strategy, you will not delete or reject any spam, you will just be modifying the headers and forwarding on to the individual users.  From there, the users can setup their own filters on their e-mail client of choice based on the "X-Spam:" header.

 Helpful Hint...

If you choose not to use level 1 spam, you will lose the default file blocking (virus) as well as the "to" and the "from" blocking.  You can always add your own rules for anywhere in the message that can do the same thing.  Make sure you have REGEX checked and for file extensions enter:

name=".*.(pif|scr|url|vbs|zip|etc)"

 

Whitelists and the Levels Explained

A) Set up whitelists.  A "whitelist" is a list of allowed entries where mail matching any of these entries will bypass all other spam filters. Start by entering your whitelists that will apply to all selected POP accounts.  Click the "Customize Whitelists" link in your Appies.

The screen will refresh and expand with rules WL-1 through WL-4 where you can add your whitelists.

WL-1) Subjects to whitelist: Subjects matching your entries will bypass spam filters -- one entry per line.  By default, REGEX is not checked.  Therefore, any special characters you add such as * ^ $ () or others will be taken literally.

WL-2) TO addresses / domains to whitelist: "To" or "Cc" addresses matching your entries will bypass spam filters -- one entry per line.  This is where you would enter your "top secret" spam free e-mail address that you do not list on any website. 

WL-3) From addresses / domains to whitelist: "From" addresses matching your entries will bypass spam filters -- one entry per line.  This allows for partial matches so if you enter "@hotmail.com" then any hotmail.com address will be allowed to send any message to you -- even a virus!  It is best to use compete addresses whenever possible since most viruses are sent from forged e-mail addresses -- usually from someone you know.

 Be Careful...

Each line looks for matches only, so if you chose to accept the e-mail address bo@hotmail.com you will also be accepting dumbo@hotmail.com, baggins.bilbo@hotmail.com, and this_is_a_virus_bo@hostmail.com.

WL-4) IP addresses / networks to whitelist: IP addresses or networks in the "Received" header matching your entries will bypass spam filters -- one entry per line. Like the e-mail addresses, partial matches allowed, so use as much of the IP or network name as possible.

 Helpful Hint...

Since it is easy for spammers for forge headers, your best bet is to create whitelists based on the subject of the e-mail, assuming the subject is specific.

After you make your entries, click the "customize whitelist" button to save your changes.

B) Customize Level 1.  By default, level 1 (and all other levels) are activated as indicated by the checkbox. Click the "Customize Level 1 Spam Protection" link in your Appies.

The screen will refresh and expand with rules L1-1 through L1-5 where you can remove terms from the default lists as well as add your own.

L1-1) File extensions to block: Enter one extension per line. Remove or add as many file extensions as you wish.  By default, a known list of common virus extensions have been entered.  The ".zip" extension is included as well.  Even though this is a common file extension of legitimate files, it is perhaps the most common way to send viruses.  If you commonly accept .zip files, have users send them to a whitelisted address.  It is very uncommon for a virus to be over 150k in size -- so the option exists to allow all larger files with those extensions.

L1-2) Default subjects being BLOCKED: These are the default blocked subjects updated and maintained by BDC. You can use your control key while selecting multiple entries that you do NOT want blocked -- those NOT selected are automatically blocked. Note: REGEX (regular expressions) used for matching. These are in order based on date added.

L1-2b) Custom subjects to BLOCK: Add your own subjects to block in this area -- one per line. These blocked e-mails will go directly to the trash so be careful with what you add.

 Be Careful...

It is strongly suggested that you do NOT block  any term fewer than 4 characters in the subject.  You can use spaces, and that does count as a character.

L1-3) Default spam terms being BLOCKED: These are the default blocked spam terms updated and maintained by BDC. You can use your control key while selecting multiple entries that you do NOT want blocked -- those NOT selected are automatically blocked. Note: REGEX (regular expressions) used for matching. These are in order based on date added.

 Warning...

Many of these terms are x-rated.  In fact, I feel like I have to go to confession every time I update this filter.  But it is better that we see these terms here than be exposed to them several times a day accompanied by graphic pictures.

L1-3b) Custom spam terms to BLOCK: Add your own spam terms to block in this area -- one per line. These blocked e-mails will go directly to the trash so be careful with what you add.

 Be Careful...

It is strongly suggested that you do NOT block  any term fewer than 5 characters anywhere in the message.  You can use spaces, and that does count as a character.  These filters search through attachments as well and very often Microsoft® Word or pdf documents will contain the "f" word in its binary code.

L1-4) Messages sent TO these addresses / domains will be BLOCKED: Add full e-mail addresses or just domains in this area -- one per line. These blocked e-mails will go directly to the trash so be careful with what you add.

L1-5) Messages sent FROM these addresses / domains will be BLOCKED: Add full e-mail addresses or just domains in this area -- one per line. These blocked e-mails will go directly to the trash so be sure it is spam you are blocking.

L1-6) Messages encoded with base64 for no good reason:  Spammers encode their messages often so filters like these cannot detect common spam patters and terms.  However, we are one step ahead!  If a mesage is either plain text or HTML, there is no reason to encode it whatsoever, therefore seen as spam and deleted.  This rule can be deselected if for some reason your valid users like to encode their e-mail.

After you make your entries, click the "customize level 1 rules" button to save your changes.

C) Customize Level 2.  By default, level 2 (and all other levels) are activated as indicated by the checkbox. Click the "Customize Level 2 Spam Protection" link in your Appies.

The screen will refresh and expand allowing you to customize your autoresponse, with rules L2-1 through L2-2 where you can remove terms from the default lists as well as add your own.

Customize Autoresponse) From / reply-to address of autoresponse: Enter a fully qualified e-mail address that your autoresponder will have as its "From" and "Reply-to" address. You do NOT want to enter an e-mail address here that is actually delivered!  Incase spammers do have some kind of program setup that grabs reply addresses, you can just be giving them another address.  You do want to use an alias that goes to trash -- like "nobody" at your domain.  Do make sure the domain is setup in your aliases to go to /dev/null.

Customize Autoresponse) Autoresponse to spam: The message that will be sent back to the reported senders of e-mail that is flagged as level 2 spam. It is suggested you include a spam free e-mail address in this email so legitimate senders who read this can resend you their message. Plain text only -- no HTML.

 Technical Note...

"But I don't want spammers to have a spam free address of mine!" Of course not.  There are few things that are virtual certainties in this life and one of them is that spammers do not read autoresponses to their spam.  In fact, it is almost certain that they do not even use a valid reply-to address to begin with.  So only users who send you a valid e-mail that gets flagged by the level 2 filters will actually get the autoresponse.

By default, the following message is used:

This message was NOT received -- it was flagged as spam and rejected by our mail system.  If this is NOT spam, please resend your message to "spamfree" at [your domain].

Remember to make sure a) the alias "spamfree" is setup in Appies and b) this alias is in your whitelist.  If it is not, you will not get the resent legitimate message.

L2-1) Default subjects being BLOCKED: These are the default blocked subjects updated and maintained by BDC. You can use your control key while selecting multiple entries that you do NOT want blocked -- those NOT selected are automatically blocked. Note: REGEX (regular expressions) used for matching. These are in order based on date added.

L2-1b) Custom subjects to BLOCK: Add your own subjects to block in this area -- one per line. These blocked e-mails will be rejected so be careful with what you enter.

L2-2) Default spam terms being BLOCKED: These are the default blocked spam terms updated and maintained by BDC. You can use your control key while selecting multiple entries that you do NOT want blocked -- those NOT selected are automatically blocked. Note: REGEX (regular expressions) used for matching. These are in order based on date added.

L2-2b) Custom spam terms to BLOCK: Add your own spam terms to block in this area -- one per line. These blocked e-mails will be rejected so be careful with what you enter.

After you make your entries, click the "customize level 2 rules" button to save your changes.

D) Customize Level 3.  By default, level 3 (and all other levels) are activated as indicated by the checkbox. Click the "Customize Level 3 Spam Protection" link in your Appies.

"Possible Spam" Forwarding Address) Forwarding E-mail / Account: Enter a fully qualified e-mail address that messages flagged as level 3 spam will be sent to. If "default forwarding address" is selected in the pulldown menu, the forwarding address entered will be used for all level 3 spam, for all accounts. Otherwise, you can set individual forwarding addresses for each account using spam filters.

To explain this further, remember that level 3 spam is a) modified to include two headers that will allow e-mail clients to detect it as possible spam and b) forwarded to any e-mail address.  Now, because this e-mail is actually forwarded and not trashed or rejected, it is important that it be delivered to the correct recipient.  This is why, you have the option of setting a specific forwarding address for each POP account.  Sometimes, users will have multiple POP accounts and they will want all their possible spam to go to one "spam" account.  This is where the default address comes in handy.

The forwarding address does not have to be a POP account -- it can be any valid e-mail address.  It does not even have to be a whitelisted address, because one of an "X-loop" header is added to the e-mail preventing the filter from catching a second time.

L3-1) Default subjects being BLOCKED: These are the default blocked subjects updated and maintained by BDC. You can use your control key while selecting multiple entries that you do NOT want blocked -- those NOT selected are automatically blocked. Note: REGEX (regular expressions) used for matching. These are in order based on date added.

L3-1b) Custom subjects to BLOCK: Add your own subjects to block in this area -- one per line.

L3-2) Default spam terms being BLOCKED: These are the default blocked spam terms updated and maintained by BDC. You can use your control key while selecting multiple entries that you do NOT want blocked -- those NOT selected are automatically blocked. Note: REGEX (regular expressions) used for matching. These are in order based on date added.

L3-2b) Custom spam terms to BLOCK: Add your own spam terms to block in this area -- one per line.

 Helpful Hint...

It is suggested that if you will be entering new subjects or terms to block anywhere in the e-mail, try them out with level 3 filters first.  If you mess up by entering too few characters, or enter incorrect REGEX (if you choose that option), the message will still be delivered.  Once you are confident your rule is successful, move it to level 2 or level 1.

After you make your entries, click the "customize level 3 rules" button to save your changes.

Setting Up Your E-mail Client to Filter Level 3 Spam

When spam is flagged as level 3 spam, it is forwarded to the forwarding address specified in the level 3 customization section.  Now when the filters detect this spam, the headers of the message are modified to include the following:

X-Spam: possible spam
X-Loop: infobot_reply
X-Match: [specific term/entry matched that flagged it as level 3 spam]

All e-mail clients are different, but most have the ability to filter messages based on headers.  Here is what you do with Microsoft® Outlook™.  From these instructions you should be able to figure out how to do this with any e-mail client, or Google "[mail client name] filter setup".

1) First create a new folder under your inbox called "possible spam" (or any other name).  To do this, in your folder list, right click on "inbox" and select "create new folder". 

2) Under "Tools" select "Rules Wizard"

3) click the "New" button to create a new rule

4) for type of rule to create, select "check messages when they arrive" click "Next"

5) check "with specific words in the network header"

6) click on the "specific words" link and enter "X-Spam: possible spam" exactly and click "Next"

7) check "move it to the specified folder"

8) click the "specified folder" link and select your folder that you created in step 1, then click "Finish"

 Awesome Feature...

The "X-Match" header is for your use to help you refine your filters.  To see this, you must view headers in your e-mail message.  In Outlook™, right click the message and select "options".

 

[ Table of Contents ]
Copyright 2004-2007, Greentree Hosting, LLC.