Testing Guide

Introduction

This guide provides you with tips on testing ORF in a controlled environment. Such testing allows getting an overall impression of ORF and its email filtering capabilities without actual risks.

---

Overview

Deployment for testing

ORF specializes in real-time email filtering. Due to this, it is recommended to deploy ORF to its "live" location on the network (i.e. to the server it would normally run). Consult the Deployment Guide on choosing this location.

Deploying ORF on a computer that is not involved in live email traffic will allow limited testing only. In particular, simply forwarding archived spam emails to such isolated installations will result in poor performance and will give a false impression of ORF's capabilities.

Testing methods

There are two main ways to test ORF.

• Test Mode: ORF switched into this mode will simulate normal operation, but email actions will not be carried out, i.e. emails will not be rejected or tagged. Logs, reports and statistics produced by ORF enable you to verify the operation.
• Tagging: ORF can be configured to tag spam. This tagging in turn can be used to redirect the emails to the end-user's Junk Email folder. Feedback from the end-users can help you estimate the effect of ORF on your organization. Any accidentally misclassified emails ("false positives") can be restored without administrator assistance.

The sections below detail using the above methods.

---

Method #1: Testing Using Test Mode

Overview

Test Mode enables evaluating ORF without actually altering your email traffic. In this mode, the administrator can check the ORF logs and reports for "false positives" (legitimate email classified as spam) and "false negatives" (spam missed). This gives the administration very good quality information, but sorting through the logs can also be time-consuming.

Among others, ORF logs will provide the administrator with the following information:

• Email arrival time
• Email sender and recipient address
• Email subject (see note below)
• Email status (e.g. blacklisted or accepted)
• Detailed explanation of the email status

Note Email subjects in particular help a lot with deciding on-the-spot whether a logged email is spam. This field is logged at the On Arrival filtering point only, however. Make sure to have all tests assigned to On Arrival (Filtering / Tests page in the ORF Administration Tool). This is also the default setting in ORF.

Enabling Test Mode

Enable this mode in the ORF Administration Tool on the Filtering / Actions page by setting the Enable Test-Only Mode checkbox. Make sure to save your configuration to apply the changes.

---

Method #2: Testing Using Tagging

Overview

You can configure ORF to place a tag on junk emails (e.g. prepend the email subject with a "[SPAM]" label). This tagging can also be used to move tagged email into the end-users' Junk Email folder directly.

When using this method, the administrator requests the end-users to report:

• False negatives: Any untagged spam emails that arrive in their Inbox
• False positives: Any legitimate email that was tagged or moved to the Junk Email folder

Testing this way has the benefit of taking workload off the administrator. However, by our experience this method is severely inaccurate, because end-users are often unable or unwilling to comply and mistake legitimate newsletters for spam.

A further limitation is that the tagging action is not entirely universal in ORF: a few tests such as the Recipient Validation Test will always reject the email.

Enabling Tagging

Configure tagging using the ORF Administration Tool: select the Filtering / Actions page and click the Edit button in the On Arrival section. In the dialog, select Accept email and perform further actions and set your tag.

Redirecting email to the Junk Email folder of end-users is available from Microsoft® Exchange 2003. Please see this Knowledge Base article to learn how to set up ORF and Exchange for this redirection.

As tagging is available only at the On Arrival filtering point of ORF, make sure that all tests are assigned to On Arrival (Filtering / Tests page in the ORF Administration Tool). This is also the default setting in ORF.

Save your configuration (CTRL-S or File | Save Configuration menu) to apply the changes.

Variant: Email Redirection

ORF can also be configured to redirect blacklisted email to a specific email address (just choose redirection instead of tagging in the above steps). This allows the administrator to set up a "catch-all" mailbox and check it for false positives. Information about false negatives (spam that got through) still has to be gathered from the end-users, though.

---

Measuring Performance

Indicators

There are two major indicators of spam filtering performance, the Spam Catch Rate % and the False Positive Rate %. Calculating these require the following figures:

• Total Number of Emails (TE) – see below how to get this number
• Total Number of Emails Blacklisted (TB) – see below how to get this number
• Number of False Negatives (FN)
• Number of False Positives (FP)

The Spam Catch Rate (SC%) tells you how much of the total spam is caught by ORF. For instance, if this rate is 99%, ORF misses only every 100th spam. The higher this rate the better.

SC% = 100 - (100 / (TB - FP) * FN)

The False Positive Rate (FP%) tells you how much of your emails are misclassified by ORF as spam (percentage of legitimate email misclassified). For instance, if this rate is 0.01%, ORF is will misclassify only every 10,000th incoming email as spam. The lower this rate the better.

FP% = 100 / TE * FN

Use the ORF Reporting Tool or the Log Viewer to determine the Total Number of Emails (TE) and Total Number of Emails Blacklisted (TB) numbers.

Using the Reporting Tool

Generate a report during your evaluation period to get detailed statistics on the performance of ORF and its tests. Note two figures:

• Approximate number of emails checked = TE
• Approximate number of emails blacklisted = TB

The Reporting Tool will not generate a report for the current day, because the data for the current day is not final yet. If you need data for the current day, check the next section.

Using the Log Viewer

The filtering feature of ORF will help you to determine the two figures you need. Filters can be created using the View | Filter menu. The number of events shown equals the number of emails — this information can be found on the status bar (e.g. 1000 from "1000 of 2000 events shown").

• Total number of emails blacklisted (TB): Define an Event Class filter for the Blacklist value.
• Total number of emails (TE): First define an Event Class rule for the following event classes: Pass, Whitelist and Blacklist and add a Filtering Point rule for the On Arrival filtering point. Note the resulting number. Clear your filter and start a new one with an Event Class rule for Blacklist and a Filtering Point rule for Before Arrival. Add the resulting number to the noted number. The sum of the two numbers is TE.

Evaluating Results

In case the Spam Catch Rate % is less than 95% or if the False Positive Rate % is more than 0.001%, consult our Best Practices Guide for fine-tuning ORF or contact our Customer Service for assistance.

Note When calculating the spam catch rate, please consider that ORF does not test whitelisted emails. That is, if a spam email was whitelisted by ORF, that indicates a configuration problem and the email should not be considered a false negative (i.e., spam which was missed by ORF), as the email has never been actually checked by blacklist tests.

Virus-infected emails which passed filtering should not be considered false negatives either, as ORF is designed to filter spam emails and not viruses (you will need a separate virus filtering software installed for the latter).