The Spyware Warrior Guide to

Anti-Spyware Testing

by Eric L. Howes

Oct. 2-4, 2004 /
Oct. 8-9, 2004 /
Oct. 13-15, 2004 
 

Note on the Dates of these Tests: 

The testing reported on in these pages was conducted two years ago -- October, 2004. All of the applications tested have been replaced by newer versions. Moreover, the adware, spyware, and malware landscape has changed dramatically since the time period of these tests. Thus, these tests should not be relied upon to give an accurate picture of the current detection and removal capabilities of the applications tested.

Since these tests were conducted I have become an employee of Sunbelt Software, the makers of the CounterSpy anti-spyware application, and this affiliation precludes me from conducting any further anti-spyware testing for public consumption.

I continue to take an interest in anti-spyware testing, and can recommend the following links for those wanting more information:

  • Some Thoughts on Anti-Spyware Testing -- a short post by me on the state of anti-spyware testing and 
    what constitutes quality anti-spyware testing. Links to relevant papers, sites, & organizations included.
  • Malware-Test.com's Anti-Spyware Test Reports -- recent anti-spyware testing that strongly resembles 
    the testing I did in Oct. 2004. For my thoughts on this group's testing, see HERE.
  • Catalogs of Anti-Spyware Tests: #1, #2 --  two lists of comprehensive anti-spyware tests & reviews, from 
    ConsumerResearch.com & FirewallGuide.com.

by Eric L. Howes (5 Aug. 2006)

Overview

As the the threat of "spyware" and "adware" has escalated over the past few years, the number of "anti-spyware" scanners available on the Net has grown equally fast. At present there are over 100 anti-spyware scanners available for download -- some for free, some for pay. Spyware and adware are themselves complex enough to prove bewildering to most average users, however. So confusing in fact is the threat of spyware and adware that users often have trouble distinguishing effective anti-spyware scanners from less effective ones. Although a number of "tests" of anti-spyware scanners have been reported on the Net, many if not most of those tests are of limited value because the design, methodology, and execution of the tests is not fully and publicly documented, leaving even experienced users and experts to wonder just how meaningful those tests really are. Still worse, some of those "tests" are touted by webmasters who are affiliates for the companies whose products were "tested."

The tests documented on these pages are intended to partially remedy these several problems with our knowledge of anti-spyware scanners and how well they perform. At present, there are three groups of tests documented here.

Users looking for a short list of recommendations for anti-spyware products can find such a list HERE. For a more comprehensive list of anti-spyware products, see HERE. For a comparative breakdown of features available in the more reputable anti-spyware programs, see HERE. And if your PC is already overrun with spyware or adware, see my tips for what to do HERE.

The Tests: Summary & Description

Three rounds of tests have been conducted. The results for each round are reported on two "test results" pages. A table summarizing the applications tested can be found below. Tables summarizing the "critical detections" identified for each round of tests are found here.

Group 1 (Oct. 3-4)

In the first group of tests, twenty anti-spyware scanners were pitted against a collection of 15 adware and spyware programs that were installed with the latest version of Grokster available from CNET's Download.com. The spyware and adware installed with Grokster were documented and then broken down into 134 "critical" detections, which included a mix of files, processes, and Registry entries (see this table for details). Each anti-spyware scanner was then allowed to scan and remove every instance of spyware or adware that it could find. The results of each anti-spyware scanner's performance in finding and removing the 134 critical detections are reported on separate "results" pages:

For an interesting and illuminating analysis of the Grokster installation, see Ben Edelman's excellent write-up HERE.

Group 2 (Oct. 8-9)

In the second group of tests, the anti-spyware scanners were matched against a mish-mash of 25 different adware and spyware programs picked up via "drive-by-download" at the Innovators of Wrestling web site. Once again the installed adware and spyware was broken down into "critical detections," this time numbering 153 (again, see this table for a summary). The anti-spyware scanners were then allowed to find and remove spyware and adware. As before, the performance of the 20 anti-spyware programs is reported on two "results" pages:

Two substitutions were made in the anti-spyware scanners used for the second group or round of tests. First, SpyBouncer was substituted for SpywareNuker 2004. SpywareNuker 2004 requires users to activate the product online. As SpywareNuker 2004 was uninstalled after the first round of tests, it had to be reactivated when it was reinstalled. TrekBlue's server refused to activate, indicating that the registration number had already been used to activate a copy of SpywareNuker 2004. At that point, SpyBouncer was substituted for SpywareNuker 2004 for the second round of tests.

Second, as BPS Spyware & Adware Remover crashed at the beginning of removals during the test, Tenebril SpyCatcher was tested on Oct. 15 (a week later than the other applications for this round) and substituted for BPS Spyware & Adware Remover.

Group 3 (Oct. 13-15)

In the third group of tests, the anti-spyware scanners were pitted against yet another hodge podge collection of adware and spyware programs. These 23 different programs were picked up by surfing 3 web sites in succession  (007 Arcade Games Games, LyricsDomain, and Innovators of Wrestling ). As before, the installed spyware and adware was broken down into "critical detections," 138 total for this third round (see this summary table for a breakdown). The anti-spyware scanners were then unleashed on the PC to find and remove whatever spyware and adware they could. Their performance is reported on two "results" pages:

One substitution was made in the anti-spyware scanners used for the third group of tests. As ZeroSpyware 2004 froze at the outset of removals during the test, Tenebril SpyCatcher was tested instead and substituted for ZeroSpyware 2004.

Notes

Before moving to the test results pages, please read the information below about the tests themselves, esp. the Disclaimers section.

PC Pitstop publishes a "Top 25 Spyware and Adware" list, which is updated regularly. The three tests documented here include all of the top 10 spyware/adware applications on the PC Pitstop list (as of Oct. 18, 2004), and a good number of the remaining 15 in the top 25.

Users looking for a short list of recommendations for anti-spyware products can find such a list HERE. For a more comprehensive list of anti-spyware products, see HERE. For a comparative breakdown of features available in the more reputable anti-spyware programs, see HERE. And if your PC is already overrun with spyware or adware, see my tips for what to do HERE.

The Tests: Design & Methodology

The same testing process was used for both rounds of tests. 

Installation

Before testing, all "anti-malware" protections were disabled, including all resident "anti-malware" scanners, spyware "immunizations," custom browser security settings, and other system configurations designed to block the installation or execution of "malware." The spyware and adware was then installed from the internet.

  • For the first round of tests (reported on "results" pages 1 and 2) Grokster version 2.6 was installed from Download.com. In addition to installing the main P2P file sharing application, the stub downloader/installer (grokstersetup.exe) itself downloaded and executed a number of other installers for other applications. 
     
  • For the second round of tests (reported on "results" pages 3 and 4) Internet Explorer was pointed to iowrestling.com, where a flurry of ActiveX Warning boxes was encountered for automated installations of spyware and adware. No less than 7 different boxes were clicked through, initiating installation processes for around 25 different adware and spyware applications.
     
  • For the third round of tests (reported on "results" pages 5 and 6) Internet Explorer was taken to three web sites in succession, all of which popped up ActiveX Warning boxes for automated installations of spyware and adware. Although only 5 boxes were clicked through, 23 different adware and spyware programs were installed on the test PC as a result.

After all significant hard drive and network activity had ceased, the PC was rebooted to allow the various installers to finish setup activity. Once that activity had completed and the installed software components were in a relatively "stable" state, the personal firewall installed on the computer was configured to block all network traffic to prevent further installations or changes. An InCtrl5 installation log was generated as well as a preliminary HijackThis! log. 

"Critical" Detections

From those logs as well as from information gleaned by manual inspection of the hard drive and Registry, a list of "critical" detections was generated, with each detection being assigned a unique ID (see this page for details). Included in these "critical" detections were:

  • executable files (.EXE / .COM)
  • dynamic link libraries (.DLL)
  • BHO-related Registry entries
  • toolbar-related Registry entries
  • browser setting-related Registry entries
  • browser extension-related Registry entries
  • auto-start Registry entries

These "critical" detections comprise only a subset of the complete collection of files and Registry entries added to the test PC by the installed spyware and adware. As such, the test results reported here do not provide a complete picture of the performance of the anti-spyware applications tested. 

Nonetheless, these detections are "critical" because they constitute the most important files and Registry entries installed by the spyware and adware applications that accompanied Grokster. These detections represent the changes that would most visible and/or important to users. Any good anti-spyware application would necessarily have to succeed at detecting and removing a significant number of these files and Registry entries in order to be considered useful or effective, even if it left a significant number of less important files and Registry keys -- that is to say, inert "junk" -- behind. 

Moreover, these "critical" detections do provide a useful measure of the performance of these anti-spyware applications because they test how well the programs:

  • find and remove files on the hard drive
  • kill running processes and remove the associated files
  • correctly uninstall BHOs, browser toolbars, and other browser extensions
  • find and remove Registry entries critical to the functioning of the spyware and adware applications

One significant aspect of these applications that was tested only in the third round of tests, however, was how well the applications remove Winsock LSP hijacks (if removed incorrectly, the network connection of the PC may be broken).

It should also be noted that not all applications installed by the Grokster setup program are represented in the detections for the first group of tests. Not included are:

  • Grokster
  • P2P Networking
  • Flashtalk

By contrast, all programs installed by iowrestling.com are represented in the detections for the second group of tests. The same holds true for the programs installed during the third group of tests.

Along with the list of "critical" detections, a full Registry backup and copy of all newly installed or changed files was archived. This Registry backup, combined with the archived files, was used to restore the test PC to a "newly installed" state before each anti-spyware scanner test.

Scanning & Removal

After the test PC had been restored to a "newly installed" state, each anti-spyware application was allowed to scan and remove every instance of spyware and adware that it could find. Where possible, each scanner was configured to scan only the C-drive and the L-drive (containing the Temporary Internet Files directory and main TEMP directory) on the test PC. Each scanner was also configured to perform a "full" or "deep" scan of the Registry. If the anti-spyware application requested a system reboot to complete the detection and removal process, a reboot was performed. In all cases the latest definitions databases available for the applications were used. Scan logs were archived when possible, though this was not always feasible.

To check the performance of each anti-spyware scanner, a custom-built batch file was executed. This batch file generated a list of the "critical" files and Registry entries that were not removed by the anti-spyware scanner. In some cases anti-spyware scanners may have detected and attempted to remove certain files and Registry entries only to fail. As the batch file checked for "critical" detections actually left in place at the conclusion of a scan, the test results reported here reflect only actual removals, not mere detections or attempted removals. Finally, false positives were noted and reported when they were generated.

Readers should be aware that in some cases anti-spyware applications may not have removed the files and Registry entries for particular adware or spyware programs because of deliberate policy decisions by the vendors not to target those programs for removal.

Test PC

The PC used for these tests was a 1.8 Ghz Pentium 4 w/ 512 mb RAM. Installed on the computer were Windows 2000 w/ SP4, Internet Explorer w/ SP1, and Office 2000 w/ SP3. The network connection was provided by InsightBB's cable broadband service. The network connection was monitored by Agnitum Outpost Firewall Pro 2.1.

Disclaimers

Although the test results reported on these pages are detailed, readers should be aware of several significant limitations of the tests performed:

  • The test results reported here constitute but a few tests with three collections of spyware and adware programs. The anti-spyware scanners tested here may perform differently with other collections of spyware and adware.
     
  • The tests results report only actual removals of a select number of "critical" files and Registry keys, and thus do not give a complete account of the removals performed by any of the anti-spyware scanners tested.
     
  • These tests do not pit the anti-spyware scanners against what is undoubtedly the toughest spyware application of them all, CoolWebSearch.

Given these limitations, readers should not regard the test results reported here as any kind of "definitive" guide to anti-spyware scanners, nor should readers use these test results as the sole basis for purchasing decisions. The information presented on these pages is designed to supplement other information about anti-spyware applications found on the Net, not completely replace it. 

Moreover, nothing in these test results should be taken as an endorsement of or recommendation against the use of any particular anti-spyware scanner by the author of these web pages. These tests are primarily intended to help users gain better insight into the issues surrounding anti-spyware scanners and the kind of performance that might be expected from them.

Finally, it should be noted that I have no financial relationship with any of the companies or individuals whose products were tested. I am not an employee, affiliate, representative, or other agent of any of these companies or individuals. 

Lessons & Conclusions

If any lessons or conclusions can be drawn from these tests at all, they are quite general:

  • Spyware and adware can prove quite difficult to remove, even for dedicated anti-spyware scanners. 
     
    In the second and third group of tests, for example, one of the installed programs prevented the anti-spyware scanners from running on reboot, a common method used by anti-spyware scanners to remove stubborn spyware and adware that is currently in memory on a PC. As a result, some spyware and adware was not removed by the anti-spyware scanners during reboot that otherwise might have.
     
  • No single anti-spyware scanner removes everything. (1) Even the best-performing anti-spyware scanner in these tests missed fully one quarter of the "critical" files and Registry entries.
     
  • It is better to use two or more anti-spyware scanners in combination, as one will often detect and remove things that others do not.
     
  • Where possible, users should become familiar with the use of HijackThis! in order to remove stubborn spyware and adware that standard anti-spyware scanners fail to remove. Less experienced users should know how to get help from the expert volunteers who provide free HijackThis! log advice and analysis at major anti-spyware forums.
     
  • Prevention is always preferable to scanning and removal, and users should securely configure their PCs and install anti-malware protection to prevent the installation of spyware and adware in the first place. 
     
  • Moreover, users should learn to practice safe computing habits, which include avoiding web sites and programs of unknown or dubious provenance and carefully reading End User License Agreements and Privacy Policies.

Users looking for a short list of recommendations for anti-spyware products can find such a list HERE. For a more comprehensive list of anti-spyware products, see HERE. For a comparative breakdown of features available in the more reputable anti-spyware programs, see HERE. And if your PC is already overrun with spyware or adware, see my tips for what to do HERE.
 

Return to top...

Short Table of Applications & Tests
 
As explained above, three rounds of tests have been conducted. The results for those three rounds of tests are reported on six different "test results" pages (two pages for each round). Each "test results" reports the performance of ten anti-spyware applications.

Given that readers might find the various "test results" pages a bit confusing, included below is a table summarizing the applications tested and where the results for each can be found.

Application / Round (Date) 1: Oct 2-4 2: Oct 8-9 3: Oct 13-15

...found on Test Results page

#1 #2 #3 #4 #5 #6
Aluria Spyware Eliminator 3.0.32 X   X   X  
BPS Spyware & Adware Remover 8.2.0.10   X       X
GIANT AntiSpyware 1.0 X   X   X  
Intermute SpySubtract Pro 2.51 X   X   X  
Lavasoft Ad-aware SE Personal 1.05 X   X   X  
McAfee AntiSpyware 1.00.1126 X   X   X  
NoAdware 2.01   X   X   X
OmniQuad AntiSpy 4.2   X   X   X
PC Tools Spyware Doctor 2.1.0.254 X   X   X  
Pest Patrol 4.4.3.24 X   X   X  
Spybot Search & Destroy 1.3.1 tx X   X   X  
SpyHunter 1.5.83   X   X   X
SpyKiller 2005 1.00   X   X   X
Spyware C.O.P. 10.0   X   X   X
SpywareNuker 2004 2.13   X        
SpywareStormer 1.4.7   X   X   X
Tenebril SpyCatcher 3.00.46       X   X
Webroot Spy Sweeper 3.2 X   X   X  
Xblock.com X-Cleaner Deluxe 4.0.0.249 X   X   X  
XoftSpy 3.45   X   X   X
ZeroSpyware 2004 1.00   X   X    

...found on Test Results page

#1 #2 #3 #4 #5 #6
Application / Round (Date) Oct 2-4 Oct 8-9 Oct 13-15

 

Return to top...

News & Other Information

These tests have received coverage and notice across the Internet. Here are a few samples of news stories and other reactions:

Background & Bio

I am currently Director of Malware Research at Sunbelt Software.

Over the past five years I have maintained a personal web site -- first at the University of Illinois; now at SpywareWarrior.com -- to supply internet users with resources to protect their privacy and security on the internet. Among those resources are several utilities and "block lists" that allow users of Microsoft's Internet Explorer web browser to protect themselves against the flood of unwanted software and content pushed on them by aggressive advertising and marketing entities. 

In June 2004 I began collaborating with Suzi of SpywareWarrior.com to create and maintain a number of pages with information on anti-spyware applications, including the "Rogue/Suspect Anti-Spyware List." I attended the FTC's Spyware Workshop (April 2004) and was a panelist at the CNET AntiSpyware Workshop (May 2005) as well as the AntiSpyware Coalition Workshop (Feb. 2006).

Prior to joining Sunbelt I was a graduate student in the Graduate School of Library and Information Science (GSLIS) at the University of Illinois at Urbana-Champaign. For twelve years I taught business and technical writing at the University of Illinois. During 2004-2005 I taught a course in GSLIS titled "Literacy in the Information Age." For three years I also taught composition courses at Parkland Community College in Champaign. 

In recognition of my work to help internet users protect their privacy and security, Microsoft awarded me its MVP (Most Valued Professional) Award (http://mvp.support.microsoft.com/).

Full disclosure: from late November 2004 to December 2005 I performed part-time consulting work as an independent contractor for Sunbelt Software, makers of CounterSpy. At the time of the testing reported on these pages (Oct. 2004), I had no relationship whatsoever with Sunbelt. At present I am a full time employee of Sunbelt Software. Because of that relationship and the conflict of interest that it represents, I must recuse myself from public comment on CounterSpy. That means that I cannot and will not publicly evaluate, test, or even recommend Sunbelt's anti-spyware product. The anti-spyware products that I do recommend, all of which are competitors to CounterSpy, are listed here. As noted above, though, the tests documented on these pages were conducted well before I had any kind of relationship with Sunbelt Software.

Questions & Contact

If you have questions or comments about any of the information presented on these pages, please don't hesitate to ask. 

Best regards,

Eric L. Howes

These pages are generously hosted by

Forums: http://spywarewarrior.com/index.php

Blog: http://www.netrn.net/spywareblog/

© Copyright 2004-2006 Eric L. Howes