| Note on the Dates of these
Tests:
The testing reported on in these pages was conducted two years
ago -- October, 2004. All of the applications tested have been
replaced by newer versions. Moreover, the adware, spyware, and
malware landscape has changed dramatically since the time period of
these tests. Thus, these tests should not be relied upon to give an
accurate picture of the current detection and removal capabilities of the
applications tested.
Since these tests were conducted I have become an employee of Sunbelt
Software, the makers of the CounterSpy
anti-spyware application, and this affiliation precludes me from
conducting any further anti-spyware testing for public consumption.
I continue to take an interest in anti-spyware testing, and can
recommend the following links for those wanting more information:
- Some
Thoughts on Anti-Spyware Testing -- a short post by me
on the state of anti-spyware testing and
what constitutes quality anti-spyware testing. Links to relevant
papers, sites, & organizations included.
- Malware-Test.com's
Anti-Spyware Test Reports -- recent anti-spyware testing
that strongly resembles
the testing I did in Oct. 2004. For my thoughts on this group's
testing, see HERE.
- Catalogs of Anti-Spyware Tests: #1,
#2
-- two lists of comprehensive anti-spyware tests &
reviews, from
ConsumerResearch.com & FirewallGuide.com.
by Eric L. Howes (5 Aug.
2006)
|
Overview
As the the threat of "spyware" and "adware" has
escalated over the past few years, the number of "anti-spyware"
scanners available on the Net has grown equally fast. At present there are over 100 anti-spyware
scanners available for download -- some for
free, some for pay. Spyware and adware are themselves complex enough to
prove bewildering to most average users, however. So confusing in fact is
the threat of spyware and adware that users often have trouble
distinguishing effective anti-spyware scanners from less effective ones.
Although a number of "tests" of anti-spyware scanners have been
reported on the Net, many if not most of those tests are of limited value
because the design, methodology, and execution of the tests is not fully
and publicly documented,
leaving even experienced users and experts to wonder just how meaningful
those tests really are. Still worse, some of those "tests" are touted by
webmasters who are affiliates for the companies whose products were
"tested."
The tests documented on these pages are intended to partially remedy these
several problems with our knowledge of anti-spyware scanners and how well
they perform. At present, there are three groups of tests documented here.
Users looking for a short list of recommendations for anti-spyware
products can find such a list HERE.
For a more comprehensive list of anti-spyware products, see HERE.
For a comparative breakdown of features available in the more reputable
anti-spyware programs, see HERE.
And if your PC is already overrun with spyware or adware, see my tips for
what to do HERE.
The Tests: Summary &
Description
Three rounds of tests have been conducted. The results for each round
are reported on two "test results" pages. A table summarizing
the applications tested can be found below.
Tables summarizing the "critical detections" identified for each
round of tests are found here.
Group 1 (Oct. 3-4)
In the first group of tests, twenty anti-spyware scanners
were pitted against a collection of 15 adware and spyware programs that
were installed with the
latest version of Grokster
available from CNET's Download.com.
The spyware and adware installed with Grokster were documented and then
broken down into 134 "critical" detections, which included a mix
of files, processes, and Registry entries (see this table for details). Each anti-spyware scanner was then allowed to scan and remove every
instance of spyware or adware that it could find. The results of each
anti-spyware scanner's performance in finding and removing the 134
critical detections are
reported on separate "results" pages:
For an interesting and illuminating analysis of the Grokster
installation, see Ben Edelman's excellent write-up HERE.
Group 2 (Oct. 8-9)
In the second group of tests, the anti-spyware scanners were matched
against a mish-mash of 25 different adware and spyware programs picked up
via "drive-by-download" at the Innovators
of Wrestling web site. Once again the installed adware and spyware was
broken down into "critical detections," this time numbering 153
(again, see this table
for a summary).
The anti-spyware scanners were then allowed to find and remove spyware and
adware. As before, the performance of the 20 anti-spyware programs is
reported on two "results" pages:
Two substitutions were made in the anti-spyware scanners used for the
second group or round of tests. First, SpyBouncer was substituted for
SpywareNuker 2004. SpywareNuker 2004 requires users to activate the
product online. As SpywareNuker 2004 was uninstalled after the first round
of tests, it had to be reactivated when it was reinstalled. TrekBlue's
server refused to activate, indicating that the registration number had
already been used to activate a copy of SpywareNuker 2004. At that point,
SpyBouncer was substituted for SpywareNuker 2004 for the second round of
tests.
Second, as BPS Spyware & Adware Remover crashed at the beginning of
removals during the test, Tenebril SpyCatcher was tested on Oct. 15 (a
week later than the other applications for this round) and substituted for
BPS Spyware & Adware Remover.
Group 3 (Oct. 13-15)
In the third group of tests, the anti-spyware scanners were pitted
against yet another hodge podge collection of adware and spyware programs.
These 23 different programs were picked up by surfing 3 web sites in
succession (007 Arcade
Games Games, LyricsDomain,
and Innovators
of Wrestling ). As before, the installed spyware and adware was
broken down into "critical detections," 138 total for this third
round (see this summary table
for a breakdown). The
anti-spyware scanners were then unleashed on the PC to find and remove
whatever spyware and adware they could. Their performance is reported on
two "results" pages:
One substitution was made in the anti-spyware scanners used for the
third group of tests. As ZeroSpyware 2004 froze at the outset of removals
during the test, Tenebril SpyCatcher was tested instead and substituted
for ZeroSpyware 2004.
Notes
Before moving to the test results pages, please read the information
below about the tests themselves, esp. the Disclaimers
section.
PC Pitstop publishes a "Top
25 Spyware and Adware" list, which is updated regularly. The
three tests documented here include all of the top 10 spyware/adware
applications on the PC Pitstop list (as of Oct. 18, 2004), and a good number of the
remaining 15 in the top 25.
Users looking for a short list of recommendations for anti-spyware
products can find such a list HERE.
For a more comprehensive list of anti-spyware products, see HERE.
For a comparative breakdown of features available in the more reputable
anti-spyware programs, see HERE.
And if your PC is already overrun with spyware or adware, see my tips for
what to do HERE.
The Tests: Design &
Methodology
The same testing process was used for both rounds of tests.
Installation
Before testing, all "anti-malware" protections were disabled,
including all resident "anti-malware" scanners, spyware
"immunizations," custom browser security settings, and other
system configurations designed to block the installation or execution of
"malware." The spyware and adware was then installed from the
internet.
- For the first round of tests (reported on "results" pages
1 and 2) Grokster version 2.6 was installed from
Download.com. In addition to installing the main P2P file sharing
application, the stub downloader/installer (grokstersetup.exe) itself
downloaded and executed a number of other installers for other
applications.
- For the second round of tests (reported on "results" pages
3 and 4) Internet Explorer was pointed to iowrestling.com, where a
flurry of ActiveX Warning boxes was encountered for automated
installations of spyware and adware. No less than 7 different boxes
were clicked through, initiating installation processes for around 25
different adware and spyware applications.
- For the third round of tests (reported on "results" pages 5
and 6)
Internet Explorer was taken to three web sites in succession, all of
which popped up ActiveX Warning boxes for automated installations of
spyware and adware. Although only 5 boxes were clicked through, 23
different adware and spyware programs were installed on the test PC as
a result.
After all significant hard drive and network activity had
ceased, the PC was rebooted to allow the various installers to finish
setup activity. Once that activity had completed and the installed software
components were in a relatively "stable" state, the
personal firewall installed on the computer was configured to block all
network traffic to prevent further installations or changes. An InCtrl5
installation log was generated as well as a preliminary HijackThis!
log.
"Critical" Detections
From those logs as well as from information gleaned by manual
inspection of the hard drive and Registry, a list of
"critical" detections was generated, with each detection being
assigned a unique ID (see this
page for details). Included in these "critical" detections
were:
- executable files (.EXE / .COM)
- dynamic link libraries (.DLL)
- BHO-related Registry entries
- toolbar-related Registry entries
- browser setting-related Registry entries
- browser extension-related Registry entries
- auto-start Registry entries
These "critical" detections comprise only a subset of the
complete collection of files and Registry entries added to the test PC by
the installed spyware and adware. As such, the test results reported
here do not provide a complete picture of the performance of the
anti-spyware applications tested.
Nonetheless, these detections are
"critical" because they constitute the most important files and
Registry entries installed by the spyware and adware applications that
accompanied Grokster. These detections represent the changes that would
most visible and/or important to users. Any good anti-spyware application
would necessarily have to succeed at detecting and removing a significant
number of these files
and Registry entries in order to be considered useful or effective, even if it left a
significant number of less important files and Registry keys -- that is to
say, inert "junk" -- behind.
Moreover, these "critical" detections do provide a useful
measure of the performance of these anti-spyware applications because they
test how well the programs:
- find and remove files on the hard drive
- kill running processes and remove the associated files
- correctly uninstall BHOs, browser toolbars, and other browser
extensions
- find and remove Registry entries critical to the functioning of the
spyware and adware applications
One significant aspect of these applications that was tested only in
the third round
of tests, however, was how well
the applications remove Winsock LSP hijacks (if removed incorrectly, the
network connection of the PC may be broken).
It should also be noted that not all applications installed by the Grokster setup program
are
represented in the detections for the first group of tests. Not included are:
- Grokster
- P2P Networking
- Flashtalk
By contrast, all programs installed by iowrestling.com are
represented in the detections for the second group of tests. The same
holds true for the programs installed during the third group of tests.
Along with the list of "critical" detections, a full
Registry backup and copy of all newly installed or changed files was
archived. This Registry backup, combined with the archived files, was used
to restore the test PC to a "newly installed" state before each
anti-spyware scanner test.
Scanning & Removal
After the test PC had been restored to a "newly installed"
state, each anti-spyware application was allowed to scan and remove every
instance of spyware and adware that it could find. Where possible, each
scanner was configured to scan only the C-drive and the L-drive (containing the
Temporary Internet Files directory and main TEMP directory) on the test
PC. Each scanner was also configured to perform a "full" or
"deep" scan of
the Registry. If the anti-spyware application requested a system reboot to
complete the
detection and removal process, a reboot was performed. In all cases the
latest definitions databases available for the applications were used. Scan
logs were archived when possible, though this was not always feasible.
To check the performance of each anti-spyware scanner, a custom-built
batch file was executed. This batch file generated a list of the
"critical" files and Registry entries that were not removed by
the anti-spyware scanner. In some cases anti-spyware scanners may have
detected and attempted to remove certain files and Registry entries only
to fail. As the batch file checked for "critical" detections
actually left in place at the conclusion of a scan, the test results
reported here reflect only actual removals, not mere
detections or attempted removals. Finally, false positives were noted and
reported when they were generated.
Readers should be aware that in some cases anti-spyware applications may not have removed
the files and Registry entries for particular adware or spyware
programs because of deliberate policy decisions by the vendors not to
target those programs for removal.
Test PC
The PC used for these tests was a 1.8 Ghz Pentium 4 w/ 512 mb RAM.
Installed on the computer were Windows 2000 w/ SP4, Internet Explorer w/
SP1, and Office 2000 w/ SP3. The network connection was provided by InsightBB's
cable broadband service. The network connection was monitored by Agnitum
Outpost Firewall Pro 2.1.
Disclaimers
Although the test results reported on these pages are detailed, readers
should be aware of several significant limitations of the tests
performed:
- The test results reported here constitute but a few tests with three
collections of spyware and adware programs. The anti-spyware
scanners tested here may perform differently with other collections of
spyware and adware.
- The tests results report only actual removals of a select number
of "critical" files and Registry keys, and thus do not give
a complete account of the removals performed by any of the
anti-spyware scanners tested.
- These tests do not pit the anti-spyware scanners against what is
undoubtedly the toughest spyware application of them all, CoolWebSearch.
Given these limitations, readers should not regard the test results
reported here as any kind of "definitive" guide to anti-spyware
scanners, nor should readers use these test results as the sole basis for
purchasing decisions. The information presented on these pages is designed
to supplement other information about anti-spyware applications found on
the Net, not completely replace it.
Moreover, nothing in these test results should be taken as an
endorsement of or recommendation against the use of any particular
anti-spyware scanner by the author of these web pages. These tests are
primarily intended to help users gain better insight into the issues
surrounding anti-spyware scanners and the kind of performance that might
be expected from them.
Finally, it should be noted that I have no financial relationship with any of the companies
or individuals whose products were tested. I am not an employee, affiliate, representative, or other agent of any of these
companies or individuals.
Lessons & Conclusions
If any lessons or conclusions can be drawn from these tests at all, they are quite
general:
- Spyware and adware can prove quite difficult to remove, even for
dedicated anti-spyware scanners.
In the second and third group of
tests, for example, one of the installed programs prevented the anti-spyware
scanners from running on reboot, a common method used by
anti-spyware scanners to remove stubborn spyware and adware that is
currently in memory on a PC. As a result, some spyware and adware was
not removed by the anti-spyware scanners during reboot that otherwise might have.
- No single anti-spyware scanner removes everything. (1)
Even the
best-performing anti-spyware scanner in these tests missed fully one
quarter of the "critical" files and Registry entries.
- It is better to use two or more anti-spyware scanners in
combination, as one will often detect and remove things that others do
not.
- Where possible, users should become familiar with the use of
HijackThis! in order to remove stubborn spyware and adware that
standard anti-spyware scanners fail to remove. Less experienced users should know how to get help from
the expert volunteers who provide free HijackThis! log advice and
analysis at major
anti-spyware forums.
- Prevention is always preferable to scanning and removal, and users should securely
configure their PCs and install anti-malware
protection to prevent the installation of spyware and adware in
the first place.
- Moreover, users should learn to practice safe computing habits,
which include avoiding web sites and programs of unknown or dubious
provenance and carefully reading End User License Agreements and
Privacy Policies.
Users looking for a short list of recommendations for anti-spyware
products can find such a list HERE.
For a more comprehensive list of anti-spyware products, see HERE.
For a comparative breakdown of features available in the more reputable
anti-spyware programs, see HERE.
And if your PC is already overrun with spyware or adware, see my tips for
what to do HERE.
|
