Informal Trojan Detection Test # 2

 

I
N
D
E
  X  

Table of Contents

 

A Note About These Tests (2/1/03)

The anti-trojan tests found on this page and the Informal Trojan Detection Test # 1 page are now a year old. All of the applications that were tested have been updated since these tests were performed -- some several times. In many cases, new functionality has been added to those applications. Consequently, the relevance and meaningfulness of these tests is much less than it was a year ago when these tests were first posted. Please keep that in mind as you look over these tests.

Also, due to space limitations on the UIUC server which hosts these pages, the screenshots for each test result have been removed. Thus, clicking on the numbered links to view screenshots will give you a "page not found" error. If you are interested in obtaining a complete copy of the tests with the screenshots as originally posted, please contact me at eburger68@myrealbox.com . I am more than happy to supply interested parties with complete copies of the tests (probably burned onto a CD-R and delivered via snail mail).

Introduction/Overview

If you read the web page detailing my previous series of Informal Trojan Detection Tests, then you'll recall that popular anti-virus and anti-trojan applications can encounter difficulties when confronted with trojan servers that have been packed with the many executable compression programs available on the Internet. 

In response to a number of questions about how well other applications would perform under the same conditions, I've run a new round of tests. As before, I used the Sub7 2.13 MUIE server, packed several different ways (and even left completely unpacked in one case). Once again, I selected seven programs to test: four dedicated anti-trojan programs and three standard anti-virus programs. I also ran the same types of tests as before, mixing on-demand scans file scans with tests designed to check the performance of real time memory and process monitoring components. 

In essence, this is the same series of tests as performed in the original Informal Trojan Detection Tests, but with a different set of applications.

The results of these tests are detailed below. Before you jump to any hasty conclusions based on what you see here, please take the time to read the Disclaimers & Limitations section at the end of this document. 

I would also urge you read the Note On Re-testing for an explanation of the updates to this page made since its initial release.

I hope you find these tests interesting and useful.


Updates from Vendors:

Two author-vendors have released updates that address issues raised by these tests. You can get the latest updates for these programs here:

BOClean:            Latest Update File (web site)
                                 Latest Update File (forum)

TrojanHunter:   Latest TrojanHunter Build
                                  Latest Ruleset Update (web site)
                                  Latest Ruleset Update (forum)



 
Applications Tested

Seven anti-virus or dedicated anti-trojan programs were tested. Every attempt was made to get recent engine and database updates for each program. Configuration options of note are detailed for each application.

ANTS 2.1 Update 15012002 (1/12/02)
Notes: Scan for unusual signatures on
           Scan within alternate data streams
on
           Scan for clients, editors, etc. on
           Scan priority: Normal priority

           English language file for ANTS 2.1 installed


AVG Anti-Virus 6.0.336 Free Edition  Virus Database 188 (3/11/02)
Notes: Resident Shield: Check Boot Viruses, Check Executable Viruses, Check Macro Viruses, Use heuristics all on.
           Program Settings: Run memory test at program startup.
           Complete Test Settings: Testing: Internally Compressed, Archives 
                                                    Test All Extensions
                                                    Use These Methods: Integrity Check, Heuristic Analysis

BOClean 4.09 Update 03/17/2002 23:17:00
Notes: Scan automatically in background on
           Scan for trojans every 10 seconds

Initial tests done with Update 03/13/2002 23:31:28. 
Re-testing  done with
Update 03/17/2002 23:17:00.


McAfee VirusScan 5.21.1000.1 Scan Engine 4.160 / Virus definitions 4.0.4191
Notes: Scan Settings: Scan Memory, Scan boot sectors, Scan compressed files, All files
           Advanced Scan Settings: Enable heuristics (macro & program file heuristics scanning)
           VShield/System Scan: Scan files on: Run, Create, Copy, Rename
                                               What to scan: All Files, Compressed Files
                                               Advanced Scan Settings: Enable heuristics (macro & program file
                                                                                         heuristics scanning)

Norton AntiVirus 2001 (7.00.51.F) Virus definitions 3/13/02
Notes: Manual Scans: Items to scan... Memory, Boot records, Master boot records
                                    File types to scan: All files, Scan within compressed files
                                    Bloodhound Heuristics
(enabled) sensitivity level: Highest
            Auto-Protect:  Scan files when they are: Run or opened, Created or downloaded
                                    File types to scan: All files, Scan within compressed files
                                    Bloodhound Heuristics
(enabled) sensitivity level: Highest
                                    Advanced: Virus-like Activity Monitors:
all set to Ask me what to do

PestPatrol 3.0 [3/1/02] (3.2002.2.26) PestInfo.dat: 3/2/02 / PestPatrol.dat: 3/2/02 / ppmem.dat: 3/7/02
Notes: Include In Scan: All Files Including Archives
           Check For: Hacker/Security Tools, Spyware Cookies, Spyware
           Check These Directories: Temp Directories
           Check for This Spyware:
all checked
           Process Priority: Normal

TrojanHunter 2.53 build 588 Ruleset Update: 25x-2002-03-18 (updated manually)
Notes: Scan Options: Scan binded executables, Scan zip files, Scan rar files
           TH Guard: Automatically remove trojans
both checked and unchecked (see Tests #2-A & 2-B for details)
           Custom Scan List: Registry checker, Inifile checker, Prot checker, Process checker,
                                          Application start method checker, Shell executable checker, Win.ini startup entry checker 

Initial tests done with build 581 and Ruleset Update: 25x-2002-03-08.
Re-testing  done with
build 588 and Ruleset Update: 25x-2002-03-18.

Note: All programs' options were left at their defaults unless noted above or in specific tests below..

For links to all of these applications on the web, see the Links section at the end of this document.

 

Files Tested

Tests were performed with six files, each a version of Sub7 2.13 MUIE server, a common RAT (Remote Administration Trojan). Only the third file (THAT-PACKED.EXE)  was left un-manipulated. All other server files were compressed or edited as described below.

Location/FILE Description
D:\trojan-01\that\THAT.EXE Sub7 2.13 MUIE edited, but unpacked
D:\trojan-02\that-aspack\THAT-ASPACK.EXE Sub7 2.13 MUIE packed w/ ASPack 2.11d
D:\trojan-03\that-packed\THAT-PACKED.EXE Sub7 2.13 MUIE packed (as shipped)
D:\trojan-04\that-rar\THAT-ASPACK.RAR Sub7 2.13 MUIE packed w/ ASPack 2.11d, RAR'ed w/ WinRAR 2.80
D:\trojan-05\that-upx\THAT-UPX.EXE Sub7 2.13 MUIE packed w/ UPX 1.20
D:\trojan-06\that-asprot\THAT-ASPROT.EXE Sub7 2.13 MUIE edited, then packed w/ ASProtect 1.2

To compress servers, servers were first packed with one of the compression programs listed above with the programs' default options. In the case of THAT-ASPROT.EXE, the server was compressed w/ ASProtect with the following options: Max compression & Preserve extra data

The packed servers were then edited with the Sub7 2.13 MUIE EditServer.exe to append server configuration options.  Server configuration options were the defaults (startup method: WIN.INI, installation: use random name, automatically start server on 27374). In the case of THAT.EXE (the unpacked server), the configuration options were: startup method: WIN.INI & registry-Run, installation: THAT-EDIT.COM, automatically start server on 20000.

This set of trojan files is the same set that was used in Informal Trojan Detection Test # 1.

For links to all the compression programs used, see the Links section at the end of this document.

 

Tests Performed

Each application was run through a set of three tests. In each test, all six of the server test files detailed above were either scanned or launched. Thus, each application (w/ the exception of two) was tested a total of 18 times (3 main tests x 6 test files = 18 total tests). F-Prot and TDS-3 were excluded from Test # 2-B (see below for more details about the reasons). 

Test Description
Hard Drive Scan simple scan of hard drive
Memory Scan (A) monitoring started, then server loaded
Memory Scan (B) server loaded, then monitoring started

Note: No cleaning/disinfecting of any type was performed by the programs being tested (unless noted below). The programs were configured  to "report only" and/or prevent program load. Any trojan servers launched were killed from memory with PrcView and changes to auto-start locations (the Registry, WIN.INI) were reversed with MSCONFIG.EXE or System Mechanic. Finally, any trojan servers copied to C:\WINDOWS were deleted.

This set of tests is the same set that was run in Informal Trojan Detection Test # 1.

 

General Notes

Every attempt was made to regularize the system on which the tests were performed as well as the conditions under which they were performed.  The tests were conducted over two days (3/14/02 - 3/15/02). No new applications were installed during that time on the test system. Neither were any major system configuration changes made.

System Description P166MMX w/ 64bm RAM; Windows 98 SE (all updates), Internet Explorer 6.0 (all updates)., Microsoft Office 2000 (all updates).
Running Programs AtGuard 3.22.11 (iamapp.exe), Microsoft Office 2000 Toolbar (msoffice.exe), Paint (mspaint.exe), PrcView (prcview.exe), Metapad (notepad.exe). Also (occasionally): System Configuration Utility (msconfig.exe), System Mechanic (sysmechanic.exe). 

All other programs (including active monitoring/scanning programs) shut down except program being tested and default Windows programs (e.g., systray.exe, explorer.exe, et al).
Cleaning/Disinfecting No cleaning/disinfecting of any type was performed by programs being tested. Programs were configured  to "report only" and/or prevent program load.
Killing Servers PrcView was used to kill running servers whenever necessary. Any servers copied to C:\WINDOWS were manually deleted.
Editing Auto-start Locations Auto-start locations were edited and cleaned up with either MSCONFIG.EXE or System Mechanic 3.6f. 
Monitoring Connections Network connections were monitored w/ AtGuard 3.22.11 (Statistics).

This is essentially the same system setup as was used in Informal Trojan Detection Test # 1, with these key differences:

 

Test Results Key

In the "Detect?" row of the test results below:

1 indicates that a trojan was detected. You can click on the 1 to see the corresponding screenshot. 1, 2 indicates that a trojan was detected and that there are multiple corresponding screenshots (multiple numbers do not indicate multiple trojans detected). 

--- indicates that no trojan was detected. --- / --- indicates that no trojan was detected and that there are multiple corresponding screenshots.

In all cases, please read the "Comments:" that follow the "Results:" for an explanation of the "Results:" and the corresponding screenshots.

 

Test #1: Hard Drive Scan

The first test involved a simple file scan of each folder in which the trojan servers were loaded. This test forced the applications to rely exclusively on "signature" scanning. Scans were conducted one folder at a time. Every main folder that was scanned was empty of all other files and folders except the trojan server files and the sub-folders in which they were stored.

 

ANTS 2.1

Notes: The main scanning application was loaded and a series of scans performed.

Results:

File Unpacked ASPack Packed (def) RAR/ASPack UPX ASProtect
Detect? 1 --- 1 --- --- ---

Comments: ANTS 2.1 detected the Unpacked server and the default Packed server. It missed all the other packed servers. Presumably, the Unpacked and Packed servers would be the easiest to detect. This performance puts it roughly on par with two of the other applications tested..

 

AVG Anti-Virus 6.0 Free Edition

Notes: The main scanning application was loaded and a series of scans performed. The "Complete Test" was used to perform scans, though the specified "Test Files" for the "Complete Test" were adjusted for each scan. AVG 6.0 automatically removed detected trojans ("Automatic Healing") and provided no apparent way to turn this feature off for manual file scans. After scans were run with AVG 6.0, deleted trojans were restored from backup. 

Results:

File Unpacked ASPack Packed (def) RAR/ASPack UPX ASProtect
Detect? 1 --- 1 --- --- ---

Comments: AVG 6.0 detected the Unpacked server and the default Packed server. It missed all the other packed servers. Presumably, the Unpacked and Packed servers would be the easiest to detect. This performance puts it roughly on par with two of the other applications tested..

 

BOClean 4.09

Notes: As BOClean does not perform on demand file scans, BOClean was excluded from this test.

Results:

File Unpacked ASPack Packed (def) RAR/ASPack UPX ASProtect
Detect? n/a n/a n/a n/a n/a n/a

Comments: BOClean works primarily as a resident, real time memory/process scanner. See the BOClean home page for more information.

 

McAfee VirusScan 5.21

Notes: The main scanning application was loaded and a series of scans performed.

Results:

File Unpacked ASPack Packed (def) RAR/ASPack UPX ASProtect
Detect? 1 1 1 --- 1 1

Comments: VirusScan 5.21 detected every trojan server, except the ASPack-ed server embedded in an .RAR file. This was the best on-demand file scanning performance of all six applications tested.

 

Norton AntiVirus 2001

Notes: The main scanning application was loaded and a series of scans performed.

Results:

File Unpacked ASPack Packed (def) RAR/ASPack UPX ASProtect
Detect? --- --- 1 --- --- ---

Comments: Norton AntiVirus 2000 failed to detect all but the default Packed server. Surprisingly, it even missed the completely Unpacked server. This performance makes it one of the worst performers for this on-demand file scanning test..

 

PestPatrol 3.0

Notes: The main scanning application was loaded and a series of scans performed.

Results:

File Unpacked ASPack Packed (def) RAR/ASPack UPX ASProtect
Detect? 1 --- 1 --- --- ---

Comments: PestPatrol 3.0 detected the Unpacked server and the default Packed server. It missed all the other packed servers. Presumably, the Unpacked and Packed servers would be the easiest to detect. This performance puts it roughly on par with two of the other applications tested...

 

TrojanHunter 2.53

Notes: The main scanning application was loaded and a series of scans performed by selecting a folder to scan from the main program window and then running "Full Scan."

Results:

File Unpacked ASPack Packed (def) RAR/ASPack UPX ASProtect
Detect? 1 1 1 1 1 1

Comments: TrojanHunter 2.53 detected every trojan. This performance puts it at the top of all the applications tested for on-demand file scanning.

Re-test Notes: In the initial tests run with Ruleset Update 25x-2002-03-08, TrojanHunter detected none of the trojan servers. TrojanHunter does allow users to grab "signatures" from files and use those signatures to generate new custom File Rules. During re-testing it was determined that TrojanHunter would detect all six trojans if File Rules were added for each of the six trojan files. Still further, after Update 25x-2002-03-18 was applied manually (with the custom File Rules removed), TrojanHunter detected all six trojan files.

 

Test #2-A: Memory Scan # 1

In this test the application's monitoring component was started, and then each trojan server was loaded into memory one at a time. This test allowed the applications to detect trojan servers not only on their signatures, but also on their changes to auto-start locations as well as their attempts to set up to listen in on ports.

When launched, each trojan server attempted to copy itself (usually w/ a randomly chosen name) to C:\WINDOWS, thus leaving two copies of itself on the hard drive: one in the original location, one in C:\WINDOWS. Each trojan server then attempted to launch the renamed copy it had made of itself from C:\WINDOWS. Finally, each trojan tried to edit the Run= line of WIN.INI to launch the renamed copy of the server in C:\WINDOWS on reboot.. In the case of the unpacked THAT-UNPACKED.EXE, the server also attempted to add the copied server to the Registry auto-start location HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Run.

In the case of the RAR'd THAT-ASPACK.RAR, the RAR file was opened with WinRAR 2.80 and then the THAT-ASPACK.EXE contained within the archive was launched from within the RAR file. This process of launching the server from within the RAR file created an additional copy of the server in the TEMP directory (E:\TEMP) on top of the original server file and the renamed copy in C:\WINDOWS

Given the original and copied files, as well as the changes to the auto-start locations, there were several things that could be detected:

None of the applications tested ever identified the un-RAR'd file in the TEMP directory. 

If trojan servers successfully loaded, they were killed with PrcView, copied servers deleted from C:\WINDOWS, and any changes to auto-start locations reversed between tests. 

 

ANTS 2.1

Notes: As ANTS 2.1 does not have a real time memory/process scanning component, it was excluded from this test.

Results:

File Unpacked ASPack Packed (def) RAR/ASPack UPX ASProtect
Detect? n/a n/a n/a n/a n/a n/a

Comments: ANTS 2.1 works primarily as an on-demand file scanner. See the ANTS home page for more information. 

 

AVG Anti-Virus 6.0 Free Edition

Notes: The Resident Shield was loaded at system startup.

Results:

File Unpacked ASPack Packed (def) RAR/ASPack UPX ASProtect
Detect? 1 --- 1 --- --- ---

Comments: AVG 6.0 detected only the Unpacked and default Packed servers as they attempted to load. When AVG 6.0 detected a trojan loading, it entirely stopped the trojan from loading and popped up a warning box, which read (no screenshot available):

[ AVG Resident Shield ]

Virus
Trojan horse Backdoor.Subseven 

is found in file 
D: \TROJAN-01\THAT\THAT.EXE

Enable access?

[ Yes ] [ No ] [ Heal ]

[ A - Always disable access ]

In so doing, AVG 6.0 effectively prevented a trojan copy from going to C:\WINDOWS and also prevented the WIN.INI and Registry auto-start locations from being edited. (The screenshots included for the Unpacked and default Packed servers show Windows error messages after AVG prevented those servers from loading, not AVG 6.0's warning as rendered above.)

 

BOClean 4.09

Notes: BOClean was loaded at system startup.

Results:

File Unpacked ASPack Packed (def) RAR/ASPack UPX ASProtect
Detect? 1 1 1 1 1 1

Comments:  BOClean detected all six servers as they attempted to load. When BOClean detected a trojan loading, it did not prevent the trojan from loading but did offer to kill it. If BOClean was allowed to kill the running trojan, BOClean successfully killed the running process, removed the WIN.INI Run= edit, and removed the trojan copy from C:\WINDOWS. It left the HKLM/.../Run  entry in the Registry (now invalid) for the Unpacked server, however.

Re-test Notes: In the initial tests run with Update 03/13/2002 23:31:28, BOClean detected only two of the trojan servers (Unpacked and default Packed). Once Update 03/17/2002 23:17:00  was applied, BOClean detected all six trojan servers.

 

McAfee VirusScan 5.21

Notes: VShield (System Scan only) was loaded at system startup.

Results:

File Unpacked ASPack Packed (def) RAR/ASPack UPX ASProtect
Detect? 1 1 1 1 1 1

Comments: VirusScan 5.21 detected every single trojan as it attempted to load. When VirusScan 5.21 detected a trojan loading, it completely stopped the trojan from loading, thus preventing a trojan copy from going to C:\WINDOWS and also preventing the WIN.INI and Registry auto-start locations from being edited.

 

Norton AntiVirus 2001

Notes: Auto-Protect was loaded at system startup.

Results:

File Unpacked ASPack Packed (def) RAR/ASPack UPX ASProtect
Detect? 1 1 1 1 1 1

Comments: In five of the six trials for this test, NAV 2001 failed to detect the trojan itself as it loaded (the default Packed server being the only exception), but did detect the trojans' attempts to write to WIN.INI. When NAV 2001 detected a write to WIN.INI, it popped up a warning and offered to stop it (no screenshot available):

Norton AntiVirus Auto-Protect 

VIRUS-LIKE ACTIVITY: The application D: \TROJAN-01\THAT\THAT.EXE is attempting to write to the file C: \WINDOWS\WIN.INI.

What would you like to do?

[ Stop ] [ Continue ] [ Exclude ]

It still allowed the write to HKLM\...\Run (for the Unpacked server), allowed the trojan to be copied to C:\WINDOWS, and allowed the trojan to load into memory, even after stopping the write to WIN.INI. (Note: the screenshots available for these five instances show the trojan server running in memory after Norton AntiVirus prevented a write to WIN.INI, not the warning box detailed above.)

In the one case that NAV did detect the trojan as it attempted to load (the default Packed server), it stopped the trojan from loading, giving the user a menu of potential actions. In so doing, it prevented the trojan from being copied to C:\WINDOWS and prevented a write to the WIN.INI Run= line.

 

PestPatrol 3.0

Notes: As PestPatrol 3.0 does not have a real time memory/process scanning component, it was excluded from this test.

Results:

File Unpacked ASPack Packed (def) RAR/ASPack UPX ASProtect
Detect? n/a n/a n/a n/a n/a n/a

Comments: PestPatrol works primarily as an on-demand file scanner. See the PestPatrol home page for more information. 

 

TrojanHunter 2.53

Notes: TrojanHunter Guard was loaded at system startup. 

Results:

File Unpacked ASPack Packed (def) RAR/ASPack UPX ASProtect
Detect? 1 1 1, 2, 3 1 1 1

Comments: TrojanHunter Guard caught every trojan as it loaded. When TrojanHunter Guard detected a trojan loading, it did not prevent the trojan from loading, but would kill the process and rename the trojan file that had been copied to C: \WINDOWS.

If "Automatically remove trojans" was not selected in TrojanHunter Guard,  TrojanHunter Guard offered to kill the running trojan process (see example screenshots 1 & 2). If  "Automatically remove trojans" was selected in TrojanHunter Guard,  TrojanHunter Guard killed the running trojan process outright  without further user intervention (as shown in all the other screenshots).

In both cases, TrojanHunter Guard successfully killed the running process, removed the WIN.INI Run= edit, and renamed the trojan copy in C:\WINDOWS with the extension .TCF. It left the HKLM/.../Run  entry in the Registry (now invalid) for the Unpacked server, however.

Re-test Notes: In the initial tests run with Ruleset Update 25x-2002-03-08, TrojanHunter detected none of the loading trojan servers. TrojanHunter does allow users to grab "signatures" from running processes and use those signatures to generate new custom Process Rules. During re-testing it was determined that TrojanHunter would detect all six trojans if one Process Rule was added for the Unpacked trojan file. Still further, after Update 25x-2002-03-18 was applied manually (with the custom Process Rule removed), TrojanHunter detected all six trojans as they loaded.

During the initial tests, TrojanHunter Guard (build 581) also had issues renaming the trojan files that had been copied to C: \WINDOWS and removing the WIN.INI Run= edit. After build 588 was installed, TrojanHunter Guard successfully renamed all copied trojan files and removed references to those trojan files from WIN.INI.

 

Test #2-B: Memory Scan # 2

In this test trojan servers were loaded into memory one at a time, and then each application's monitoring component was launched. This test allowed the applications to detect trojan servers based not only on their signatures, but also on their changes to auto-start locations and their listenening on pre-designated ports.

When launched, each trojan server copied itself (usually w/ a randomly chosen name) to C:\WINDOWS, thus leaving two copies of itself on the hard drive: one in the original location, one in C:\WINDOWS. Each trojan server then launched the renamed copy it made of itself from C:\WINDOWS. Finally, each trojan edited the Run= line of WIN.INI to launch the renamed copy of the server in C:\WINDOWS on reboot.. In the case of the unpacked THAT-UNPACKED.EXE, the server also added itself to the Registry auto-start location HKLM\SOFTWARE\Microsoft\Windows\Current Version\Run.

In the case of the RAR'd THAT-ASPACK.RAR, the RAR file was opened with WinRAR 2.80 and then the THAT-ASPACK.EXE contained within the archive launched from within the RAR file. This process of launching the server from within the RAR file created an additional copy of the server in the TEMP directory (E:\TEMP) on top of the original server file and the renamed copy in C:\WINDOWS

Given the original and copied files, as well as the changes to the auto-start locations, there were several things that could be detected:

None of the applications tested ever identified the un-RAR'd file in the TEMP directory. 

Between tests, the trojan servers were killed with PrcView, copied servers deleted from C:\WINDOWS, and any changes to auto-start locations reversed. All monitoring programs were closed in between tests and then restarted after the next trojan server was loaded.

 

ANTS 2.1

Notes: As ANTS 2.1 does not have a real time memory/process scanning component, it was excluded from this test.

Results:

File Unpacked ASPack Packed (def) RAR/ASPack UPX ASProtect
Detect? n/a n/a n/a n/a n/a n/a

Comments: ANTS 2.1 works primarily as an on-demand file scanner. See the ANTS home page for more information. 

 

AVG Anti-Virus 6.0 Free Edition

Notes: AVG Control Center was started, but with Resident Shield disabled. After trojans were loaded, AVG Resident Shield was activated from the AVG Control Center.

Results:

File Unpacked ASPack Packed (def) RAR/ASPack UPX ASProtect
Detect? 1, 2 --- 1 --- --- ---

Comments: Although AVG Resident Shield did eventually detect the Unpacked and default Packed servers running, it is important to recognize that the Resident Shield initially missed these trojans when Resident Shield was activated through the Control Center. Only when a trojan was directly accessed (for example, in the PrcView process list) or if a reboot was forced would AVG Resident Shield detect the trojan and pop up the warning box (no screenshot available):

[ AVG Resident Shield ]

Virus
Trojan horse Backdoor.Subseven 

is found in file 
C: \WINDOWS\THAT-EDIT.COM

Enable access?

[ Yes ] [ No ] [ Heal ]

[ A - Always disable access ]

If the trojan was already running (as was the case when it was accessed through the PrcView process list), Resident Shield would NOT shut down the trojan, however. In such cases, the copied trojan was left in C :\WINDOWS.  AVG always left the modifications to Windows auto-start locations (WIN.INI Run= and HKLM\...\Run) intact.

AVG 6.0's behavior in this test is similar not only to that of McAfee VirusScan and Norton AntiVirus (see below), but also to what was seen with Innoculate IT PE and Kaspersky Labs AVP on the original set of trojan detection tests. See the "Comments" for both applications on Test #2-B on the original Informal Trojan Detection Test page.

(Note: The second screenshot for the Unpacked server shows the Windows error message that resulted after Resident Shield prevented the trojan loading from WIN.INI on reboot. All other available screenshots show Resident Shield running while a trojan server is in memory.)

 

BOClean 4.09

Notes: After trojans were loaded, BOClean was launched from the Start menu.

Results:

File Unpacked ASPack Packed (def) RAR/ASPack UPX ASProtect
Detect? 1 1 1 1 1 1

Comments: After BOClean was launched, it detected all six trojan servers. When BOClean detected a trojan in memory, it offered to kill it. If BOClean was allowed to kill the running trojan, BOClean successfully killed the running process, removed the WIN.INI Run= edit, and removed the trojan copy from C:\WINDOWS. It left the HKLM/.../ Run entry in the Registry (now invalid) for the Unpacked server, however.

Re-test Notes: In the initial tests run with Update 03/13/2002 23:31:28, BOClean detected only two of the trojan servers (Unpacked and default Packed). Once Update 03/17/2002 23:17:00 was applied, BOClean detected all six trojan servers.

 

McAfee VirusScan 5.21

Notes: VShield was loaded at system startup, but with System Scan disabled. After trojans were loaded, System Scan was enabled through the VShield tray icon (Properties >> System Scan)

Results:

File Unpacked ASPack Packed (def) RAR/ASPack UPX ASProtect
Detect? 1, 2, 3 1 1 1 1 1

Comments: While VShield System Scan eventually detected all six running trojans, it initially missed these running trojans when System Scan was loaded. Only when the trojan was accessed (from the process list or System Mechanic's StartUp Manager, for example) would System Scan pop up a warning that offered a range of options. The options offered by System Scan were not always effective, though, and the user was required to go through a whole series of unsuccessful attempts at deleting or quarantining the trojan files before System Scan finally removed the trojans:

Even when finally deleting the trojan from C:\WINDOWS, however, System Scan left the edits to Windows auto-start locations.

As was the case with AVG 6.0 (see above) and Norton AntiVirus (see below), if the system was rebooted, System Scan would stop the trojan trying to load on reboot and offer to delete it. Again, though, all changes to Windows auto-start locations remained intact.

VirusScan's behavior in this test is similar not only to that of AVG 6.0 (see above) and Norton AntiVirus (see below), but also to what was seen with Innoculate IT PE and Kaspersky Labs AVP on the original set of trojan detection tests. See the "Comments" for both applications on Test #2-B on the original Informal Trojan Detection Test page.

(Note on screenshots: the three screenshots for the Unpacked server are examples from the series of attempts to delete the Unpacked trojan file with System Scan. The screenshots for the other trojans show only the initial popup box from System Scan when running trojan servers were accessed with PrcView, though the same series of problems deleting the trojan files were experienced with these other trojans.)

 

Norton AntiVirus 2001

Notes: After trojans were loaded, Auto-Protect was enabled and launched from the main NAV application window (Options >> Auto-Protect).

Results:

File Unpacked ASPack Packed (def) RAR/ASPack UPX ASProtect
Detect? --- --- 1, 2, 3, 4 --- --- ---

Comments: NAV Auto-Protect detected only one of the previously loaded trojans: the default Packed server. Even in this instance, though, Auto-Protect initially missed the loaded trojan when Auto-Protect was launched, just as AVG 6.0 and McAfee VirusScan did. When the trojan process was selected from the PrcView process list, NAV popped up an alert (no screenshot available), giving the user a range of options.

Norton AntiVirus Auto-Protect 

VIRUS FOUND: The file C: \WINDOWS\AJFMI.COM is infected with the Backdoor.Subseven virus.

What would you like to do?

[ Stop ] [ Continue ] [ Repair ] [ Delete ] [ Exclude ] [Quarantine ]

 "Delete" was selected, but NAV reported that it failed to delete the trojan, giving the following warning (no screenshot available):

Norton AntiVirus Auto-Protect 

Unable to delete the file C: \WINDOWS\AJFMI.COM. It is still infected with the Backdoor.Subseven virus.

What would you like to do?

[ Stop ] [ Continue ] [ Repair ] [ Delete ] [ Exclude ] [Quarantine ]

 Quarantine was then selected, but NAV reported that it failed to quarantine the trojan, again providing the warning (no screenshot available):

Norton AntiVirus Auto-Protect 

Unable to quarantine the file C: \WINDOWS\AJFMI.COM. It is still infected with the Backdoor.Subseven virus.

What would you like to do?

[ Stop ] [ Continue ] [ Repair ] [ Delete ] [ Exclude ] [Quarantine ]

Finally, "Continue" was selected and PrcView was used to kill the process, still leaving the trojan in C:\WINDOWS. When C:\WINDOWS was manually accessed with Windows Explorer, NAV popped up yet another alert with a range of options. This time "Delete" succeeded, indicating that killing the process with PrcView first was the key. All auto-start references to the trojans were still left in WIN.INI and HKLM\...\Run, however.

As was the case with AVG 6.0 and McAfee VirusScan, if the system was rebooted, System Scan would stop the trojan trying to load on reboot and offer to delete it, resulting in a blizzard of error messages, after the trojan file was deleted from C:\WINDOWS. Again, though, all changes to Windows auto-start locations remained.

NAV's behavior in this test is similar not only to that of AVG 6.0 and McAfee VirusScan (see above), but also to what was seen with Kaspersky Labs AVP and Innoculate IT PE on the original set of trojan detection tests. See the "Comments" for both AVP and Innoculate IT PE on Test #2-B on the original Informal Trojan Detection Test page.

 

PestPatrol 3.0

Notes: As PestPatrol 3.0 does not have a real time memory/process scanning component, it was excluded from this test.

Results:

File Unpacked ASPack Packed (def) RAR/ASPack UPX ASProtect
Detect? n/a n/a n/a n/a n/a n/a

Comments: PestPatrol works primarily as an on-demand file scanner. See the PestPatrol home page for more information.

 

TrojanHunter 2.53

Notes: After trojans were loaded, TrojanHunter Guard was started from the Start menu.

Results:

File Unpacked ASPack Packed (def) RAR/ASPack UPX ASProtect
Detect? 1, 2, 3 1 1 1 1 1

Comments: After TrojanHunter Guard was launched, it caught every previously loaded trojan. When TrojanHunter Guard detected a trojan, it would kill the process and rename the trojan file that had been copied to C: \WINDOWS.

If "Automatically remove trojans" was not selected in TrojanHunter Guard,  TrojanHunter Guard offered to kill the running trojan process (see example screenshots 1 & 2). If  "Automatically remove trojans" was selected in TrojanHunter Guard,  TrojanHunter Guard killed the running trojan process outright without further user intervention (as shown in all the other screenshots).

In both cases, TrojanHunter Guard successfully killed the running process, removed the WIN.INI Run= edit, and renamed the trojan copy in C:\WINDOWS with the extension .TCF. It left the HKLM/.../Run  entry in the Registry (now invalid) for the Unpacked server, however. 

Re-test Notes: In the initial tests run with Ruleset Update 25x-2002-03-08, TrojanHunter detected none of the loaded trojan servers. TrojanHunter does allow users to grab "signatures" from running processes and use those signatures to generate new custom Process Rules. During re-testing it was determined that TrojanHunter would detect all six trojans if one Process Rule was added for the Unpacked trojan file. Still further, after Update 25x-2002-03-18 was applied manually (with the custom Process Rule removed), TrojanHunter detected all six loaded trojans.

During the initial tests, TrojanHunter Guard (build 581) also had issues renaming the trojan files that had been copied to C: \WINDOWS and removing the WIN.INI Run= edit. After build 588 was installed, TrojanHunter Guard successfully renamed all copied trojan files and removed references to those trojan files from WIN.INI.

 

Conclusions

In this series of tests with several compressed variations of one well known trojan server, three applications turned in noteworthy performances:

The majority of the other applications turned in "middling" performances, detecting some trojans but missing others, both in the on-demand file scan test and the two memory/process scanning tests. 

As was the case with the original series of Informal Trojan Detection Tests, these results suggest that compressed executables present detection problems that some anti-virus and anti-trojan applications have difficulty addressing. Still further, some of these applications apparently lack the ability to kill running processes, causing severe problems when they attempted to delete trojan files from the hard drive. And even the best performers in these tests missed some files, leaving potentially dangerous trojan servers lying around, just waiting to be launched accidentally or deliberately  -- all potential causes of future havoc on the user's system.

These tests also demonstrate one important point about the unavoidable responsibilities that all users have for their systems. As we've already noted, good anti-virus and anti-trojan applications often don't finish the job. While many of these applications are undoubtedly invaluable tools for combating mal-ware like the Sub7 2.13 MUIE trojan, they can't be completely relied upon to do the job by themselves. Even after an anti-virus or anti-trojan application reports that it has deleted a virus or trojan from your hard drive, it would be wise to investigate further yourself. Important aspects of your system to check are:

And to check these aspects of Windows effectively, you have to know your system: what processes should be running, what applications are configured to launch on Windows start-up, and what files are typically found in the WINDOWS directory and its main system sub-directories. It would also be wise to run a full file scan scan of your entire hard drive, if you haven't already

In other words, your system is your responsibility. Take the initiative and start getting familiar with it, because you can't count on an anti-virus or anti-trojan application to maintain your system for you.

 

A Note On Re-testing

In response to the initial version of this web page, Privacy Software Corp.and Mischel Internet Security contacted the author. Both requested and received copies of the trojan files used in these tests. Privacy Software Corp. then released a series of updates to BOClean on 3/16 and 3/17. Mischel Internet Security released a Ruleset update on 3/17 and a series of new builds of TrojanHunter on 3/18 and 3/19. (The latest build will be soon be released as TrojanHunter 2.54.) These updates were used in a series of re-tests for BOClean and TrojanHunter that were conducted from 3/16-3/19.

There are those who will say that because these authors and  vendors were allowed to supply updates used in re-testing, the tests have been fatally compromised. In some ways they're right. In other ways, though, the charge misses the point of the tests.

The entire reason I selected the particular Sub7 2.13 MUIE trojan used in these tests was that I assumed it was a standard, widely available, easily downloaded RAT for which anti-malware applications would certainly have signatures. I was more interested in how these applications would handle that kind of standard trojan when it was compressed in various different ways, not in whether they would have signatures for it (I assumed they would). If I had wanted to test for the existence of signatures, I would have used a much, much larger trojan sample, not just one common trojan. But that wasn't the point.

As it turns out, while the default packed file was indeed a "standard" version of the Sub7 2.13 MUIE trojan, the other files I used were based on and generated from a "non-standard," unpacked version that had been modified in important ways.  In fact, I was unknowingly using two different Sub7 2.13 MUIE versions: a standard, well-known version, and a significantly altered, unpacked variant which I then compressed myself with ASPack, ASProt, and UPX. Quite unwittingly, I had stumbled across a Sub7 variant that initially eluded detection by at least two of the anti-trojan applications being tested.

In ironic fashion, this turn of events plays right to original purpose of the test, but not in any way that I had anticipated. I had sought to investigate whether altering a standard trojan would cause anti-malware applications problems in detecting those modified versions. Q.E.D. They did have problems, but not necessarily because of the modifications that I had intentionally made.

If nothing else, the test does indeed demonstrate that modified trojan variants can easily escape detection. The test also reinforces the importance of keeping current with updates for applications, because new trojan variants that can elude detection are always being created and then discovered. If you're a user of one of the applications that was re-tested, I would recommend downloading and installing the latest updates for these programs -- see the Updates from Vendors section at the top of this page.

And that is the most important thing at this point: problems which existed in several applications have been addressed by the authors and vendors, making for more robust programs. They weren't necessarily problems I expected to find, but they have been fixed.

 

Disclaimers & Limitations

Now that the results of the three tests I ran have been laid out, I feel it important to call my readers' attention to several limitations on these tests and to issue some important disclaimers.

First, this series of tests cannot pretend to be comprehensive or exhaustive. It is one series of tests run on one system by one person against one particular Remote Administration Trojan (in several variations) with one group of applications. In no way can this series of tests give readers a complete picture of the capabilities or weaknesses of any of these applications. Even to attempt to do that, one would need to test a much broader range of trojans under more rigorous and controlled testing circumstances.

Second, in no way should the test results detailed above or the comments I have added after them be interpreted as any kind of endorsement of a particular application or even a recommendation for or against any particular application. Readers will notice that I have not provided a quick, easy summary table of all the results. This is quite deliberate. I wanted to resist the urge to look at these tests as some easily tallied scorecard. At best, the tests I conducted can merely help readers acquire greater insight into some of the capabilities and limitations of the applications tested, as well as some of the issues involved with dealing with the particular type of trojan used in these tests.

(In case anyone is wondering, the programs I have used and relied on regularly over several years are The Cleaner, F-Prot for DOS, and Innoculate IT PE -- all tested in the first round of tests. Going into this second round of tests, I was much less familiar with the applications used.)

Third, I am not a computer professional. I am not a programmer, a network administrator, or a systems analyst. I am also not an experienced hacker or cracker. Although I have made every attempt to make these tests as accurate and reliable as I can, it is entirely possible that I have made errors in the design, execution, interpretation, and reporting of these tests.

All that having been said, I hope that readers find these tests to be insightful or helpful in some way. They certainly helped me gain a better appreciation for the complexities of trojan detection and removal, though they undoubtedly haven't made me an "expert" by any means.

Comments, questions, criticisms, and alerts to errors are, of course, always welcome. You can email me at:

          eburger68@myrealbox.com

Full Disclosure:  

BOClean is not available as a "shareware" trial download, unlike the other applications tested. Privacy Software Corporation generously supplied a licensed copy of BOClean 4.09 to be used in these tests. Before this web page was first posted, Privacy Software Corp. was provided via email with the "raw results" of BOClean's performance in these tests. At the time this web page was initially posted on March 15, no response or reaction from Privacy Software Corp. had been received.

At the conclusion of the re-testing process, Mischel Internet Security supplied a licensed key file for TrojanHunter 2.53 for assistance in troubleshooting several issues encountered in the build of TrojanHunter used for the initial round of tests.

 

Links

 
Anti-Trojan Applications Anti-Virus Applications Compression Programs
     
ANTS
http://www.ants-online.de/ 
 
AVG Anti-Virus 6.0 Free Edition 
http://www.grisoft.com/html/us_index.cfm 
ASPack
http://www.aspack.com/
BOClean 4.09 
http://www.nsclean.com/boclean.html
McAfee VirusScan
http://download.mcafee.com/eval/evaluate2.asp

or http://www.mcafeeb2b.com/naicommon/buy-try/try/products-evals.asp
 
ASProtect
http://www.aspack.com/asprotect.htm
Pest Patrol 3.0 
http://www.safersite.com/PestPatrol/About.asp
or http://www.pestpatrol.com/PestPatrol/About.asp
 
Norton AntiVirus 
http://www.symantec.com/nav/nav_9xnt/
or http://www.symantec.com/nav/indexA.html
UPX
http://upx.sourceforge.net/
TrojanHunter 2.53 
http://www.misec.net/trojanhunter.jsp
WinRAR
http://www.rarlab.com/

 

Other Trojan Tests On the Web
Anti-Virus Software Trojan Testing
http://www.claymania.com/tests-trojan.html
 
Best Anti-Trojan Programs
http://www.techsupportalert.com/best_anti_trojans.htm
 
PC Flank - Personal Firewalls vs Leak Tests
http://www.pcflank.com/art21.htm
 
PC Flank - Trojan Tests (1)
http://www.pcflank.com/art17d.htm
 
PC Flank - Trojan Tests (2)
http://www.pcflank.com/art26a.htm
 
PestPatrol -Comparison
http://www.safersite.com/Whitepapers/NSTL_Report.pdf (PDF document)
or http://www.pestpatrol.com/Whitepapers/NSTL_Report.pdf (PDF document)
 
PestPatrol -Removing Sub7
http://www.pestpatrol.com/Whitepapers/Comparison/Removing_subseven.asp
http://www.safersite.com/Whitepapers/Comparison/Removing_subseven.asp
 
Rokop Security - Tests
http://www.rokop-security.de/main/search.php?query=&topic=5
 
Tauscan - Comparison
http://www.agnitum.com/products/tauscan/compare.html
 
The Cleaner - Comparison
http://www.moosoft.com/thecleaner/comparison.php
 
Trojaner-Info - Test Center
http://www.trojaner-info.de/testcenter.shtml

 

Revision History

Mar 20, 2002
Mar 16, 2002
  • Added an Updates from Vendors section at the top of this page.
  • "Notes" and "Comments" sections for BOClean on Test # 2-B revised to correct an error. The original passages implied that BOClean was loaded before the trojans, when in fact the reverse was true. Apologies for any confusion.
Mar 15, 2002
  • Initial release of this document.

Home [frames]        Home [no frames]

© 2000, 2001, 2002 Eric L. Howes (eburger68@myrealbox.com)