by Eric L. Howes

Adware apologist Eric Goldman has been touting a recently released study [PDF] of users' responses to installation notice practices that, according to Goldman, "destroys many of the foundational assumptions of regulators and anti-spyware advocates about consumer behavior and psychology, thus highlighting how many current regulatory/consumer protection efforts are misdirected." Goldman, it should be noted, generally dismisses calls from the anti-spyware community for improved notice and disclosure during installation of spyware and adware. (Curiously, though, Goldman has been insisting that the anti-spyware industry provide better notice to users and adware vendors about the software their applications detect by adopting a widely accepted accepted, industry-standard definition of "spyware.") It comes as no surprise, then, that Goldman would warm to one of the primary conclusions of this new paper, which was previewed at the Boalt Spyware Conference in April 2005:

Our study indicates that while notice is important, notice alone may not be enough to affect users’ decisions to 
install an application. (from the abstract)

I agree with Goldman that this is an important and valuable contribution to the small but growing research literature on spyware and adware, and I would encourage folks in the anti-spyware community to give it serious consideration.

Where Goldman and I part company, however, is in our understanding of just what this study tells us about users and their installation experiences with spyware and adware. Goldman hastily concludes that the study confirms his deepest suspicions about the general futility and uselessness of attempts to improve notice and disclosure of key terms and functionality that users ought to know about in order to make informed decisions during the installation of spyware and adware.

Having looked over the study, which is full of fascinating insights and provocative questions, I think Goldman reads far too much into the study's data, which are more ambiguous and muddied than Goldman lets on. Moreover, in his haste to trumpet findings that are apparently friendly to his own established position, Goldman neglects to mention other aspects of the study that confirm what many of us in the anti-spyware community have been saying for years.

On this page:

• Notes & Observations
• Familiar Lessons
• Valuable Insights
• Closing Thoughts

The study in question can be downloaded here:

Stopping Spyware at the Gate: A User Study of Privacy, Notice and Spyware (Good, et al, 2005) [PDF]

Notes & Observations

In no particular order, let me first address some of the more problematic and/or noteworthy parts of this study.

1. The scenario presented to study participants is itself somewhat flawed: 

Users are asked to play the role of installing software on others' PCs, not their own (4.1.2, p. 4). Users are never put in the position of having to take responsibility for the PC or the resulting effects of their installation decisions -- a burden that could significantly affect their installation decisions. We are always more careful and considerate when maintaining our own PCs.

2. The scenario itself tilts towards installations: 

As presented to study participants, the scenario description (4.1.2, p. 4) arguably creates the unstated goal of installing at least some applications for the fictitious friend of the scenario. Installing no applications was apparently never explicitly offered as an option. 

While this condition or situation may loosely resemble that of a typical bundled installation (i.e., an installation in which adware piggybacks on a host program that users are interested in or intent on intstalling), it certainly doesn't resemble one of the most common situations in which adware is installed -- namely, unsolicited, automated ActiveX installs that are sprung on unsuspecting users at third-party web sites. This installation context differs markedly from that used in the study, which had participants clicking on desktop links to initiate the installation of software that is presumably attractive to the scenario's fictitious friend.

Notice and disclosure in automated ActiveX installs is a critical issue precisely because users are not expecting or requesting to install software, unlike the scenario used for this study, in which users consciously initiate the installation of software packages that they may want. Moreover, ActiveX installations are presented to users in somewhat confusing and pressured circumstances (as compared with the deliberate, organized, and methodical environment of the study), and the software itself can be mistaken for browser plug-ins necessary to view content at the sites visited.

3. The study tests only traditional setup executables & bundled installs:

All of the software used in the study (4.1.1, p. 4) takes the form of traditional setup executables that users must deliberately download and then run to initiate the installation process. Some of these setup programs bundle one or more programs in addition to the main program (or host program) that users were interested in installing. These installations replicate but one type of installation process through which adware can be delivered to users' PCs. As noted just above, however, another common, though markedly different, installation process for adware involves automated ActiveX installs performed at third-party web sites. 

ActiveX installations are significantly different than traditional bundled installs in several ways, including: the context of installation  (user-initiated and requested installs vs. unrequested ActiveX installs); the actions performed by the user (a series of decisions and clickthroughs vs. a single click in many if not most ActiveX installs); the software installed (many ActiveX installs involve just adware and no functionally attractive host program at all); and the information put before users (ActiveX installs typically present dramatically less information by default). Given these significantly different installation characteristics, it is an open question whether the main conclusions of this study can be easily and completely extended to ActiveX installs. 

4. One type of "notice condition" is faulty and misleading:

At least one of the "notice conditions" (the generic Microsoft notice; 4.1.3, pp. 4-5) is premised on incorrect assumptions concerning the "trustworthiness" of an application with respect to whether a publisher is "recognized" or not -- i.e., whether or not the applications were signed with a valid, unexpired digital certificate issued by a recognized CA. That an application is digitally signed provides next to no indication of the "trustworthiness" of an application, and validating a digital signature certainly doesn't serve as "notice" in the same way as the other forms of "notice" assessed in this study. Put simply, the participants of the study were fed bad data, and some participants even based decisions on that bad data (e.g., some users refused the eDonkey install and accepted the KaZaA install because the eDonkey publisher was not "verified" whereas the publisher for the KaZaA install was -- 5.2.5, p. 7).

5. Several minor errors appear in the write-up:

Table 1 (5.2.4, p. 6) is tallied wrong: 10 + 10 + 11 = 31 (total subjects in study), not 30 as indicated in "Total" row of the table. Similarly, the screenshot for Figure 1 (4.1.3, p. 4) is incorrectly labeled: pictured is the EULA for Weatherscope, not Webshots. 

6. Key information about the tested applications and study participants is not provided in some instances:

For example...

• KaZaA installations typically include several sponsor programs, each with their own EULAs and notice screens, not just a single EULA and notice screen. Which was used for the study?
   
• Some programs were digitally signed, others were not. Given that the report of an "unknown publisher" had some effect on users' decisions, it might be important to know which had "recognized publishers" and which did not, yet we are never told.
   
• We never learn just how the users (who varied in computer savviness and experience, to say nothing of gender) were distributed among the notice/installation types. Still further, these users employed different "strategies" (5.2.2, p. 6) in their decision-making processes. How were users with differing installation "strategies" distributed? Again, we never learn.Given the small sample size (which the authors acknowledge; 5.5, p. 9), even small differences in the distribution of users could yield idiosyncratic results.
   
• The study indicates that the quality of information in the EULA had effects not only on users, but also on the ability of the authors themselves to construct meaningful "short notices" (5.2.9, p. 8). Despite this, we are never told (outside of the discussion of the KaZaA EULA vs. the eDonkey EULA) which applications had better EULAs and which had worse or what the nature of those qualitative differences was.

7. Application differences are not fully addressed:

The paper fails to discuss at length the differences between the key terms and functionality of the programs (e.g., which include adware functionality and effects that are generally disliked by users and which do not), only vaguely alluding to or briefly mentioning those differences in passing. In fact, these differences could prove to be quite important to understanding the results observed among user-performed installations. (To their credit, the authors note that the mix of applications selected for the study "could have influenced our results" [5.5, p. 9] and state that they "plan to perform a more controlled experiment on notice" [7., p. 10]) 

8. The installation data are ambiguous and of limited value:

The data on installation rates are muddied enough that they are of limited use. Table 2 (5.2.4, p. 7) appears to indicate that changes in notice (setting aside the erroneous assumption that checking digital signatures constitutes meaningful "notice") make very little difference at all in installation rates. Yet Table 4 (5.2.7, p. 7) tells a somewhat different and more complicated story.

• The two programs with the most consistent installation rates (Google Toolbar and Webshots) also had the highest installation rates, accounting for over 50% of the installs -- more than the remaining three combined. Given this pattern, it isn't necessarily surprising that the installation totals would not have varied significantly across different types of provided notice.
   
• Of the remaining three programs that showed significant variations in installation rates:
   
  - at least one program's installs were significantly affected by inappropriate data being presented to users (eDonkey's "untrusted publisher").
  
  - another program's installations were skewed upward by the provision of information that was "vague" (by the admission of the authors; 5.2.9, p. 8) in both the EULA as well as custom "short notice," which was based on the original EULA. This problem with the KaZaA installation data is only compounded by the "known/unknown publisher" issue.
  
  - tellingly, the Weatherscope installation rate plunges dramatically as better information (in the form of the "short notice") is provided, just as expected, though Weatherscope's installation rates were never high to begin with.

Given the significant differences between the programs and their installation mechanisms and processes, the low number of study participants, as well as the problematic data fed users during one type of installation, how are we to draw any meaningful conclusions from this data? Does this study usefully and reliably measure what it purports to measure?

It is certainly true that the reported installation rates, when taken in sum, do seem to suggest that improved notice has very little effect on user installation decisions. Once we separate out the installations by application (as in Table 4) and start to interpret the data in light of what we known about the peculiar qualities of the applications and installations, though, a different picture begins to emerge.

9. Table 4 demonstrates the preference among users for non-adware programs:

According to Table 4 (5.2.7, p. 7), the most frequently installed programs were (in order):

• Google Toolbar: this application is fairly innocuous, displaying no pop-up advertising on users desktops, not hijacking their browsers, and collecting and transmitting only a minimal amount of information. Moreover it has at least one feature highly desired by users (pop-up blocking). Not surprisingly, users reported that Google's reputation was good and installed it the most frequently.
   
• Webshots: a less well known application, this software also has few features (if any) that would be objectionable to users. Not surprisingly, it also was frequently installed, though less so than the Google Toolbar, perhaps because fewer users were interested in digital photos.
   
• eDonkey: topping the bottom tier of installed applications, eDonkey was still more frequently installed than its adware infested cousins, even with the erroneous information provided users in the generic warning screen. That could be because eDonkey allows users to complete the installation without installing adware. In other words, users can install eDonkey without fear of an adware infestation because they may opt-out of the adware installation. By contrast. both Weatherscope and KaZaA include adware by default and do not allow users to opt-out of the adware. (eDonkey's installation rates were lower than KaZaA's in Notice Condition 3, but that may be attributed to comparatively vague disclosures provided by the KaZaA EULA, on which the "short notice" was built -- see 5.2.9, p. 8.)
   
• Weatherscope & KaZaA (tied): rounding out the bottom tier are the two adware programs. Not coincidentally, these two also happen to be the programs with the worst reputations among users.

Returning to value of the data and what story that data tells, we could argue that the Google Toolbar and Webshots installations are comparatively inconsequential, given that these applications include little if any functionality that most users would object to. Thus, we wouldn't expect that variations in notice would have much effect on installation rates. Moreover, the KaZaA installations are problematic because of the uniformly vague information provided in the various installation types. The eDonkey installs are somewhat suspect as well, given the problematic warning provided to users. 

That leaves the Weatherscope installations, which do include functionality that many users would find objectionable (pop-ups, privacy effects). Moreover, Claria/GAIN EULAs tend to be rather detailed and involved (and Claria/GAIN programs tend to be digitally signed). Not surprisingly, the detailed "short notice" created from the Claria/GAIN EULA drove down installation rates significantly (from a high of 60% to a low of 27%).

Familiar Lessons

Beyond the several problems discussed above, folks in the anti-spyware community will find much in this study that confirms what we have been saying for years. To wit:

Users hate pop-up advertising

Among the reported concerns of users, pop-ups ranked second:

Popups (~60%) – popup advertising was the second largest concern out of our participants, across all categories of users. Many users had strong reactions to them. “I hate them!” was a reaction echoed by several participants. Many were extremely reluctant to install a program that had popup advertising or seemed like it would. One participant stopped an installation after she saw the word “GAIN,” which reminded her of Gator, a company that had put advertising on her machine before. (5.2.3, p. 6). 

Many of us have been insisting for some time that the effects of unwanted advertising of adware programs on users were being underrated when compared with potential privacy effects of these programs. In fact, contrary to the established PR mantra of the adware and advertising industries, which insist that "spyware" be judged and defined only on its PII data collection functionality and that the advertising displayed by adware is generally innocuous or even beneficial, the users in this study confirm what we have been arguing: namely, that unwanted, intrusive advertising is more important to users than the potential privacy effects of PII collection/transmission (60% vs. 10% reporting as a concern), and that users regard this advertising as highly objectionable -- in this study ranking it the second most important consideration (60%) after functionality (80%). 

A side note on functionality being the top concern of users: I wouldn't make too much of this, as a user population that didn't regard functionality and features as the most important consideration in downloading applications likely wouldn't be downloading applications to begin with -- thus providing no user base performing downloads to study.

Users don't read EULAs and find them impenetrable

The study reports that users generally don't read EULAs (5.3.1, p. 8), don't understand them (5.3.2, p. 8), and have problems making any sense of them (5.3.2, p. 8), let alone even reading them (5.3.3, p. 8). This comes as no shock to those of us in the anti-spyware community, who have been insisting for years that users are unable to make heads or tails of these legal documents, that these documents simply cannot be regarded as useful notice or disclosure, and that they are largely useless for securing meaningful consent. Contrast with the adware and advertising industries, which for years have insisted that as long as they stuck a EULA in front of users they had provided sufficient notice and disclosure.

Quality of the notice matters

According to the study:

Vague short notices can also lead users to assume false security
 

An interesting result discovered in the installation process was the higher number of installations for KaZaA in the short notice case as opposed to the control case (see Table 4). In talking with participants about their choice to install or not, we discovered that they were more likely to install KaZaA instead of Edonkey because it “didn’t seem as bad.” This case was especially pronounced in the case of the short notices because users typically wanted to install one or the other, and used the information in them to determine which one to install. This was interesting because Edonkey actually disclosed more, and gave users the option to opt-out of certain instances, whereas KaZaA did not have that option. However, in creating our short notices, we had to follow what was stated in the EULA, which for KaZaA was vaguer than Edonkey. In this case, providing vague information created an impression of increased security. (5.2.9, p. 8)

Still further:

We also found that providing vague information in EULAs and short notices can create an unwarranted impression of increased security. This places increased importance on the accuracy and presentation of the information that users consult to make their installation decisions. In these cases, it may be helpful to have a standardized format for assessing the possible options and tradeoffs between applications. (6., p. 10)

Again, this conclusion hardly comes as a shocker to members of the anti-spyware community, who have long decried the vague, euphemistic, near useless information provided in typical adware installations. As with so many other issues, the adware industry has generally dismissed these concerns.

Users are demanding better notice and find genuinely improved notice helpful

According to the study:

There seems to be a strong disconnect between user expectations of EULA content and actual EULA content. One user summed up this confusion by stating “They should have notices to show what they are really installing on the computer. They trick you [into] thinking it is just a license agreement, [you] hit OK, and then you get an advertising bar or a lot of junk!”" (5.3.2, p. 8)

And still further:

Our participants had several suggestions about how license presentations can be improved, but most notably they wanted them “shorter, easier to read and in very accessible language.” One participant stated that she would like to see something “that would tell you exactly what you want to know. [It would] provide a summary first, bold whatever is important, bold what is in the software, who is using it, and say if it is safe to download.” (5.3.3, p. 8)

Lord knows, folks in the anti-spyware community are all too familiar with these kinds of complaints -- we hear them every day while working with spyware and adware victims. 

EULA/notice overload can be a problem

According to the study, "notice fatigue" can set in among users:

Despite the positive reactions, some users simply ignored them as well. Despite stating in the post-interview that they would like “clear and concise” information, they made comments such as, “It is hard to say if I would read them [short EULAs] even if you flashed IMPORTANT at the top. After the third or fourth one I wouldn’t read and it would be easy to skip.” (5.2.6, p. 7)

And that's exactly the reason we have been denouncing adware installations that blitz users with four, eight, even sixteen different adware programs in a single install process, putting endless EULAs and notice screens in front of users. That kind of installation experience simply encourages users not to pay attention and simply click through mindlessly, and yet the adware industry has been employing ever more oppressive multi-adware installs over the past few years, dumping dozens of adware programs on users in one go and insisting all the while the practice is legitimate.

Users can take advantage of provided notice to choose and evaluate programs meaningfully

Good notice is hardly without value. In a significant qualification presented just after the primary conclusion, the authors note:

It may be tempting to interpret our results to claim that users do not care about privacy, especially when the utility of a software application is high for a particular user. However, we discovered that privacy and security become important factors when choosing between two applications with similar functionality. Given two similar programs (e.g., KaZaA and Edonkey), consumers will choose the one they believe to be less invasive and more stable." (6., p. 10)

This conclusion is an important one for it does suggest that users can benefit from well crafted notice and disclosure.

Valuable Insights

One aspect of this study that I found useful and noteworthy was the study's classification of users into several types or categories (5.2.2, p. 6) based on their installation strategies ("Once bitten, twice shy," "Install first, ask questions later," etc.). Users do take their own prior experiences and previously developed strategies into installation decisions, and we ought to be considering these experiences and strategies when designing notice and consent regimes.

Moreover, in following up on users' initial installation decisions to measure potential regrets (5.4, pp. 8-9), the study emphasizes that we should not regard information provision as a singular event limited to installation and setup -- that, instead, we should recognize that users are constantly in the process of evaluating their installation decisions and that their decision-making processes extend beyond the formal setup process for an application. All the more important, then, that we ensure users are provided with good information about adware programs in contexts outside of installation -- for example, in the design of pop-up advertising that appears on their desktops, or in the visible indicators of program execution that alert users to the presence and functioning of software on their systems.

Thus, even if I find the data provided in this study to be more ambiguous and difficult to interpret than Goldman apparently does, I nonetheless think this study a valuable one, for it effectively highlights the complexity of doing such studies of user behavior and provides a wealth of issues for researchers to pursue further.

Closing Thoughts

Before closing, I should be clear about that fact the we in the anti-spyware community have never regarded good notice and disclosure as a panacea to the problems caused by unwanted adware and spyware installations. As this study highlights, user installation decisions are driven by a number of factors, and good notice and disclosure practices will never completely determine the outcome of users' installation decision-making processes. 

Still further, our goal has never been to stop completely all adware installations, nor have we been under the delusion that merely providing good notice and disclosure would be sufficient to realize that enormous end. You won't find many "anti-spyware zealots" (as Goldman occasionally describes us) grumbling about freeware Eudora installs or losing sleep over the number of people downloading and installing the ad-supported versions of Opera or FlashGet. 

We well recognize that even when given good notice some users will elect to install adware programs for any number of reasons -- they may actually want the programs' functionality; they may regard such functionality as a trivial price to pay for access to "free" content; or they may simply be so careless and impatient that no notice and disclosure provided during installation could have any chance of preventing them from clicking through installation screens at a breakneck pace.

No, the goal of the anti-spyware community has always been to ensure that users who could make use of good notice and disclosure get it, and we think that even if those users are a minority the effort will have been well worth it. Our philosophy has always been: "If users really want to install such programs and they understand what they're getting into, then so be it. But they ought to at least have good information on which to base their installation decisions as well as a fair shot at making those decisions in a reasonably non-coerced context. Still further, they ought to be able to employ tools such as anti-spyware programs to detect and advise on potentially problematic software they may have downloaded and, if they so choose, use those tools to remove software that no longer accords with their installation preferences."

Finally, I cannot help but note that in present circumstances -- which include, lest we forget, the rampant use of security exploits to force install unwanted advertising software on users' PCs, outright deceptive and coercive pop-ups and banner-ads that trick users into installing adware and porn dialers, deliberately obscured notice text (yellow-on-blue font/background schemes, keyhole scroll boxes, etc.), deliberately misleading and deceptive notice language (from slippery euphemisms to outright falsehoods), installation routines targeted at kids, and on and on -- it is more than a little disingenuous to seize on but one study with a such a small sample and muddy data and triumphantly declare that users don't need better notice and disclosure. 

Internet users and adware victims deserve and demand better than that.

Eric L. Howes
14 July 2005