SIGIA-L Mail Archives: Re: SIGIA-L: tester numbers and fees
Re: SIGIA-L: tester numbers and fees
From: Whitney Quesenbery (whitneyq_at_acm.org)
Date: Thu Apr 26 2001 - 18:55:45 EDT
At 02:40 PM 4/26/01 -0400, Tema Frank wrote:
>Speaking of Jakob Nielsen, he maintains that you only need five users to
>test a site. (Or at least to get the bulk of the value from doing so.) Do
>you agree? If not, how many do you think are necessary (to test a fully
>developed web site, not just a prototype)?
What he actually says is that for a given usability test, you begin to see
a steep fall-off of results after 5 users. Many people in the usability
community advocate doing more tests with fewer users rather than a single
test with many users, especially if you are working within the design cycle
to iteratively improve the design.
I've certainly seen this effect myself. We once designed a bank ATM, and
had scheduled 20 users for each round of tests. What happened was that
there was a relatively small group of problems, some of which were found by
each user. By the afternoon of the first day, we had learned most of what
we could, but had to sit and watch user after user having the same problem.
Had we been able to try to fix the problems we observed after the first
five users, we would have been able to confirm that our "fix" was not just
another problem. Smaller sets also lets you concentrate on specific design
areas in a more focused way.
However, Jared Spool and company have been reporting some contradictory
data in his usability research on e-commerce sites. They are finding that
the number of problems found does not fall off as dramatically as expected
with increased numbers of participants. One of the theories is that in
these tests (which allow users, realistically, to set their own tasks) have
such variation in goals, operations and navigation that the permutations of
"the site that each user sees" are much larger than in a more controlled
software environment. He is also dealing with the question of "what is a
usability flaw?" If a store database displays the wrong picture for a CD,
and the user does not make a purchase because the information is ambiguous,
is that a technical bug or a usability flaw. Spool's point is that it does
not matter, because it caused the user to abandon the task.
Obviously, you don't want to do your QA in this way, but the point that the
problems observed in a usability test may have many different sources is an
important one. Did they not "find" the right button because it was:
- labeled in a way that did not match their expectations
- presented in a way that made it less noticeable
- placed on the page in a way that "hid it" from the user
- tried, but the results did not match expectations
- etc.
Doing a careful analysis of why users encountered problems is as important
as the observation detail.
Whitney Quesenbery
whitneyq_at_acm.org
Visit the Usability SIG at http://www.stcsig.org/usability
Cognetics Corporation - http://www.cognetics.com whitneyq_at_cognetics.com
"Isn't is amazing how the play fit so perfectly between the time when the
lights went up and the time the lights went down." - Picasso at the Lapin
Agile, Steven Martin
===========================================================================
List archives are available at:
http://www.listquest.com/computers/tier2/computer_misc.html
--------------------------------------------------------------------------
To subscribe or unsubscribe, send mail to majordomo_at_asis.org with the
appropriate command from the list below in the body of the message:
subscribe sigia-l
subscribe sigia-l-digest
unsubscribe sigia-l
unsubscribe sigia-l-digest
This archive was generated by hypermail 2.1.2
: Sun Nov 23 2003 - 22:54:37 EST
|
