SIGIA-L Mail Archives: SIGIA-L: some fundamental questions
SIGIA-L: some fundamental questions
From: James Weinheimer (jamesw_at_princeton.edu)
Date: Thu May 03 2001 - 16:23:41 EDT
Forwarded from Dave Collins with some reformatting.--Jim
-------- Original Message --------
I'm fairly new to the Usbility industry officially, though it was part of my
job to some extent as a developer before.
I am using Rubin's 'Handbook of Usability Testing' as reference. I have just
run my first Usability Study with 5 subjects on our newly developed software
product in its alpha phase. The test went well I think, but it has raised
some fundamental questions that I'm ruminating over. I thought maybe I could
get $ome input.
While all our real end-users will be trained, the test subjects were not,
but were instead given a document to read about the app before doing the
study. Then I gave them the tasks, which went through the major parts of the
app one by one. The tasks were pretty simple and pretty discreet. But it
seems that the success or failure or anything I learn from the test comes
down to this task list. Too much detail walks them down the garden path. Too
little, and they are stumped. The subject's actions can be drastically
altered merely on the wording of the script. How do I deal with this?
According to my user guide, they are extremely diligent in what they record
and how they record it (going so far as to develop a shorthand to track
mouse actions). I found that it was all I could to to write some rough notes
about what the user was struggling with. I found time only to write
conclusions about what I saw, rather than impartially observe, since it
seems to me, conclusions would be skew later analasis. How am I supposed to
track times, number of tasks, number of errors, severity of error,
participant's actions, their comments and my own comments? To give me valid
data, I have to do this diligently over all my users, yet in reviewing, much
of my notes have gaps. Can I just write 'many users experienced this
problem' instead of '80% of the (5) users'?
I am having trouble determining what exactly I was hoping to get out of
this. Places where they tripped up were places where we knew they would
anyway. How do I justify the time and expense of this test? I can't just say
80% (or 4 out of 5) users didn't recognize the foobar for what it was. I
can't present them a video clip showing users not using the foobar. They'll
say 'yah, we knew that'.
I am summarizing my data, but I am not sure if my observations, even
summarized, will pique much interest. I suspect I may have to take the extra
step and draw conclusions and make recommnedations on how to improve the
product. This means that the summary doc never really gets off my desk, I
end up proposing a new protoype right from here. This can't be right.
Input is greatly appreciated.
Dave Collins
Interface Designer
The Bulldog Group Inc.
184 Front St E.
M5A 4N3
v: 416-603-0324 x672
e: dcollins_at_bulldog.com
w: www.bulldog.com
===========================================================================
List archives are available at:
http://www.listquest.com/computers/tier2/computer_misc.html
--------------------------------------------------------------------------
To subscribe or unsubscribe, send mail to majordomo_at_asis.org with the
appropriate command from the list below in the body of the message:
subscribe sigia-l
subscribe sigia-l-digest
unsubscribe sigia-l
unsubscribe sigia-l-digest
This archive was generated by hypermail 2.1.2
: Sun Nov 23 2003 - 22:54:38 EST
|
