Return to http://www.info-arch.org/lists/sigia-l/0403/0379.html


SIGIA-L Mail Archives: RE: [Sigia-l] user survey questionnaire

RE: [Sigia-l] user survey questionnaire

From: Lyle_Kantrovich_at_cargill.com
Date: Mon Mar 29 2004 - 17:13:10 EST

Patrick,

A survey is just a tool. You have to consider what the questions are
that you really need answers to. Are they closed ended questions, or
open ended questions? E.g. do you want to know something like "Do the
users like this representation of the brand?" or something like "What do
they like or dislike about this design, and why?"

I've had pretty good luck with the SUS survey as a post-usability test
survey. It helps measure test participants' *perception* of the
system's usability. It's a great example of a well designed survey
based on closed-ended questions.

SUS - A quick and dirty usability scale
http://www.usability.serco.com/trump/documents/Suschapt.doc

I often will add a few open-ended questions to the SUS survey, but these
don't factor into the SUS score. And of course, the SUS score is
nothing close to a good measure of a system's usability - but it's a
good measure of perceived usability. I've had users fail at many tasks
in a test and still give a very high rating of usability in the survey.
This is just further evidence that users often blame themselves rather
than the system/design.

One client recently did a pretty extensive survey - many of the
open-ended comments were difficult to decipher and just opened up
further questions. Luckily, it wasn't an anonymous survey, so they
could follow up with the person who made the comments to understand why
they blasted the design or to clarify comments that didn't make sense to
the team.

The risk with surveys is that they may not really be measuring what you
want them to measure. If you show a few screenshots, and do a survey
asking what users think of them, is that a good measure of the design's
completeness? of the visual design or branding? of the design's
usability? of the IA? Probably not.

BUT - sometimes a survey can be used as a communications tool or as a
way to get buy-in.

Regards,

Lyle 

----
Lyle Kantrovich
User Experience Architect
Croc O' Lyle - Personal Commentary on usability, information 
architecture and design.
http://crocolyle.blogspot.com/
"Simplicity is the ultimate sophistication." 
	- Leonardo da Vinci
 
-----Original Message-----
From: gamutant_at_earthlink.net [mailto:gamutant_at_earthlink.net]
Sent: Monday, March 29, 2004 1:53 PM
To: sigia-l_at_asis.org
Subject: Re: [Sigia-l] user survey questionnaire
Thanks everyone for your responses. I should have mentioned that we 
initially requested to conduct telephone interviews with international 
users but were told that would be too difficult to organize. The 
questionnaire was intended as a back-up solution to provide us with at 
least _some_ feedback from the trenches in addition to the 'top-down' 
goals mandated by the various stakeholders. As far as the survey is 
concerned, I think we have struck a reasonable compromise between 
open-ended questions and more concise multiple choice queries with room 
for additional comments in text fields. But, I guess the big picture 
problem is a general intransigence on the client's part toward 
conducting user research. As Ziya suggested though, I think we might 
have better luck getting input once we are actually closer to launching. 
Since there is a bit of a time crunch, perhaps the solution is to go 
ahead with whatever feedback we can get now and then push a bit harder 
for access to users when we are ready with a prototype? Just thinking 
out loud. Thanks again for everyone's input.
:P
 
-----Original Message-----
From: Eric Reiss <elr_at_e-reiss.com>
Sent: Mar 29, 2004 10:40 AM
To: sigia-l_at_asis.org
Subject: Re: [Sigia-l] user survey questionnaire
Dear Patrick,
Keep in mind that all volunteer questionnaires are self-segmenting.
In other words, you only get results from people who like to take
surveys. This has a tendency to skew the results - quite a lot in
many cases.
Also, getting a questionnaire right is very tricky (if you insist,
check out Don A. Dillman's fat but practical book - "Mail and
Internet Surveys, the tailored design method").
As Jesse suggested, if you're new to this game, get someone with good
interviewing skills to call people up in person. Much better ROI. And
if you're doing this yourself, get a couple of clocks to tell you the
local time. This is particularly useful if you're dealng with
anything more than three timezones away (the mere fact that CET went
on summertime yesterday has screwed up my own "in-the-head"
calculation throughout the day).
Cheers,
Eric
Eric Reiss
Principal
e-reiss aps
copenhagen, denmark
www.e-reiss.com
------------
When replying, please *trim your post* as much as possible.
*Plain text, please; NO Attachments
Searchable list archive:   http://www.info-arch.org/lists/sigia-l/
________________________________________
Sigia-l mailing list -- post to: Sigia-l_at_asis.org
Changes to subscription: http://mail.asis.org/mailman/listinfo/sigia-l
Patrick Grizzard
140 Jackson St.
Apt. 3A
Brooklyn, NY 11211
t:646.522.9667
http://www.gneumatic.net
"And yet, it moves."
------------
When replying, please *trim your post* as much as possible.
*Plain text, please; NO Attachments
Searchable list archive:   http://www.info-arch.org/lists/sigia-l/
________________________________________
Sigia-l mailing list -- post to: Sigia-l_at_asis.org
Changes to subscription: http://mail.asis.org/mailman/listinfo/sigia-l
------------
When replying, please *trim your post* as much as possible.
*Plain text, please; NO Attachments
Searchable list archive:   http://www.info-arch.org/lists/sigia-l/
________________________________________
Sigia-l mailing list -- post to: Sigia-l_at_asis.org
Changes to subscription: http://mail.asis.org/mailman/listinfo/sigia-l

This archive was generated by hypermail 2.1.6 : Mon Mar 29 2004 - 17:19:26 EST

 
Return to http://www.info-arch.org/lists/sigia-l/0403/0379.html