Date: Thu, 26 Feb 1998 14:42:02 -0600
From: escargo@anubis.network.com (David S. Cargo)
Message-Id: <199802262042.OAA10121@brutus.network.com>
To: calvin-ui98@dagobah.stwing.upenn.edu
Subject: usertest log
THis is my user test log.
David S. Cargo
CSci 5110 Winter 1998
User testing log
I participated in three user tests, once as an observer, and twice as a
note-taker.
Tester 1, test conducted by Rye and Hokanson.
With the first tester, Jeff gave an introduction to our application,
which I thought was nicely done. What we didn't do that the other group
(Group 0) did, was ask our testers ahead of time how much and what kind
of experience that they had. It did become apparent, at least in the
three tests that I participated in, that these testers were not highly
familiar with UNIX.
The application starts at the project tab; the user glances at it and
then goes to the third tab, the archives tab. Looking at all the fields
and data there, he says, "What is this?"
Right away it becomes clear that tabs are not conveying enough sequence
information.
The tester gets a repetition of the task description for task 1.
After not finding anything of interest on the archives tab, he gets to
the mailboxes tab. Here, he doesn't see the "Add" button right away.
After adding the default mailbox to the listbox, he selects the mailbox
in the listbox before continuing.
He goes to the generate tab and pushes the generate button, completing
task 1.
For the second task, which uses multiple mailboxes, the tester tries to
add a second mailbox to the mailbox listbox, which generates and error
message. Part of the program constraints on the project tab says that
only one mailbox will be allowed. The tester gets to the project tab,
changes the selection so that multiple mailboxes are allowed, and
returns to the mailboxes tab to add more mailboxes.
One problem the user runs into is that the tk_getOpenFile does not allow
multiple selection, which the tester really wanted to use.
The tester was able to complete the remaining tasks straightforwardly.
Afterward, the tester commented that he doesn't think he's the right
user; he couldn't relate to the tasks. He also said he liked the
context help, and used it sometimes. He pointed out that the error
message about multiple mailboxes should have pointed him to the project
tab.
Tester 2, test conducted by Dang and Cargo.
Dang gives an introduction and explains what the application does.
Tester 2 starts by going to the archives tab and leaves everthing at the
defaults, since task 1 specified using the defaults. The tester then
goes to the generate tab and explores. It is still inactive, so he
goes to the mailboxes tab. He sees the default mailbox in the entry
field, and tries to enter it using a return keypress. Since that is not
implemented, he uses the "Add" button successfully.
This particular user was given the most complex task, one that required
creating and applying filters. For this task, after experiencing the
same problem with tryint to add a mailbox to the mailbox listbox that
only allows one mailbox, the tester returns to the projects tab and
answers the question about discarding messages, which activates the
filters and processing tab.
The tester switches to the filters tab and enters a name into the
current filter entry field; because of my programming error, this field
accepts when it should not.
The description of the task was sufficiently vague that the tester
doesn't know what to put into the "subject matches" entry field. After
getting a little prompting, he creates the filter, goes to the generate
tab, and pushes generate, finishing the task.
Discussions with the tester showed that he wasn't familiar with either
glob matching or regular expression matching. He suggests a somewhat
richer set of possibilities, including "subject equals" and "subject
contains." The tester wasn't sure why filters were not added
automatically; I explained how the order of filters can make a
difference in the results.
It was not obvious that a new filter name was required before adding the
filter to the filter list. The tester suggested putting the "Add"
button at the bottom instead of at the top to make it more clear that
the filter is defined, and then added.
This test also revealed a bug that allowed a filter with "" as the name
to be created.
It is also clear that the generate tab should be grayed out if filters
are required but not defined. It should also not be active until
filters are applied.
Tester 3, test conducted by Dang and Cargo.
Listening to Dang give the introduction, I am struck by the realization
that what we are calling an "HTML archive" might not be a meaningful
term to our intended users (not that I know what is, at this moment). I
think we need a better term and perhaps our application needs a better
name.
The tester sees the project page, decides that it has too much text on
it, and switches to the archives tab. Here the tester gets confused.
Dang intervenes and directs the tester to the mailboxes tab. Here the
tester sees the default mailbox name in the entry field, finds the "Add"
button, presses it, sees that the mailbox was added to the mailbox
listbox, and then proceeds to the generate tab and presses the generate
button, completing task 1.
In trying to perform task 2, the tester seems to have trouble
understanding the task. He tries changing things on the generate tab.
Eventually, he goes to the project tab and tries to change the working
directory, and not to the mailbox tab to change things there.
With coaching, we are able to get the tester through task 2. The user
runs into the one mailbox limit; in some sense, this is a memory load
problem, since none of the testers really remembered accepting that
limitation on the first tab.
After the tests, the tester said that the most confusing thing was that
the tabs seem to imply no fixed order; experience showed that the order
was not fixed, but that there were interdependencies between the tabs.
It wasn't clear to this tester what was necessary and what was not.
Again, making the generate tab smarter about being grayed out or not may
be appropriate.
After this last test, the tester asked more about what our application
actually did. When we should him some of the indexes and pages produces
using our intended back end, which can be found linked to the page
<http://www.skypoint.com/members/escargo/csci5110/>, he said that he saw
how the results could be useful to the right users.
dsc