Lo-to-Mid Fidelity Prototype Testing and Refinement

Write-up Due: Thursday, April 25th at classtime

Assignment Overview

In this assignment you will:
  1. Iterate on your lo-to-mid fidelity prototypes
  2. Use your prototype to conduct usability tests of your three tasks with representative users
  3. Refine your design based on feedback and lessons from these tests
  4. Write-up your evaluation method, study, and findings, design revisions

What to Do

1. Prototype Iteration

Based on the results of your last assignment (“Task Refinement, Storyboarding, and Video Prototyping”), iterate on your lo-to-mid fidelity prototypes. Remember, your lo-fidelity prototype should convey all critical aspects of your design and support the testing of at least the three tasks you have devised (easy, medium, and hard task). Consider using an inspection method (e.g., a discount usability method such as Heuristic Evaluation) to iteratively improve your design prior to testing.

If you found from the video prototyping assignment that your tasks and/or scenarios were lacking, then improve them before conducting your usability study. You should describe the improvements to your designs and/or the changes to your tasks/scenarios in your report.

2. Prototype Testing

Find at least four participants to work through your tasks using your lo-to-mid fidelity prototype. You should not use friends, members of the class, or people you know to have engineering backgrounds. Nor should you use people who have been previously exposed to your project. Target an appropriate set of people based on your design and your contextual inquiry.

Before running your lo-to-mid fidelity usability experiment, each team member should watch the Nielsen/Norman Group “Paper Prototyping: A How-To Video” to refamiliarize yourself with the study process (link). Pay careful attention to how the experimenters in the video conduct the usability study. For example, note that there are always two experimenters present during the study. This is important as one experimenter will be occupied with conducting the study while the other experimenter can take notes on observations.

At the beginning of your usability evaluation, have one of your teammates demonstrate your system to the participant. For example, show the participant how to interact with your design. Do not show your participants how to perform your tasks. Instead show them how paper prototyping works, how your system generally works, and give an example of something specific that is sufficiently different from your tasks (you don’t want to bias them by showing them how to conduct one of your research tasks).

During the experiment, make a log of critical incidents (both positive and negative events). For example, a person might make a mistake or might comment on something they like. Write it down along with a description of what was going on. Later you should discuss and prioritize these events and assign severity ratings to problems (ranging from 0 for no problem to 4 for usability catastrophe).

Bonus Points Opportunity! For 25 extra points, you can video record your experiments (similar to the videos taken in the Nielsen Norman Group link above) and submit an edited version along with your write-up. The video must be no longer than two minutes and must highlight important findings uncovered during your evaluation. Note that you must receive each participant’s written consent to allow for videotaping. This written consent form should be submitted along with your write-up. Before taking the video, you should ask whether the participant would prefer that his/her face be hidden. If yes, shoot the video from behind the participant. If you need video equipment, email me.

In order to qualify for the bonus, you must submit a half-to-full page write-up describing the video process and the advantages/disadvantages of videotaping a user study. One other incidental bonus of videotaping here is that you will have additional footage for your final video report.

3. Prototype Iteration

Refine your design based on the results of your testing. If necessary or appropriate, develop new or revised tasks and new or revised scenarios. Your tasks and scenarios are likely stable at this point, but be sure to revise them if your testing reveals a need.

What to Hand In

You will submit a report of no more than 5 pages of text, approximately 2500 words. Images are strongly encouraged, do not count against the page limit, and are thus effectively free (the limit applies to the approximate amount of text you would have if all images were removed). Your submission must be in PDF format and linked to on your team wiki.

Your report should follow this outline, and will be graded using the guidelines discussed at the end of this document. The provided page allocations are rough estimates, to help convey how to divide up the space.
  1. Project title
  2. Each team member’s name and their role on this assignment
  3. Problem and solution overview (short, 1 paragraph, iterated on since the last group assignment)
  4. Lo-to-mid prototype description, with overview and close-Ups (1 page associated text)
  5. Testing method
    a. Participants (1 paragraph text)b. Study Environment (1 paragraph text)
    c. Tasks (.5 page text)
    d. Procedure (.5 page text)
    e. Test Measures (1 paragraph text)
  6. Testing results (1 page text)
  7. Interface revisions (0.5 page text). These revisions (be it on paper, in Balsamiq, or some other medium) should clearly show previous design and new design. Provide rationale for changes made as a result of testing your lo-to-mid fidelity prototype.
  8. Summary discussion and lessons learned (0.75 page)
  9. Optional: Video report (0.5-1 page). This is required if you do the bonus.
  10. Appendices (as many pages as necessary, link from text into the appendices)
    a. include all forms handed out to participants (e.g., instructions)
    b. include raw data (cleaned up and readable)
    c. include any extra figures that do not fit in the body
    d. include any additional sketches or screenshots of prototype (as many as needed)

Examples of Previous Projects

Be forewarned, the examples on this webpage are for a different version of this class. For example, the prototypes used in their intermediate user testing assignment (analogous to this one) had to be paper prototypes. My constraints are looser—you can use mid-fidelity mockup tools if you want. With that in mind, you should examine these projects and see how they put their reports together. Improve on these. UMD can do better than UW.

http://www.cs.washington.edu/education/courses/cse440/09au/projects.html

When you click on the above link, you’ll be taken to a list of project teams. You can click on each project team to go to their project website where you are able to download and view all of their deliverables. Please feel free to do so but focus your attention on the “low fidelity prototype + testing” reports.

Writing Guidelines and Grading (130 pts)

Overall Writing Quality (10 pts)

Make sure your writing is easy to read: ensure it is clear and concise, use section headings, make liberal use of whitespace, include images in the body of the write-up with appropriate figure numbers and captions, refer to the figures in the body of your text, and check for grammatical errors.

Overview of Problem and Solution (10 pts)

This overview should be a concise statement of the problem you are tackling and a brief synopsis of your proposed solution. This should be a refinement of the problem/solution portion from your last assignment (we iterate nearly everything in this class!).

Lo-to-Mid Fidelity Prototype (20 pts)

Describe your prototype. What are the main pieces of functionality? What are the main interaction ideas? How does a person interact with it? Reference images of the interface in your description. Include at least one picture of the entire interface design with all of the elements laid out.

Testing Method (20 pts)

Describe the participants in the experiment and how they were selected. Also describe the testing environment and how the prototype and any other equipment were set up.

Describe some relevant details of your testing procedure. This should include the experimental roles of each member of the team. To prepare, you should assign team members to the different tasks (i.e., computer, facilitator, observer) and practice with someone playing the participant.

The test measures detail what you looked for or measured during the experiment. You should concentrate on process data (i.e., what is happening in the big picture) in addition to bottom-line data (i.e., time or number of errors).

Testing Results (20 pts)

Summarize the results of the experiment from your process data. What did you learn in testing?

Interface Revision (20 pts)

Discuss the appropriate changes that you made to address the important problems discovered during testing. Provide a rationale for changes, and clearly illustrate them.

Discussion and Summary (20 pts)

Reflect upon and discuss your project and your results. What did you learn from the process? How did the process shape your design? Is there something you think you did not uncover? How could you improve your usability evaluation method in the future (i.e., what could you do differently?).

Appendix (10 pts)

The appendix should include copies of all materials involved in the experiment. This includes your consent form (if you used one), demo script, and any instructions or task descriptions you handed out or read aloud to your participants. Finally, it should include all the raw process data you gathered during the experiment. Clean it up to make it easier to read. Merge the critical incidents logged by the observers and list them.

Remember, the appendix materials and screenshots do not count in your page limit.

Optional: Video and Video Report (25 bonus pts)

I will be far stricter in my grading of these bonus points than I am of the other sections. The writing quality here has to be high (clear and concise) with no grammatical errors. The report should include a description of how the videotaping was conducted, whether it was disruptive, if anything additional was learned about the interface design and participant reactions by watching the video post-hoc, the process used to edit the video, and general lessons learned about videotaping studies.

The video itself should clearly introduce the four participants and how they were recruited, the low-to-mid fidelity prototype, and the three tasks. The video should then show highlights of the experiment itself. The video should be appropriately edited and paced using an overdubbed narrative or subtitles to help the viewer understand the prototype and the study. As always, points will also be supplied for creativity.

Submission

Post your deliverable to your team website.