Final Write-up: Interactive Prototype and User Testing

Write-up Due: Saturday, May 11th, 8AM (~17 days from now!)

Assignment Overview

In this assignment you will:

  1. Revise your interface based on findings from the last assignment.
  2. Use these revised interfaces to build an interactive prototype of your application. This can be done in the programming language of your choice based on the needs of your application.
  3. Use your interactive prototype to conduct usability tests of your three tasks with three representative users.
  4. Suggest changes to your interactive prototype based on feedback and results from these tests.
  5. Write-up your evaluation method, study, and findings, and suggested design revisions.

What to Do

1. Revise Interface Based on Results from Last Assignment
Revise your interface based on the usability tests you conducted of your low-to-mid fidelity prototypes from the last assignment. You may also revise interfaces as you implement the interactive prototype (as certain constraints or usability issues become clearer).

2. Interactive Prototype

For the interactive prototype, you must implement at least enough interactive functionality to make it through each of the three tasks (easy, medium, hard) that you have been iterating on throughout the semester. The interactive prototype must be interactive—this interactivity may be achieved in many ways from Adobe Flash to an actual implementation in Java, C#, Javascript, Ruby or another language of choice. If you are building a mobile application, your prototype may still be Flash-based (with a mobile phone interface template). If you want to build your interactive prototype for a specific system (e.g., Android), you are welcome to test and demo your interfaces in the simulator (if you don’t have access to an Android device). Regardless of implementation language, the prototype should look far more high fidelity than your previous prototypes. Visual design, color choices, clear visibility and feedback are all important. Remember to follow the design principles we learned in class.

As an HCI class, it’s important that your implementation effort be focused on creating and testing interactions and interfaces. This focus paired with the practical limitation of having a very short period of time to complete this prototype means you will likely need to make difficult choices about what to implement. My suggestion is to focus on showing and polishing only what is essential. If you decide to go beyond a task-oriented fixed-path prototype (e.g., by implementing a live database, location sensing), you do so at your own risk. There is relatively little upside to working on backend functionality (i.e., parts of the program which are not user facing). The time required to actually implement such things will not excuse a failure to complete your prototype before the deadline.

3. Interactive Prototype Usability Test

Similar to the last assignment, you will be conducting usability tests of your prototype. You should largely follow the same setup as before. Each study session should be conducted by at least two experimenters: one experimenter is in charge of running the study and the other should take notes and observe the interactions. With the consent of your participants, you can video record these sessions and use these videos in your final presentation. Note: the video angle used should protect the identity of your participants (i.e., at no time should their faces be visible). You can use the video recordings in your final video. You can download the consent form template here and instructions here.

Find at least three representative users to work through your three tasks (easy, medium, hard) using your interactive prototype. You should not use friends, members of the class, or people you know to have engineering backgrounds. Nor should you use people who have been previously exposed to your project. Target an appropriate set of people based on your design and your contextual inquiry. These users cannot be the same as the participants you used in your low-to-mid fidelity tests.

At the beginning of your usability evaluation, have one of your teammates demonstrate your system to the participant. For example, show how to interact with your design in general. Do not show your participants how to perform your tasks (you don’t want to bias them by showing them how to conduct one of your research tasks).

During the experiment, make a log of critical incidents (both positive and negative events). For example, a person might make a mistake or might comment on something they like. Write it down along with a description of what was going on. Later you should discuss and prioritize these events and assign severity ratings to problems (ranging from 0 for no problem to 4 for usability catastrophe).

4. Recommend Changes

Based on your observations and an analysis of the critical incidents, suggest at least three primary changes to your interface that you would make before launching your product. Prioritize your suggestions based on the severity of the problem and the importance to the functionality.

5. Bonus! Create Project Website

You can earn up to 30 bonus points for creating a well-designed, aesthetic website for your project. For past examples, please see http://www.cs.washington.edu/education/courses/cse440/09au/projects.html and click on the individual project names (e.g., .calm, Consumester, Fitter). You should list this website at the top of your wiki project page. At a minimum, your website should include:

  1. A one sentence mission statement
  2. Links to the various deliverables in class (e.g., documents, presentations, videos, etc.). Do not forget to link the new deliverables: final write-up, final presentation, final video.
  3. A small “about us” section that describes who you are, your degree program, and your primary roles on the project.
  4. If relevant, a link to your interactive prototype

In addition to the list above, you might also want to include a section on differentiating your application from competing products, a section highlighting the various inspirations for the project, and/or a section enumerating the tools used to create your mockups and prototypes throughout the semester. It is up to you on how you design the information architecture of the website (e.g., the number of individual pages, content embedding, etc.).

Note that a creative, visually appealing, and well-designed website (see, for example, getOut or StoneSoup) will earn more points than a bland, pedestrian website such as Fitter or Foodwatch. Remember, as you look at these websites, that these projects are based on a different version of this class taught at UW and may have had different requirements.

What to Hand In

You will submit a report of no more than 8 pages of text, approximately 3500 words. Images are strongly encouraged, do not count against the page limit, and are thus effectively free (the limit applies to the approximate amount of text you would have if all images were removed). Your submission must be in PDF format and linked to on your team wiki (and the bonus website if you choose to create one).

Please carefully layout your pages such that prose and figures are neatly aligned and integrated. Remember, I love figures and find that they can greatly increase the understandability of your documents (this is HCI after all); however, the images must flow well within the document. Each figure should have a figure number and caption and the figure itself should be referenced from prose. You cannot simply dump images randomly into your documents and expect “good layout” to just happen. You have to work on this.

Your report should follow this outline, and will be graded using the guidelines discussed at the end of this document. The provided page allocations are rough estimates, to help convey how to divide up the space.

  1. Title page (does not count towards page limit):
    1. Project title
    2. Short value proposition or mission statement (e.g., “.calm aims to support those living with anxiety by encouraging self-awareness through the use of cognitive behavioral therapy”).
    3. Each team member’s name and their role on this assignment and their primary roles on the project as a whole. This should probably be two-to-three sentences per person.
  2. Problem and solution overview (1-3 paragraphs). You have now iterated on versions of this since the first team assignment. This should feel solid—no grammatical mistakes. The text should be polished.
  3. Background and review of past work (i.e., competing products) relevant to your application (1 page)
  4. Description of interactive prototype (1-2 pages).
    1. Overview of interactive prototype (e.g., general interface structure, programming language used to build it). Clearly delineate what is implemented and what is “faked.” Some teams are using mock-up data to simulate sensors. This should all be clear. (0.5 page)
    2. Describe the interactions and interfaces for each of the three tasks. You can use accompanying figures to help with this (e.g., sequence diagrams). (0.3 – 0.5 page per task)
  5. Testing method (1-2 pages)
    1. Participants (1 paragraph). Tip: pictures of participants using the interface will help your text seem more engaging. You should angle the camera to protect the identity of the participants.
    2. Study environment (1 paragraph). Tip: A picture of the study environment(s) helps the reader better understand where the tests were conducted.
    3. Tasks (.5 page). Tip: you can refer back to figures in the previous section.
    4. Procedure (.5 page)
    5. Test measures (1 paragraph)
  6. Testing results (1 page text)
  7. Suggested interface revisions (0.5 – 0.75 page text) based on tests. Provide rationale for suggested changes made as a result of testing your interactive prototype.
  8. Summary discussion and lessons learned (0.75 page). Please reflect on the entire project as a whole (not just this assignment).
  9. Optional: Website report (0.5 – 1 page); does not count towards page limit
  10. Appendices (as many pages as necessary, link from text into the appendices)
    1. include all forms handed out to participants (e.g., instructions)
    2. include raw data (cleaned up and readable)
    3. include any extra figures that do not fit in the body
    4. include any additional sketches or screenshots of prototype (as many as needed)

Examples of Previously Successful Projects

Be forewarned, the examples on this webpage are for a different version of this class taught at UW. The UW class was 10-weeks whereas our class is 16-weeks. Each team assignment also differed in a number of ways. However, there is sufficient similarity between the two to warrant looking at them:

http://www.cs.washington.edu/education/courses/cse440/09au/projects.html

When you click on the above link, you’ll be taken to a list of project teams. You can click on each project team to go to their project website where you are able to download and view all of their deliverables.

Writing Guidelines and Grading (200 pts total)

Overall Writing Quality (20 pts)

Make sure your writing is easy to read: ensure it is clear and concise, use section headings, make liberal use of whitespace, include images in the body of the write-up with appropriate figure numbers and captions, refer to the figures in the body of your text, and check for grammatical errors.

Overview of Problem and Solution (20 pts)

This overview should be a concise statement of the problem you are tackling and a brief synopsis of your proposed solution. This problem and solution statement should be written with regards to the entire project as a whole, not just this assignment. Think back to the first team assignment where I asked you to write an introduction that “clearly articulated the problem and why it is important.” You should be sure to describe how your proposed solution is novel or, at the very least, how it may extend existing solutions.

Background and Review of Past Work (20 pts)

This section should provide a background context on the problem as well as a review of past work. I expect at least eight specific references to either related commercial products or research papers. When relevant, you should include a figure of the related application, tool, product, or webpage with a numbered figure caption below. When referring to past work it’s important to include a description of the past work, why it’s relevant and significant to the proposal at hand, and how your proposed solution is different. It’s absolutely crucial that this section is written well; feel free to start with your “background” section from team assignment #1 as a foundation and iterate it for your final report.

Interactive Prototype and Description (50 pts)

Describe your prototype. What are the main pieces of functionality? What are the main interaction ideas? How does a person interact with it? Reference images of the interface in your description. Include at least one picture of the entire interface design with all of the elements laid out. Make sure to describe any changes or new features since the last assignment.

Testing Method (20 pts)

Describe the participants in the experiment and how they were selected. Also, describe the testing environment and how the prototype and any other equipment were set up. Describe the relevant details of your testing procedure. This should include the experimental roles of each member of the team. The test measures detail what you looked for or measured during the experiment. You should concentrate on process data (i.e., what is happening in the big picture) in addition to bottom-line data (i.e., time or number of errors).

Testing Results (20 pts)

Summarize the results of the experiment from your process data. What did you learn in testing?

Suggested Interface Revisions (20 pts)

Recommend specific changes to the interface based on your results. If relevant, include a sketch or mockup of the change (you are also welcome to implement this change and take a screenshot of your interactive prototype, but this is not necessary).

Discussion and Summary (20 pts)

Reflect upon and discuss your interactive prototype and your results. What did you learn from the process? How did the process shape your design? Is there something you think you did not uncover? How could you improve your usability evaluation method in the future (i.e., what could you do differently?).

Also, include a subsection on discussing the successes/failures of the project as a whole.

Appendix (10 pts)

The appendix should include copies of all materials involved in the experiment. This includes your consent form (if you used one), demo script, and any instructions or task descriptions you handed out or read aloud to your participants. Finally, it should include all the raw process data you gathered during the experiment. Clean it up to make it easier to read. Merge the critical incidents logged by the observers and list them.

Remember, the appendix materials and screenshots do not count in your page limit.

Optional: Project Website and Write-up (30 bonus pts)

I will be far stricter in my grading of these bonus points than I am of the other sections. The writing quality here has to be high (clear and concise) with no grammatical errors. The report should include at least one screenshot of the website (along with a link) and a description of how the website was designed and implemented. Make sure to document what kind of tools were used to make the website, how the website was tested across web browsers, and how the website was constructed (e.g., html5, javascript, css3, bootstrap, etc.).