QA and TestingMobile Strategy

Learn the Basics of Usability Testing


“A user interface is like a joke. If you have to explain it, it’s not that good.” –Martin LeBlanc

What is Usability Testing?

The act of building a great product is challenging because it requires the successful execution of multiple phases: discovering the right customer problem to be solved, designing a solution amidst an infinite number of choices, and building an artifact that admirably accomplishes its goals.

Before the digital age, evaluating how effectively your product was performing in these respects often required paying usability and/or domain experts to do the evaluation. Accordingly, this process is often called Expert Review (and sometimes Usability Inspection), and although it is still widely used, it has become displaced by Usability Testing as the primary product evaluation technique.

Usability Testing is the process of directly engaging real customers to evaluate the usability of products. The benefit of this approach over the expert-driven approach is that it provides an unbiased and direct examination of your product user experience through the eyes of the people that are using the product to accomplish a task. Research shows that, compared to Expert Review, Usability Testing is more likely to highlight issues with knowledge gaps (i.e. “How is this supposed to work?”) and task flows (i.e. “Which steps should I take to complete this process?”). In fact, there is only a 41% overlap in the problems highlighted between the two methods.

Types of Usability Testing

Usability Testing can be leveraged during the full product development lifecycle. When beginning to formulate the design of a product, it can be useful to understand how a customer would use it if it were to be built. Learning about opportunity areas early on allows an organization to avoid beginning costly development until the effectiveness of the design has been validated. Similarly, once a product has been built, it can be useful to perform the same type of testing when looking to make improvements or add new features to the existing product.

These types of “what if” testing scenarios fall into the category of Formative Usability Testing (sometimes called Problem Discovery Studies), where we are looking to answer the question “What is wrong about this design?” The term formative comes from educational research where it’s used to describe testing as a method for diagnosing problem areas in a student’s learning.

Once you’ve designed and built a product or feature, it’s important to understand how usable the finished version is. At this point, the question shifts away from “What is wrong about this design?” and towards “How useable is this finished product?” Techniques for assessing this information fall into the category of Summative Usability Testing (sometimes called Benchmark Studies) because the focus is on assessing outcomes and setting baselines.

Formative and summative testing complement each other well, with the learnings of each informing the other as the product moves through development phases.


Formative studies often occur early in the lifecycle of a product and focus on attitudinal data - that is, “what are people saying as they interact with a design?” As such, these studies typically encourage honest and descriptive user feedback, with the two main testing formats being:

  • Moderated - users give design feedback to a moderator in a live 1-on-1 conversation

  • Unmoderated / Remote - users leave design feedback via Usability Testing software and designers review that feedback later

Regardless of the format, common techniques are:

  • Tree Testing - users are shown an application’s navigation bar and asked where they would go to find a given piece of information, e.g. “Where would you go to find store hours?”
  • Card Sorting - users are asked to organize “cards” representing different screens of the application into logical groups, e.g. “About vs. Shopping.”
  • Wireframes - users are shown lo-fi static designs of the product and are asked how they would use it if it were interactive.
  • Prototypes - users interact with lo-fi clickable versions of the design and give feedback as they “use” the product.

In contrast, summative studies occur later in the product lifecycle and focus on behavioral data - that is, “How are people using this product?” As such, these studies typically focus on statistically significant methods for assessing how long certain tasks take or how many times certain interactions occur. Popular approaches are:

  • Click/Eye Tracking - software is added to the application to track where users are clicking/tapping/scrolling.
  • A/B Testing - two versions (A vs. B) of the website are tested as a controlled experiment for understanding whether there are behavioral differences between users who see the experiment version and those who see the control version.
  • Usability Lab Studies - behavioral metrics are tracked while users engage with the product in a live setting.

The Neilsen Norman Group has put together this great visualization showcasing the landscape of user research methods:

Screen Shot 2019-06-07 at 11.19.58 AM

When to Use Which User-Experience Research Methods

Usability Testing Tools

One of the reasons for the shift from expert-driven to user-driven usability testing is the proliferation of UX tools that allow organizations to directly interact with users in a variety of ways. Here is a quick rundown of popular tools per the different usability techniques mentioned above:

As for Usability Lab Studies, it’s not (yet) easy to leverage tools for this particular Usability Testing approach, meaning that organizations must tackle the process on their own. At WillowTree, we have seen major benefits from investing in our own one-of-a-kind Usability Lab for this purpose, which includes an observation room, a 2-way mirror, and a testing room with multiple cameras and microphones.

Running a Usability Testing Process

Broadly, the steps for running any usability testing study are:

  1. Decide what you want to test and create a hypothesis
  2. Determine the best technique for testing that hypothesis
  3. Create a plan with user goals and a standard for success
  4. Find participants and conduct the study
  5. Analyze the data and report findings

In conducting the study, consider these types of questions:

  • Demographics - what are the user’s background and interests?
  • Attitude - what do they say about your product and the problem it addresses?
  • Behavior - how do they use the current product and/or its newly design version?
  • Impressions - how do they feel about the planned changes?

We’ve put together a Usability Testing Checklist, which contains a set of detailed questions to consider during each phase of the Usability Testing process.


Usability Testing is a growing discipline and though it takes practice for an organization to deeply embed these techniques into its product development process, the benefits are numerous:

WillowTree has years of experience in doing user experience research and in performing Usability Tests with our partners and their customers, so if you need help improving your Usability Testing process, please drop us a line!

Join our team to work with Fortune 500 companies in solving real-world product strategy, design, and technical problems.

Find Your Role