Research and Insights

Three Considerations for Choosing the Right Method for Product Research

One of the greatest challenges in executing actionable research is identifying the right method for the questions you need to answer. At WillowTree we’ve worked with a variety of clients across financial services, media, D2C, and more — depending on their experience and needs, some have preference for qualitative methods like interviews or ethnographic observations, while others are more comfortable with quantitative approaches like surveys and experiments. It’s our job as research experts to help clients determine when alternative methodological choices are better aligned to the scope and demands of the project.

In this post, I share some tips for thinking through your methodological needs and the selection of quantitative or qualitative approaches. In a nutshell, you need to consider:

  1. Whether or not you already know what phenomena matter
  2. How you want to use the data
  3. What is feasible

Walking clients through this thinking can also help to create buy-in from key stakeholders and facilitate a more targeted approach to the partnership.

1. Known vs. Unknown Phenomena

One of the first questions that I ask in any research engagement is whether or not we already know the key factors or phenomena that will influence our findings. Qualitative approaches are usually best when you don’t already know what matters within a given space, context, or market. Although quantitative methods can discover new relationships between phenomena, we need qualitative methods in order to discover new things.

For this reason, we almost always include qualitative approaches in our innovation work - such as envisioning a new type of digital product or reimagining our client’s strategy for customer acquisition and growth.

Consider an example where we want to define a new consumer product to offer a loan at the point-of-sale. We could survey consumers, asking them to indicate the usefulness and delight of features included in current, competitive buy-now-pay-later (BNPL) experiences. But if we want to leapfrog over that competition, we need to know what consumers value, what they struggle with in their current payment experiences, and how they approach the purchasing journey. Though we may confirm the generalizability of the findings with subsequent quantitative work, we need open-ended qualitative exploration to discover the range of values, needs, and habits among consumers. Those insights allow us to innovate by solving for problems and contexts not accounted for in competitor products. It is through this kind of qualitative research that we are best able to provide a vision for consumer experience, enhance real-world experiences with digital ones, and generate a rich understanding of consumers and their needs and values.

On the flip side, quantitative methods are often an efficient choice when you’re tasked with identifying a relationship between known concepts, such how age influences technology adoption or product preferences.

2. Using Your Data: What, Why, Who, and Where

It is fairly common to hear researchers say that quantitative data can tell you what is happening, but it cannot tell you why. Although there are some exceptions, this is a useful way to think about the uses of data.

Quantitative data is great for telling you what is happening. For example, if you are tracking analytics in your new app, quantitative metrics will tell you which features exhibit high engagement and which features have low engagement. Assuming that you are tracking all of the users that you want to study, this kind of data has the added benefit of looking at your entire population (rather than a sample), meaning that you do not need statistical methods to figure out whether or not the patterns you are observing are generalizable.

Quantitative data can also tell you the who and where (if you’ve been careful to use sound sampling methods). Let’s say that you suspect that some consumers of enterprise software will be more willing to adopt a set of proposed features that you want to build. A quantitative survey can be generalized to tell you who is more comfortable adopting new technology (perhaps those who are more time-strapped, younger, and more technologically savvy) and whether those in certain workplaces or regions might be more likely to use your proposed tools.

However, if you want to know why you see these patterns, you will need to add qualitative methods to the mix. Returning to the example above, low engagement for a particular feature may not be a bad thing. It may be that the feature is so effective, it only requires occasional engagement to have the intended effect. Or perhaps a small percentage of users engage with that feature, but it is vital to their experience and the primary reason that they chose your product over a competitor. In either case, you would not want to cut that feature, despite its low engagement. On the flip side, low engagement may be due to a design flaw, but you would need qualitative work to identify the issue and allow for the possibility of fixing that flaw. Either way, pairing qualitative research with the quantitative evaluation is essential to making the right business decision.

3. Feasibility: What’s possible and practical?

Another factor that needs to be considered is the feasibility of collecting and analyzing that data (either quantitatively or qualitatively) and whether or not that data capture activity makes the best use of your allotted time and budget.

For example, if I wanted to understand how the concession experience related to purchase choices at movie theaters, I would need a data source that contained information about the experience (such as variety in products, average wait times), purchases made by consumers, and perhaps data about the consumers themselves. If you can access a pre-existing data set with that information, quantitative methods may be appropriate. However, if such a data source does not exist,I could observe movie-goers, recording their time spent in line, the number of people in their party, whether or not they have kids with them, how many items they purchased, etc. With enough observations I could certainly create a data set and analyze it for statistically significant relationships. However, investing the research hours needed for such an analysis is probably not the best use of resources. In this case, an interview study, aimed at unpacking the experiences, habits and decision-making process of movie-goers may be a more efficient path to providing directional insights to improve the customer experience. In this case, the goal is to right-size the research effort by balancing the resources required against the value of the insights a method can generate.

divider 600

We often find that clients come to us with a variety of questions and blind spots related to their product and users. The key to answering those questions effectively is to recognize that there is no single research method that outperforms the rest. Different tools are needed for different jobs, and it is our job to advocate for the right tool. Regardless of what methods you’ve used before, you can be certain that the WillowTree team will help keep the user at the center when finding the right methodological fit.

Looking for the right research methods to inform your product roadmap?

Get in touch with a researcher

A Step-by-Step Guide to Affinity Diagrams: Turning Chaos into Key Findings

User research often yields rich, qualitative data that can have direct impacts on...

Read the article