question mark in electric lights

Win with Research: When to Use Quantitative vs. Qualitative Research

The goal in product research is to establish the most accurate interpretation of your users’ needs. To do that, you need to know which approach to take — and why.

It’s impossible to discuss qualitative research without considering its quantitative counterpart. Several misconceptions exist about the interaction between the two — so let’s dispel them.

The more dominant research methodology, quantitative is often viewed as a more scientific approach than qualitative research. Many smart people put the two approaches in direct opposition with each other. Even worse, an old school of thought leaves many only using qualitative methods as a last resort when quantitative analysis isn’t possible. 

Quantitative methodology relies on basic statistical and probabilistic techniques that have been refined and taught over centuries and focus on testing a theory. Qualitative research, a far less standardized field, is no less scientific. Although students don’t learn a “core”set of techniques, researchers formulate hypotheses or theories using resources of logic and set theory. 

“How do we improve buyer experience and therefore conversion rate?” is a fundamental qualitative question. “Changing the ‘Buy’ button from red to green will improve conversion rate” is a quantifiable theory that can be empirically tested. Qualitative research often precedes quantitative. It helps generate hypotheses to further understand quantitative data. They both employ scientific methods to generate valid causal influences.

Qualitative research uses “within-case” analysis to make inferences about an individual case, whereas quantitative research involves “cross-case” analysis to make inferences about a population. However, this doesn’t necessarily follow that qualitative research uses smaller samples than quantitative research. 

In “A Tale of Two Cultures,” Goertz and Mahoney reject the belief that the quantitative vs. qualitative argument boils down to use of “numbers vs. words”. Instead, they suggest that quantitative and qualitative are two different cultures. Social scientists organize themselves into two different communities, and lean on different scientific fundamentals to dictate their research.

The two methods are not incompatible or mutually exclusive either. The authors seek to increase awareness and communication between the two fields to facilitate understanding of strengths, weaknesses and use cases. 

When you begin research, it’s important to understand which methodology is best suited to the study.

Read on to discover the strengths and weaknesses of qualitative and quantitative research. and important differences between the two.

“Two Distinct Cultures of Research”

QualitativeQuantitative
Scientific FoundationLogic & Set TheoryProbability & Statistical Theory
DescribesIndividual experiences, behaviors and beliefsCharacteristics of a population
AssumptionsAssumes a dynamic and negotiated realityAssumes a fixed and measurable reality
Study DesignResearchers alter study direction and design as they go alongDesigned before study commences
When to UseInductive – focuses on developing a theory when no hypothesis existsDeductive – focuses on testing a theory; use hypothesis to support or reject theory
Research GoalsDescriptiveNormative / Prescriptive
Data Collection  MethodsSemi structured – In-depth interviews, focus groups and participant observationHighly structured – Standardized questionnaires and surveys
Question TypeOpen ended questions – responses affect follow up questionsClose ended questions – responses do not affect what researchers ask next
Data AnalysisData analyzed by themes from participant descriptionsStatistical analysis – descriptive and inferential statistics
OutputDetailed information on a small sample – rich understanding, but reduced generalizabilityBroad, measurable and generalizable set of findings
Criteria to evaluate dataCredibility, transferability, authenticity & confirmabilityInternal & external validity, reliability, objectivity

Qualitative Research

AdvantagesDisadvantages
Detailed insights in a descriptive , narrative style. Provides an insider view of users needs and pain points missed by quantitative data. (Lack of) Generalizability as the data focuses on a small sample or isolated context, and therefore may be biased and not representative of the population.
Idea generation – open doors for new opportunities and product enhancements to improve customer experienceSubjectivity due to the researcher’s role in designing and carrying out the study – interpretations of the same data can vary
Flexibility – design of study can be altered as new data emergesNot replicable – as the research is filtered through the researcher, no two studies can be the same
Natural setting – Data collection occurs in a real world contextLabor & time intensive – it takes a seasoned researcher 8-10 hours to transcribe an interview of 20-30 pages of dialogue. [Until now! Marvin frees up time so your team can focus on what’s important.]
Causation – suggests possible relationships and allows for ambiguities and contradictions in data (portrays a more accurate reflection of reality)Can fall prey to observation bias (3 types):Hawthorne effect – participants change their behavior when they know they are being observed
Observer-expectancy effect – participants change behavior to satisfy researcher’s desired effect
Artificial scenario effect – information collected is not accurate

Quantitative Research

AdvantagesDisadvantages
Quantifiable – easy to enumerateSample size – a high chance of greater variability of data in a small sample; need large amounts of data (time + money)
Objectivity – rooted in mathematical logic; rationalContext – do not occur in natural settings
Tests and validates already constructed theories(Lack of) Detail – Participants cannot explain their choices
Rapid analysis – a multitude of tools make our lives easier to process large amounts of dataCan fall prey to confirmation bias – researcher misses phenomena due to focus on theory / hypothesis testing, rather than theory generation
Replicable – numeric data is less open to interpretation; not reliant on researcher

For some research goals, quantitative methods are more appropriate, and for others, qualitative. Depending on the task, the researcher may combine both quantitative and qualitative methods to reach their goal. This is called a mixed-method approach.

Embedding Qualitative Research into Your Company Culture

“Say Cheese”, a young company selling vegan cheese, wants to boost customer awareness and sales by increasing its YouTube following. The marketing team decides to create a “how-to” recipe series, detailing numerous dishes that customers can prepare at home with the product. The first five videos fare well, and drive traffic to their website. Lately though, viewership numbers have dropped and the initial surge in sales is a distant memory.

To diagnose the reasons behind the decline, teams at Say Cheese look at both quantitative and qualitative data.

Quantitative data revealed that the view count, likes and subscriber numbers grew steadily before falling dramatically. What explains the drop-off though? A qualitative dive into user comments reveals that many subscribers suggested recipes to try out in future videos. Marketing executives at Say Cheese missed these insights and failed to tailor new content to keep users engaged.

This example, albeit simplistic, illustrates the importance of qualitative research. Everyone should conduct some form of qualitative research, as it gives you a more rounded profile of customers. Imagine understanding why customers opted for a competitors’ products instead of yours. The information can be used to enhance their experience in an effort to win them over. Qualitative data provides actionable insights – so how can you help introduce it as a core competency in your product team?

No matter what stage you are on your product roadmap, qualitative research should be part of the discussion. Here are a few examples when you can leverage qualitative research: 

  • New product – Whether you’re generating an idea for a new product or developing one, gathering qualitative data about your customers’ needs is imperative in creating a winner. Think, the Apple executives discussing customer needs for the revolutionary iPhone. 
  • Brand perception – It’s crucial for companies to understand how consumers perceive their brand. Apple executives understand their products are premium priced and aimed at creative professionals – their marketing and branding is positioned with these principles in mind. 
  • Consumer behavior – Understand more about your demographic and their purchase behavior – what choices do they have? What are their buying preferences? 
  • Marketing – Study your marketing projects to understand strengths and weaknesses. What are users saying about your products? Researchers analyze consumer reaction to marketing campaigns to measure their effectiveness and reach. Enhance your marketing strategy and don’t miss any meaningful insights like the marketing team at Say Cheese. 
  • Highlight pain points – Identify obstacles to purchase, user issues or missing features. If you’ve ever filled out a feedback form for an app, the comments section is usually open-ended. Users can describe their pain points at length, and companies act on this information by rolling out usability enhancements. Ever wonder why customer care calls are “recorded for training purposes only”? These recordings are analyzed to see how customer issues are being resolved, and if any improvements can be made to the process.

Qualitative studies must be well-designed and planned meticulously, with methodology clearly defined to eradicate research bias. Here’s how to get started:

  1. OBJECTIVE – define your problem or interest area
  2. PLAN – identify the target population; establish adequate data collection and analysis methods and document your methodology. 
  3. COLLECT data – ‘memoing’ is a process for recording a researchers’ thoughts and ideas as they evolve throughout the study. 
  4. EXPLORE and organize data – develop a data coding system; identify recurring themes.
  5. ANALYZE and categorize data – assign codes and describe details of each category. 
  6. PRESENT your findings to develop a theory or hypothesis. Increase collaboration and idea exchange by involving everyone in the process. 

Remember, the best research teams use a combination of quantitative and qualitative data, evaluating both empirical data and the user experience.

A New(ish) Approach to Research

The goal in product research is to establish the most accurate interpretation of your users’ needs and break down their biggest pain points into the best possible solutions. 

As companies ramp up their data and data science capabilities, their focus has been on the traditional statistical quantitative approaches. Every link we click on and the time spent on webpages is tracked to create a profile of who we are as consumers. A (relatively) new research method has emerged, one that aids in understanding the reasons behind consumer behavior.

Qualitative research humanizes quantitative data and elevates your customer voice. It offers insights into why users act and think the way they do. If you don’t use some form of qualitative research, you’ll miss a critical component from your research process. Qualitative research only makes up a meager slice of worldwide research today, but we (and many other experts) expect that number to grow exponentially. 

As a hark back to the humanities and social science origins of qualitative research, we leave you with a little philosophy in the transitive property:

Truly great products begin with a user problem. User problems are unearthed by rigorous research (both quantitative and qualitative). Therefore, great products are rooted in meticulous and detailed research. 

The answers are all out there. You just have to know where (and how) to look.

Ready to build user research into your product road map? Here’s a quick overview to get you started.

Photo by Emily Morter on Unsplash

withemes on instagram