Evidence of quantitative data is highly prevalent in our lives. Personalized algorithms on YouTube and Netflix churn out one stellar recommendation after another, keeping us hooked.
It’s important to remember though that the biggest (and best) companies invest in quantitative and qualitative research. Each methodology uncovers different insights about users.
A recent clamor for data scientists indicates that industries lean heavily on quantitative research. But can an over-reliance on this method stifle product innovation?
Rewind to before the pandemic hit, circa 2020. Educ8, a fictional tech company, was developing an online classroom application. The app aimed to eliminate the need for teachers and pupils to be in the same room. Suddenly, COVID-19 became rampant across the globe, forcing everyone to stay home. Although the app was incomplete, Educ8 sensed an opportunity to accelerate the rollout of their product.
What are the Pitfalls of Quantitative Research?
The fictional Educ8 decided to conduct user research to refine its product before launch. The research team proceeded with quantitative methods. Steeped in statistics, it’s often perceived as more “scientific and reliable.” They rolled out a survey to various students, schools and colleges in their home state. It included a link to a beta version of the app, accompanied by a survey where respondents had to rate various aspects of the app on a scale of 1 to 5 (easily quantifiable).
The survey revealed that over 70% of respondents accessed the app through their personal computers. Educ8 used this as the basis for further development of their app — they catered the experience more toward desktop users. They used an unrepresentative sample to make inferences about its users. A colossal mistake — instead of their app being accessible to all, Educ8 neglected to make the app user-friendly on mobiles. They forgot about children in developing nations, who would access the app from their parents’ smartphones.
Scientific, but not reliable. Any errors in the experimental setup (like Educ8, above) or researcher biases can render the results of a study invalid.
Quantitative research aims to support or refute a hypothesis with mathematical logic. Researchers at Educ8 theorized that pupils would score similar on tests using their platform versus in-class sessions (pre-pandemic). Students were given a test after using the app to see how well they retained taught material. Researchers were looking for statistical significance, the likelihood that the data represents the actual picture, or whether results are due to chance. For their study to be statistically significant, Educ8 needed (a) a large number of responses (greater than 30), and even then, (b) respondents may have provided incorrect or incomplete information or (c) the survey may have been biased. The research team at Educ8 could not wholeheartedly rely on the survey results, as responses likely did not represent the population as a whole.
Researchers at Educ8 noticed that their beta app is rated decently well, but an in-class voting feature received a poor score, with an average of 1.2 out of 5. Quantitative data has helped them identify a problem. Now what?
As their responses were limited to assigning a score, respondents couldn’t explain why they rated this feature so poorly. Quantitative data is descriptive, but we encounter difficulty interpreting the results meaningfully.
Without qualitative research data, product teams at Educ8 were in the dark about what to modify about this feature. Future errors in development can then arise from a lack of clarity.
What are the Pitfalls of Mixed-Method Research?
Consequently, Educ8 decided to employ mixed-method research — a complementary approach that seemingly provides better results than one procedure in isolation. It is an extension of quant and qual research rather than choosing just one. Researchers at Educ8 refined the survey by including open-ended questions in addition to purely numerical ones.
It is vital to consider what insights you gain by employing mixed-method research. Sale et al identify the two as different paradigms, each studying differing phenomena. Quantitative researchers look at the truth objectively and view it as separate from the observer, whereas qualitative researchers look at reality through peoples’ experiences, a constantly changing reality.
“If we increase the price of Educ8 by 20%, what will be the impact on customer retention?” is a clear quantitative question.
“What aspects of the in-classroom environment do students and teachers alike need replicated in the application?” is a more open-ended, qualitative question with no definitive answer. The authors in the study linked above argue that mixing the two diminishes the value of both methods.
Driscoll et al found that when rich qualitative data is enumerated or coded into quantitative data or variables, it suffers a loss in depth and flexibility, rendering it “one-dimensional and immutable.” Imagine trying to quantify the nerves students feel when they give a presentation to their peers.
Several barriers exist to the widespread adoption of the mixed-method research approach:
- Data integration & time required – making the two types of data compatible (coding) while answering the original research question is challenging. Researchers need a lot of time to process and make sense of both data types. Teams at Educ8 had to pore over qualitative responses to understand what elements of a regular classroom are missing from their app.
- Data access & rigor – researchers need to consider what data they have access to and the reliability and validity of their data throughout the process. A respondent gave the voting feature a score of 4 (out of 5). However, they were highly critical in the free response question. How does the researcher reconcile the two?
- Researcher skills, knowledge & methodology – researchers may not be highly proficient in both methodologies, i.e. their skills may be better suited to one. Researchers construct qualitative studies. Thus, coding qualitative data can be highly subjective and vary from one person to another.
- Budgetary constraints – all companies have finite resources (people, time and money) and must decide how to allocate these.
What does this all mean for a company like Educ8? The researchers would need to decide what question is most urgent. They would then choose the research tactic that best answers that specific question. In this hypothetical world, the voting feature was rated unfavorably after the first survey. Researchers at Educ8 sent a follow-up, open-ended questionnaire to understand better understand the students’ experience.
How to Unlock Valuable Insights from Qualitative Research
Educ8 researchers found themselves at a stage where they wanted to understand more about their users and their interaction with their app. Quantitative research tends to be explanatory, a snapshot of a phenomenon. For example, the survey revealed that 90% of respondents had a largely favorable experience using the Educ8 app. Or that a majority of them rated the voting system poorly.
Most exploratory research is qualitative. Qualitative research aims to discover and unearth user problems, investigate why they face difficulty, and understand how to remedy the situation. Testing is typically carried out with 6 to 8 users in small focus groups to enable a rich understanding of user experience and pain points. The qualitative study reveals why Educ8’s voting feature is confusing and tough to navigate – students are overwhelmed by the number of steps required to vote. This insight equipped the product team with the knowledge to improve app functionality.
Another benefit of using qualitative research? It can provide near-instantaneous results, saving time in the process. Qualitative studies do not need statistical significance – you only need a few responses to understand a user problem.
Imagine a small town evaluating road safety at a four-way intersection. Only a few instances of road accidents identify that it is a problem. They resolve to install stop signs in all directions so that travelers proceed cautiously. Problem solved. Similarly, Educ8 need not interview 30 people to identify issues with their online voting feature – if 5 out of 8 people find it confusing to navigate, chances are, this extends to the wider population too.
Faster insights enable Educ8 to refine its voting system and roll out enhancements to make students’ and teachers’ lives easier. This can be the difference between retaining customers or losing them to a competitor such as Google Classroom.
User Research Should Drive Your Product Roadmap
Meticulous user research is the key behind all successful products. And the best research roadmaps combine quantitative and qualitative data to give you a clear landscape of what your customers really need.
Companies with an over-reliance on quantitative research can miss underlying reasons for user problems and behavior. Researchers focus on testing theories rather than generating new ones or getting to the bottom of user issues. Purely quantitative methods lack the detail that explains the numbers.
Ideally, Educ8 would have conducted copious amounts of quantitative and qualitative research, ensuring that their app was fully equipped to hit the market. However, the requirement dramatically changed with the landscape — suddenly, it went from being a studying supplement to being the only method available to students.
Moreover, companies do not have infinite time and resources to conduct research. Knowing the research questions you need answers to, what stage of product development you’re at, and your resource limitations helps you decide what methodology to use.
All forms of research reveal more about customers, so you need to be clear about your objectives when you begin. Successful studies continually generate new questions — which means the research wheel never stops turning!
Want to learn more about the bright future of qualitative research? Watch Why Qualitative Research is the Linchpin of Better AI to learn the principles behind building digital experiences that transform our lives for the better.
Photo by Joshua Sortino on Unsplash
