Blog posts

7 Reasons Qualitative Testing is Unscientific Message Testing

Newristics 18 July 2022

Lack of innovation in message testing research

To have a successful marketing campaign, it is essential to know what your target audience is thinking and how they will receive your marketing message. Message testing research can give you valuable insight into what impression your messaging is making on consumers. The ability to refine and optimize messaging based on actionable message testing feedback can help you win in your category.

Given the impact message testing research has on the success of the campaign, there should be continual innovation and progress in the field of message testing research. Yet, despite all the innovation seen in market research over the past decade, one type of market research that is still operating in the old model is message testing research. Market research for messaging testing has seen little to no innovation over the past 30 years. The fundamental approach used in qualitative testing is the same:

  1. Show messages randomly to respondents in a survey
  2. Ask respondents to rate/rank the messages
  3. Ask respondents to explain their ratings

Qualitative testing methodologies are riddled with challenges because they use scientific principles that have been disproven as a result of research into the field of human decision-making over the past few decades. Messaging decisions that influence billions of dollars in revenue are often based on the top 2 qualitative testing market research methods:

  • In-depth Interviews (IDI) - An IDI is a one-on-one conversation led by a market research interviewer and respondent. It is usually the first step for a B2B company. The sample may include 15-20 individuals and each interview may last between 30-60 minutes over the phone/internet or in person.
  • Telephone depth interviewing (TDI) - An TDI is also a conversation where 1 skilled interviewer interviews 1 respondent over the telephone. A company may do 12-60 interviews each lasting 30-90 minutes.

#1 reason to give up qualitative testing in message testing research
The precise timeline depends on budget and objectives, but no matter how you crunch the numbers qualitative message testing is time-consuming. It is a long road from research to execution with workshops, message review sessions, and more qual. Every piece has to be planned, scheduled, and transcribed. The long process opens up room for human biases and errors in marketing judgment.

#2 reason to give up qualitative testing in message testing research
1-on-1 interviews are very expensive on a cost-per-interview basis, especially when they are held in person. The cost of travel and other logistical expenses lends to limited sample size, thereby in-person interviews reflect only a small microcosm of a large urban area. Given the expensive reality of qualitative message testing in market research, important messaging decisions are made on the basis of a few anecdotes. Marketers are not supposed to make statistical inferences on qualitative research, but people always do!

#3 reason to give up qualitative testing in message testing research
Decision heuristics science tells us that people form opinions quickly and irrationally.Decision heuristics science is the three-time Nobel prize-winning field of research that explains how human beings make 95% of decisions using heuristics or mental shortcuts without being aware of them. Often, respondents make irrational decisions because of heuristics and rationalize them after the fact, both to themselves and the interviewer.

#4 reason to give up qualitative testing in message testing research
Compared to focus groups and surveys, 1-on-1 conversations allow for intimacy and can be useful for sensitive topics. Researchers can even read respondents’ body language for unspoken insight. However, given the personal interactions, respondents can be easily distracted and lose focus mid-way through an interview - especially on TDIs. Conversations might even divert from the main topic at hand and waste marketing dollars.

#5 reason to give up qualitative testing in message testing research
There is a gap between findings and actionability because results are directional at best. Furthermore, segment differences become blurry in qualitative message testing even though the market is becoming more segment-based. In order to achieve actionable results, the entire process must be methodical from design to recruitment to execution. Since qualitative message testing is based on opinion and judgments, it is impossible to achieve the level of accuracy needed for actionability.

#6 reason to give up qualitative testing in message testing research
Can’t test a lot of messages! The constraints on time and money force brands to make difficult decisions on what to test. The end result is bad solutions for message bundling and storyflows, and many potential combinations are left unexplored. In order to find winning bundles, improve messages, and personalize messages to each segment and channel, you need to test 100s of messages!

#7 reason to give up qualitative testing in message testing research
The sample size is always small and often composed of professional respondents - a small number of respondents show up in all interviews. The iterative approach to changing messages between interviews compresses the sample size to an even smaller group. Small numbers don’t get you to statistical significance, mathematical measurements, and hard data that are the basis of scientific research.

The takeaway

Qualitative testing in message testing research goes against every principle of scientific research and won’t give you the actionability needed for today’s complex marketing environment. In an industry defined by science, qualitative message testing is the most unscientific form of message testing in market research and there is an urgent need for innovation in message testing methodologies.