Customer market research is regularly used to test “ideas” before launching them in the market. Ideas can be tested in the form of new product concepts, positioning statements, messages, etc. The basic concept of market research for idea testing is that by showing ideas to a smaller group of people before launching them to millions, researchers can uncover key insights into why and what people will buy, identify the best ideas to launch, and even estimate demand for their ideas in the market.
Message testing surveys are typically used to identify messages with the greatest customer appeal, prioritize messages from best to worst, get insights about why a message has high/medium/low appeal, and get ideas on how to improve messages. A message testing survey allows you to test messages with your existing customers before they are launched in marketing campaigns so that you can anticipate how well the campaign will likely perform.
Campaign message testing also helps brand teams optimize their campaigns and make improvements continuously through A/B testing while the campaign is live. A/B message testing is better suited for digital channels like landing pages, banner ads, email subject lines, etc. Several options can be tested in-market at the same time for a few days/weeks and if 1-2 winners emerge, the other options can be discarded and all campaign dollars can be focused on the winners.
Market researchers and marketers can pick from a variety of message testing methodology options to test messages before launch. However, there are significant drawbacks to A/B testing marketing and it is not the best method for most brands:
1. A qualitative message testing survey includes conversations with a small group of customers through in-person focus groups, phone/web interviews or online chat sessions. Typically, a moderator exposes respondents to marketing messages and gets unaided/aided feedback on the messages.
2. A quantitative message testing survey involves the use of online surveys in which customers are asked to review messages and rank/rate them based on how motivated they are. Messages are scored on 5-point or 7-point scales on diagnostic measures like Uniqueness, Believability, Relevance and Motivation to Purchase.
3. A more advanced quantitative message testing survey includes choice-based methodologies in which respondents are exposed to a series of messaging choices and asked to make preferences. Choices are constructed using a design of experiments (DOE) to ensure that every message is tested with enough respondents and against many other messages to get representative scores.
There are limitations of surveys for all message testing methodologies available to market researchers and there are no one-size-fits message testing solutions. All message testing surveys have their pros/cons.
A qualitative message testing survey is excellent at diving deep into individual messages to understand the psychology behind each one. Moderators can explore each message with respondents in detail and customize their line of questioning based on their responses. Moderators can also ask respondents to suggest improvements to each message, comment on specific words and phrases, etc. However, there are significant limitations of surveys used in qualitative message testing:
Only a small number of messages can be tested, forcing brands to make tough choices on what to test. Many times, the judgment of marketing team members is not aligned with the preferences of customers and they don’t even choose to test some ideas that might have tested very well with customers.
Respondent feedback to messages is all ‘stated’ and there are no derived insights. Behavioral science tells us that what people say they do is not what they actually do in life. The same principles apply to qualitative research as well. Respondents may say they like a message in a qualitative interview, but may not be moved by the same message in the real world at all.
It is not representative of the real world and makes respondents artificially pay too much attention to messages during the research. In the real world, consumers are exposed to >10,000 brand messages every day, which means even a 30-second exposure in research is a few seconds too long. In qualitative research, sometimes moderators can discuss one message for several minutes, making the feedback completely unrealistic.
There is little or no differentiation in message scores in qualitative message testing surveys and the results are often a regression to the mean for all the messages. Typically, messages are scored on a scale of 1-5 and most of the messages end up in the 3-4 range, which doesn’t help researchers and marketers make the best messaging decisions.
Feedback from outliers is neglected in qualitative message testing surveys even though there are many outliers in the real world. It is difficult to spot segments in qualitative research and to map whether different segments are showing a preference for different messages.
Qualitative message testing surveys do not offer a good solution for message bundling/story flow. Marketers need to understand the best message bundles and story flow for the brand and with qualitative message testing, too many combinations are unexplored.
Improvements suggested by respondents are rarely useful since they are not marketers.
A quantitative message testing survey can test a lot more messages than a qualitative message testing survey but it does not allow for deep exploration of each message and doesn't provide detailed drivers/barriers of appeal for each message. A quantitative message testing survey is also not well suited to getting ideas from respondents on how to improve messages.
Some quantitative message testing survey methodologies utilize monadic or sequential monadic design which means messages are shown to respondents one by one so that each message is evaluated independently. Monadic design-based message testing survey methodologies have similar limitations as qualitative research, i.e. they can handle only a limited number of messages and don’t produce enough differentiation in results.
More advanced message testing surveys use choice-based research methodologies. Messages are presented to respondents in the form of choice sets, with each set containing 4-5 messages. A design of experiments is created to make sure all messages are tested many times and in combination with many other messages. While choice-based methodologies can test a lot of messages, they have other limitations of their own:
The Design of Experiments approach works well with up to 30-40 messages, after which, either respondents have to be shown an overwhelming number of choices or the sample size has to be increased.
If individual messages are tested, then message bundles have to be modelled with a simulator, which is not ideal because interaction effects between messages are not adequately accounted for.
When message bundles are tested, scores for individual messages have to be modeled, which is also not ideal because many messages end up having similar scores.
The Design of Experiments does not take into account individual respondent-level choice drivers, which means that irrelevant choices could be tested with many respondents in many choice sets.
Traditional choice-based quantitative research surveys don’t provide feedback on why messages do/don’t do well in research and how to improve them.
An ideal message testing survey methodology would overcome the limitations of surveys and produce a deliverable that includes the following:
After learning exactly what is A/B testing in marketing and the pros and cons of A/B testing, it is clear that there is a need gap between A/B testing marketing deliverables and what brands need to succeed. In today’s omnichannel marketing, a message hierarchy or TURF analysis is not enough. This is an incomplete deliverable and brands are not campaign-ready. What brands need is a messaging playbook for each channel and each segment. Innovation in decision heuristics science and AI provides a better way to test messages and create winning campaigns. Decision heuristics science is a better way to test messages in market research. Heuristic-based messaging enables testing of 100s of messages so that brands can find better message bundles out of billions of possibilities and improve their chances of finding winning messaging campaigns. This innovation provides better data to create even more successful messaging campaigns in the future. AI is a better way to use research data to design campaigns. Algorithms can use research learned to produce the best bundles for each segment by channel turning research into segment-based, omnichannel campaigns.
Marketers and market researchers can feel constrained by the limitations of surveys that are no longer able to meet their campaign readiness needs. While there has been significant innovation in the market research industry at large, little innovation has been put forth on how messages can be tested in market research to find the best marketing message.
Share this Blog posts to your social media handles.
Back
Blog posts
Newristics
18 July 2022
Share
Video
Time : 33 Mins
Infographics
