The point of surveys is to discover things you didn’t know previously not to artificially create a conclusion that readers should buy your product
I noticed a survey put out by a large technology company the other day. A press release announcing the survey indicated it was conducted by a third-party research firm, had more than a thousand respondents and covered a timely topic.
This is the type of survey you’d expect would get lots of coverage, but it didn’t. I searched for both mentions and backlinks and found not a single outlet covered the survey – beyond press release syndications. The reason was very apparent: the answer to all the problems the survey identified was to buy the company’s product.
No journalist will cover a study that concludes their audience needs to buy your product. Why? Because that’s not a story, it’s an advertisement.
The proliferation of junk survey research
It’s never been easier to conduct a survey. Web-based survey tools have made this task simple. As a result, all sorts of junk research studies – and the press releases promoting them – have proliferated.
I’m not going to get into sampling errors in this post – there are plenty of studies decrying the demise of the random sample. However, I would argue that every web-based survey conducted today is self-selecting.
Aside from formal academic training, which I do recommend, there are some simple ways to produce better surveys.
1. Start with a compelling research objective
Too many surveys are initiated to prove a point – usually about their product. This is akin to confirmation bias – a cognitive bias where someone seeks out information that reinforces their existing beliefs (this is why cable news – both left and right-leaning – is destructive).
However, if you start with a genuine research objective – with an open-mind and genuine curiosity – chances are you’ll uncover real findings with which your target audience will identify.
A good way to start is to look at a few studies and see what’s been done – and what’s missing.
2. Write and ask objective questions
Follow up a genuine research objective with objective questions. In other words, you want to avoid influencing a respondent to select a certain answer choice. There’s a fun little story that’s been around market research circles for a long time that that makes this point:
Two priests want to know if they can smoke and pray at the same time.
One priest asks the Pope, “When I’m praying, is it okay to smoke?”
The Pope replies, “No, when you are praying that is all you should be focused on.”
The other priest poses his question to the Pope in a slightly different construct and asks, “When I am smoking, is it okay for me to pray?”
The Pope replies, “Yes, it is always appropriate to pray anytime.”
3. Don’t be a hero! Look for answers in the survey results
Too many companies that conduct surveys strive to force their product to a conclusion. So, if they’ve done a good job of coming up with a research objective and writing good questions – they toss that work out the window with such a conclusion.
A better approach is to look for the answers in the results. One way is to compare respondents by performance. This requires asking respondents to rate their performance on a scale and then comparing and contrasting answers by performance cohort. This will show you what high performers do differently.
A second technique is to ask an optional open-ended follow-up question to key questions: Why? This will give you context to the quantitative answers you ask. I’m often amazed at what comes back in such comments – it’s usually gold.
Finally, ask an open-ended question at the end. One on occasion, a respondent’s answer to such a question summed up the research so beautifully that I made it the title of a subsequent report.
The point is discovery
Sometimes surveys like those I described, in the beginning, are the result of executives imposing their will on marketing or PR. They’ll often say if the research doesn’t conclude our product is the best, what’s the point of doing it?
The point of a survey is to discover something you didn’t know previously. That’s the sort of story in which a reporter is interested.
Good research projects spark an industry conversation and a real discussion about solutions that may or may not include your product. We often colloquially refer to it as thought leadership. There’s a ton of research that shows thought leadership that doesn’t sell – sells.
If you enjoyed this post, you might also like:
Your Brand is Not the Hero…Your Customer is the Hero
Image credit: Unsplash
