Sometimes surveys deserve flack and sometimes they earn headlines. Credant Technologies, an IT security company got the former out of a survey they issued last week.
Erik Sherman, a long-time freelance writer, for a variety of publications takes exception with this survey in an article titled, “Credent Uses Funny Numbers in PR Campaign.” Sherman’s points are well-taken and certainly serve as a cautionary tale. However, having used surveys successfully in the past, both for market research and for PR efforts (and often both), I’d like to offer a different perspective.
Credent had surveyed 227 security professionals at the Infosecurity Europe 2009 trade show that found, 35-percent “just don’t get around to using a password on their business phones and smartphones, even though they know they should” because they hold sensitive data. Sherman points out the sample selection was anything but random and therefore undermines the statistical confidence – and ostensibly the newsworthiness – of the survey.
In an academic sense, Sherman is absolutely right, but a random selection is an age-old argument in market research – dating back to selecting every N-th name from an old-fashion telephone book. Even this academically accepted methodology has its confidence flaws: after all are not phone book listings in and of themselves self-selecting?
There is also ample evidence to suggest some of the best-known research firms, including Gartner and Forrester, generate similar surveys following their own trade shows. Gartner, for example, puts out research notes following their Symposium events that are often based on surveys conducted on site. Indeed, being able to afford a booth at a Gartner trade show, at least for many start-up companies, is a method of self-selection
Is the data perfect? Clearly, the answer is no, but it is an indication, that in conjunction with other data points and analysis, can prove informative. Anytime you can get a group of individuals together, especially under the auspices of a professional environment, their collective opinion, however statistically invalid, is worth observing.
In my view, the strongest point Sherman makes is when he writes:
“There is also no indication of the wording of the questions and the order in which they appeared. The interviewers might have stacked the deck to lead people to the response they wanted.”
Along, this line of thought, I’d offer to four recommendations to avoid such criticism over PR and surveys:
- First, make the questions you ask honest and public to mitigate any suggestion of manipulation.
- Second, use an independent research arm, such as YouGov, whenever possible. Yes, it’s very expensive.
- Third, disclose when a survey is “non-scientific” or at least don’t pretend a survey was conducted by random sampling if it was not. I don’t care what any academic says, in the real world, a good sample (of i.e. customers) is a good sample.
- Fourth, tie your survey results to other data points in market research to demonstrate your findings are consistent with other sources. Sometimes a quote by a market researcher or analyst goes a long way. What you’ll gain in credibility vastly outweighs what you think you might lose in exclusivity.
Pragmatic Marketing, a reputable organization for product management and marketing, stated in a recent webinar that market researchers should use focus groups for discovery and surveys for validation. I would imagine that a company like Credant with nearly 800 customers has already been through their discovery phase and their survey is validation.
As for the release in question, even with all I’ve said above, I feel it is still a strong pitch and a worthwhile endeavor. More than once I have left a cell phone in the back of a taxi by mistake and only then wished I had locked it with a passcode. It’s not a pleasant experience.
It seems most IT security professionals would agree that using a passcode on a mobile device – especially a corporate device – is a good idea. To that end, it is ironic that 227 IT security professionals don’t use one on their own devices, which in my mind, defies expectations and is, by definition, newsworthy.
If you enjoyed this post, you might also like:
Market Research and Defining a Market Opportunity
Photo credit: Flickr, [AndreasS] (CC BY 2.0)
