Realism vs. Artificiality in Research Techniques
As I write this, I’m flying back from the TurboCBC conference, which is sponsored by Sawtooth Software. Terrific event. A great array of leading thinkers, brought together in a very collegial setting. If you use conjoint and ever get a chance to go, do it. The discussion covered too much to replay here – so instead let me talk about a high-level theme that came up repeatedly: Realism vs. Getting Good Information.
A few years ago, virtual store shopping was all the rage. Companies spent half a year and half a million dollars creating a “virtual store” and walking consumers through it. Some companies created huge virtual reality rooms with multiple computer screens and scent emitters and eye-tracking goggles. Star Trek’s holodeck, eat your heart out!
We were told that online research (still in its adolescence) was dead, and VR was the murderer. I had my doubts: If you make a single product change in a virtual store with 10,000 or more products, how likely was anyone to even notice the change? How much sample was needed? How did respondents participate?
Of course, reports of the demise of online surveys were exaggerated, and no one talks about VR for MR anymore. So why did virtual reality fail (in its initial form)? Among other things, it had no statistical power. That’s a fancy way of saying that it couldn’t detect any of the differences in product preference that we are interested in seeing. In any single shopping trip, the likelihood of a customer becoming aware of a new product launch or a product change is nearly zero: after all, that’s why companies advertise. Likewise, the technology is so intrusive that it’s actually less natural than simply using a computer screen.
On the far opposite end of the spectrum, we sometimes use techniques that have no realism whatsoever. For example, rank-order questions in a grid array. These questions can yield a lot of information, but we question the validity of the information because of the unrealistic way in which it was gathered.
In most situations, the tradeoff between realism and getting good information is not as obvious as in those above, but it’s there. Grid layouts in conjoint – where each feature is distinctly identified and listed in a column so that everything has equal importance – give us great information about tradeoffs by giving each item equal visual importance.
Yet we wonder whether every item should be given equal importance in presentation when they don’t have equal importance at the store shelf or on a website or in a direct mail piece. In one sense, standardizing visual attention removes bias, but in another sense it creates bias. Which is better?
It depends… If you want to know what people care about in general, then more realism may be a bad thing. If you want to know how people would respond to product changes given how those products are really marketed, more realism is a good thing… so long as we still get enough statistical power to answer our questions.
Like so many choices we make in building a study, the best study design depends on what you want to get out of the study.
