Context Effects: When Setting Changes Everything
If you ever get a chance to go to an event with Joel Huber, take it. Joel was editor of JMR (The Journal of Marketing Research) since the Dawn of Time and is perhaps the single deepest font of knowledge on market research in existence. If someone somewhere has tried to do something even vaguely related to market research, Joel knows about it. I tried to convince him to write a book. I hope he does.
At the TurboCBC event last month, he spoke about context effects in choice-based conjoint studies, in his typical modest and encyclopedic manner. For the uninitiated, a context effect is the phenomenon in which the setting of the question changes the nature of the answer. He had three big conclusions:
- Context effects are everywhere.
- Usually, they don’t cause big problems.
- Except for the uncommon occurrence when they do cause big problems!
The interesting thing about context effects is that there really is no modeling fix. The only fix is to change the context of the question or the task. However, it’s not always obvious how one should lay out a question. Among the various effects Joel discussed:
- Price, in general, is the worst attribute when studying context effects in choice studies. Of course, other methods of studying price are just as bad or worse!
- When you set a range, people key off of what they perceive as the ‘baseline’ value. Relative changes may be stable, but absolute levels are not stable.
- When you increase the range of an attribute you test, the attribute's importance increases, but less than the increase in the range. So if I’m looking at the effect of increasing the size of a burger, the impact of a 1 ounce increase is more if the test spans from 4 ounces to 8 ounces than if it spans from 4 ounces to 16 ounces.
- People like to compromise. If you give them three levels, they tend to like the middle level more than they should.
- People avoid extremes. A tested value ceases to be extreme when you add a level that is slightly above the highest (or below the lowest) level tested. Then the new level becomes the extreme.
- People learn over time (and quickly). Price sensitivity for an individual respondent changes dramatically from the beginning to the end of a choice exercise, but it’s not quite clear that the early data is "more true". It could be less true.
- Asymmetric dominance. You can get people to choose something more often by offering a much worse product at the same (or similar) price. A thing seems better when compared to something that is obviously worse.
Two of the dominant themes of context effects are that people’s preferences are not stable and that our brains make decisions using relative judgments. Model builders need to pretend that humans are fixed, but we aren’t fixed. Often, we don’t even know what we like until we see it, and we like something a lot until we find something better. Indeed, scholars don’t even know whether context effects are “merely” causing bias in our estimates of something, or whether they are actually changing the underlying measure itself.
Selfishly, there are a couple things I took away from Joel’s presentation. First, even when tools are cheap, it helps to have some practical expertise when creating a research study. Second, if you don’t know about a context effect, it’s easy to present a methodology bias as a major study finding. Third, because evolutionary algorithms directly optimize concepts without trying to measure them, they handle context effects more robustly than measurement-based approaches.
Does that mean that, in the words of Margaret Atwood, “Context is All?” I’m not willing to go that far. It is fair to say, however, that context effects are to market research what behavioral economics is to economics!
