Ecological Validity

A choice experiment has ecological validity if the way it is answered by respondents is consistent with behavior in the real world. A number of strategies exist for trying to ensure the ecological validity of a choice experiment:

• Making the questions appear as they would in the real world, such as using mock shelves, real products, augmented and virtual reality, and mock-ups of advertisements. The basic idea is that if the questions are more realistic, the data is more likely to be accurate (i.e., people are more likely to make a choice during a survey as they would in the real world).
• Structuring the design of the experiment in such a way that the information that is presented to the respondent is the same information that they would rely upon in the real world.
• Making tasks plausible. For example, not offering people a Ferrari for $10,000.
• Situation cueing, so that people are placed in a realistic frame of mind when answering the question. See Situation Cueing.
• Ensuring incentive compatibility, which is a term of art from economics, and means that you want to make sure that people have an incentive to answer honestly. For example, asking a person “Would you buy a can of Coke if it cost $5” is not incentive compatible, as a smart person will deduce that if they answer “yes” it may lead to an increase in the price of Coke. Mechanisms for achieving incentive compatibility include explaining the objectives behind the research, disguising the research question, and providing economic incentives (e.g., giving people $10 and they can keep the change that is left after they make their choices).
• Preventing satisficing (e.g., removing None of these alternatives, to avoid people choosing it instead of making a tradeoff).
• Avoiding fatigue.
• Replicate the extent of real-world overlap of attribute levels. That is, if in the real-world most alternatives that people choose from tend to share some attribute levels, then it is desirable that the choice tasks are structured in the same way.

These strategies are often at odds and judgment is required. For example, while having a screen showing a fake supermarket shelf seems like a good thing (realism), the constraints of a computer screen in practice make it difficult for people to read detail from packaging, which limits the information available. Similarly, while virtual reality can address the issue of difficulty in reading from packaging, it slows down the whole process, thereby introducing fatigue.