Checking the External Validity of a Choice Model

External validity is a term of art in research, and it refers to the extent to which conclusions drawn from a model will turn out to be true in the real world. There are three main ways of assessing the external validity of a choice model, and they are ordered from least to most powerful:

• Cross-validation
• Checking that the utilities are appropriately correlated with other data
• Ability to predict historic market performance

Cross-validation

A basic metric for any choice model is its predictive accuracy. The chocolate study has a predictive accuracy of 99.4% after removing the respondents with poor data. This sounds too good to be true, and it is too good. This predictive accuracy is computed by checking to see how well the model predicts the data used to fit the model.

A more informative approach to assessing predictive accuracy is to check the model's performance on data not used to fit the model (i.e., cross-validation). See How to Compare Choice Models (Cross-Validation).

Checking that the utilities are appropriately correlated with other data

All else being equal, we should expect that the utilities computed for each person are correlated with other things that we know about them. For example:

• The chocolate study finds that
  • diabetics have higher utility for sugar-free chocolate
  • people with higher incomes were less price-sensitive
• The study looking at carbon neutrality found that republicans and people opposed to the US having a target for carbon neutrality had lower utilities for being employed by a carbon-neutral employer and had higher utilities for salary increases.

A word to the wise: avoid using gut feeling and prejudice. For example, it is pretty common for marketers to have strong beliefs about the demographic profiles of different brands' buyers (e.g., Hershey's will be bought by poorer people than Godiva). It is often the case that such beliefs are not based on solid data. It is therefore a mistake to conclude a choice model is bad if it does not align with beliefs, without checking the quality of the beliefs.

Ability to predict historic market performance

While the goal of a choice model is to make predictions about the future, it can also be used to make predictions about the past. For example, if a choice model collects price data, it can be used to predict the historic impact of changes in price. If the predictions of history are poor, it is suggestive that the model will also not predict the future well. You can find out more about this in 12 Techniques for Increasing the Accuracy of Forecasts from Choice Experiments.