Identifying Respondents with "Irrational" Preferences

Choice models assume that people are, in some sense, rational. If respondents have clearly irrational data then the conclusions from the models are unreliable. This article first describes The idea of rational choice in choice models and then explains the need to Look for strong evidence of irrationality. 

The idea of rational choice in choice models

The term "rational" is a term of art from economics, which essentially means that the person is making choices in a way that is consistent with their self-interests. Working out if choices in conjoint questionnaires are rational is an ongoing area of research, and lots of work shows that they are often not rational (e.g., irrelevant information shown earlier in a questionnaire can influence choices). Nevertheless, in most studies, it is possible to draw some conclusions regarding what constitutes irrationality.

For example, in the chocolate study analyzed in many of the other articles on The Data Story Guide, 6% of the sample had data indicating they preferred to pay $2.49 to $0.99.

A common misunderstanding is that maybe a result like this is an insight rather than a problem. For example, perhaps it means that people use price as a cue for quality. However, a fundamental assumption of choice modeling is that people are making choices based on the information presented to them in the choice questionnaire. If instead, they are using the information presented to make guesses about other non-shown attributes (such as quality), then the resulting data is of limited utility and can only end up causing problems. For example, if a chocolate marketer is told that more people will buy chocolate at $2.49 than $0.99, they will raise the price but not the quality, and consumers will quickly learn not to use price as a cue for quality in that market).

Look for strong evidence of irrationality

Returning to the finding that 6% of respondents were found to prefer to pay more for chocolate, it is premature to conclude that such respondents are irrational. Inevitably some people will find price relatively unimportant, and for these people there will be uncertainty about their preference for price. A more rigorous analysis is to only exclude people where it is highly likely they have ignored price (e.g., with a probability of at least 95%). Such an analysis leads to only excluding 1 person (0.3% of the sample). The post Performing Conjoint Analysis Calculations with HB Draws (Iterations) describes the basic logic of such an analysis.

Another thing to be careful about when identifying irrational respondents is that the respondents should be given the benefit of the doubt. As discussed in Checking that Data is Not Impossible, it is easy to incorrectly make conclusions about such things.