Checking a Questionnaire Using Think-Alouds

A think-aloud is an approach to checking a questionnaire by getting people to complete the questionnaire and verbalize their thinking as they do so. It is particularly useful for checking more advanced question formats, such as choice-based conjoint questions.

This article describes the:

• The goals of a think-aloud
• Who should conduct the think-aloud
• Sampling for the think-aloud
• Probes/prompts for a think-aloud

The goals of a think-aloud

The goals when conducting the think-aloud are to work out, in real-time, if the respondent:

• Has an accurate internal representation of each question? That is, are they interpreting the question as it is intended?
• Has difficulty answering any questions or is fatigued by the length?
• Is giving an accurate answer? That is, based on what you know and what they are saying, does their choice make sense? This is key. If you find yourself thinking during the process “I don’t know; maybe; how can I know” whether their choice is reasonable, it means that you have not succeeded in doing the think-aloud properly (see the tips in the next section).

Who should conduct the think-aloud

The think-aloud needs to be conducted by people that were actively involved in the wording of the questionnaire, as these are the only people that can assess whether the questions are being interpreted as assumed in the experimental design. Getting an independent team of qualitative researchers to work on the process tends not to be so useful, as they do not appreciate the key decision decisions that have gone into the questionnaire. Choice-based conjoint attributes need to be specific and need to be measuring things that the end client can manipulate; it is usually difficult for a qualitative researcher to assess if the wordings of such levels are appropriate unless they have been in all the discussions with the client and understand the end use of the model.

Sampling for the think-aloud

Typically a think-aloud requires a convenience sample. It is not necessary that the sample is a random sample of the population of interest. All that is required is that they will interpret the language in the questionnaire in a way that is broadly similar to the population of interest. Consequently, it's usually appropriate to use friends, family, and colleagues when conducting think-alouds. 

Probes/prompts for a think-aloud

To be successful, you need to get people to verbalize their interpretation and thinking processes as they are completing the questionnaire. The goal is to get them to say their internal thought processes, out loud. This is not natural for most people, so they need to be encouraged. Some of the following prompts can be useful:

• “Please say everything that comes through your mind out loud.” Typically, this needs to be repeated many times until somebody gets into the routine.
• If the respondent is not thinking aloud, encourage them to elaborate:
  • What do you think this question is asking you?
  • What do you think this question means?
  • How, exactly, did you work your answer out for that question?
  • What are you thinking about?
  • What is stopping you from making a choice?
  • How did you arrive at that answer (note that we should not be asking them why)?
  • What does INSERT SPECIFIC FEATURE LEVEL mean to you?

At the end of a question or set of questions, you can also ask if the respondent thinks there is any additional information that is relevant.