Derived from cognitive psychology and adapted to the context of questionnaire development, cognitive interviewing methods have been extensively used to pre-test survey questions. Usually the researcher/questionnaire designer uses cognitive interviewing methods in the questionnaire development stage as a qualitative tool to test specific questions by establishing whether respondents interpret the questions consistently and in the way intended (comprehension). The method is also used to explore recall, judgment and response issues.

As part of a wider Economic and Social Research Council (ESRC) grant funded project, which involved experiments comparing respondents’ answers to survey questions in different modes, cognitive interviews were used as a follow up methodology to explore in more detail how mode effects happen even if they are not directly observed. Selected respondents from the quantitative experiments took part in cognitive interviews where questions were administered face to face, by phone and by web, and were followed by a retrospective think aloud. The focus of the think-aloud (used in combination with some pre-scripted probes) was not on what respondents understood by certain words or answer categories but was instead on how they had gone about answering questions in the different data collection modes.

The paper will focus on the methods used in these mixed modes cognitive interviews which involved an element of mode mimicking, how the interviews differed from standard cognitive interviewing for question testing, identifying the benefits and limitations of the approach taken, and drawing out lessons for using think aloud methods in unusual ways.