Notice: the WebSM website has not been updated since the beginning of 2018.

Web Survey Bibliography

Title Questionnaire Pretesting Methods: Do Different Techniques and Different Organizations Produce Similar Results?
Year 2001
Access date 15.04.2014
Full text

pdf (69.6 KB)


During the past 15 years, in an effort to improve survey data quality, survey practitioners have significantly increased their use of an evolving set of questionnaire pretesting methods. Several researchers have addressed issues related to questionnaire evaluation, and have attempted to determine the potential strengths and weaknesses of each (Campanelli, 1997; DeMaio, Mathiowetz, Rothgeb, Beach, and Durant,1993; Oksenberg Cannell, and Kalton, 1991; Presser and Blair,1994; Willis, 2001). Further, several empirical investigations have evaluated the effectiveness of core features of these techniques, especially the use of verbal probing within cognitive interviewing (Davis and DeMaio 1992; Foddy, 1996) and several evaluative studies have attempted to assess the effectiveness of cognitive interviews in ameliorating questionnaire problems (Fowler and Cosenza, 2000; Lessler, Tourangeau, and Salter, 1989; Presser and Blair; Willis and Schechter, 1996; Willis, Schechter, and Whitaker, 1999); these are reviewed in detail by Willis (2001). Increasingly, evaluations have focused on the side-by-side comparison of survey pretesting techniques, in order to determine the degree to which the results obtained through use of these techniques agree, even if they cannot be directly validated. However, this research is complex, as evaluation in practice must take into account the multi-faceted nature of each of the pretesting techniques, and of questionnaire design in general (see Willis, DeMaio, and Harris-Kojetin, 1999). Although two studies (Presser and Blair, 1994; Willis, 2001) have specifically compared the results of cognitive interviewing, expert evaluation, and behavior coding, when these have been applied to the same questionnaire, this research has generally not been conducted in a way that allows for the separation of the effects of pretesting method from those of the organization applying these methods. The overall objective of this study was to rectify this limitation. Overall the selected design balanced technique with organization, for the same set of questionnaires (see Lessler and Rothgeb, 1999; Rothgeb and Willis, 1999), to determine level of agreement among three pretesting techniques, when applied by each of three survey research organizations. For this research, multiple researchers within each of the organizations used three pretesting methods: Informal expert review, Formal cognitive appraisal, and Cognitive Interviewing. A classification scheme was developed to code problems identified through any of the methods, and by each organization.

Access/Direct link

Homepage (abstract) / (full text)

Year of publication2001
Bibliographic typeConference proceedings