Notice: the WebSM website has not been updated since the beginning of 2018.

Web Survey Bibliography

Title An Examination of the Relationship Between Pretest Method Results and Data Quality
Year 2013
Access date 11.07.2013

Many research studies collect data through survey questionnaires. In order to enhance the validity of the findings from these studies, it is important for the studies to employ questions thatminimize measurement error. A diverse range of question evaluation methods are available for detecting measurement error in survey questions. Ex-ante question evaluation methods are relatively inexpensive, because they do not require any data collection from actual survey respondents. Other methods require data collection from respondents either in the laboratory or in the field setting. A major gap in the literature is the general lack of evidence that the problems identified by these methods are actually problems as assessed by traditional quality standards such as reliability or validity. Although one would expect these methods to identify questions that produce low quality data, behavior coding is the only technique in the literature that has been consistently shown to predict the reliability and validity survey questions (Dykema, Lepkowski, and Blixt 1997; Hess, Singer, and Bushery 1999). This paper addresses the important gap in the literature about whether the problems identified by question evaluation methods lead to lower quality data. The research in this paper investigates how effectively these methods predict the reliability of survey questions as measured by test-retest correlations obtained from repeated measurements of sample respondents. The study uses question evaluation results from a few ex-ante methods such as expert review and QUAID, laboratory methods such as cognitive interviewing, and field methods such as behavior coding and response latency to predict the reliability of survey questions. In addition, the study evaluates how the results from question evaluation methods relate to other data quality indicators such as item missing data.

Access/Direct link

AAPOR Homepage (abstract)

Year of publication2013
Bibliographic typeConferences, workshops, tutorials, presentations