Notice: the WebSM website has not been updated since the beginning of 2018.

Web Survey Bibliography

Title Associations Between Interactional Indicators of Problematic Questions and Systems for Coding Question Characteristics
Year 2013
Access date 11.07.2013

Writing survey questions requires attention to the conceptual and operational definitions of survey concepts as well as to the technical issues that arise in composing items. These technical issues are examined in a body of research that considers how characteristics of questions (e.g., the number of categories to include in a rating scale) affect responses, their distributions and associations with other variables, and their validity and reliability. While the analysis of the properties of questions has led to the development of several ad hoc and formal systems for coding characteristics (e.g., Problem Classification Coding System (CCS) (Forsyth et al. 2004), Question Appraisal System (Willis 2005), and Question Understanding Aid (QUAID) (Graesser et al.), these systems vary considerably in the assumptions that underlie whichcharacteristics they identify as problematic, which characteristics are compared, and how dependencies among characteristics are taken into account when writing questions. Our paper has several goals. First we review and synthesize the literature on question characteristics and the systems for coding characteristics. Second, we analyze the administration of questions about physical and mental health from 350 digitally recorded and transcribed interviews with older adults in the Wisconsin Longitudinal Study. Interviewer-respondent interaction has been coded in Sequence Viewer and we have also coded the questions’ characteristics using several different coding schemes. We identify interactional behaviors that have been associated with poorer data quality and use multi-level models to determine which coding systems are best at predicting problematic outcomes, including interviewers misreading questions and respondents expressing uncertainty and requesting clarification. Our analysis adds to the small but growing body of research concerning the effects of question characteristics on interaction and data quality. Our results have implications for designing questions and interviewing procedures with an emphasis on health surveys of older adults.

Access/Direct link

UWSC Homepage (presentation)

AAPOR Homepage (abstract)

Year of publication2013
Bibliographic typeConferences, workshops, tutorials, presentations

Web survey bibliography (4086)