Web Survey Bibliography
To measure what determines people’s attitudes, definitions or decisions, social science surveys increasingly use vignettes. A vignette typically describes a situation or object as having various attributes (dimensions). Respondents are asked to form a judgment of the hypothetical situation or object described by the vignette. The values (levels) of dimensions are experimentally varied, so that their impact on respondents’ judgments can be estimated. Previous studies on related methods (conjoint analyses and choice experiments) have shown that the order in which dimensions are presented in object descriptions can affect respondents’ judgments. Similarly, it is well known that the order of single item questions, or response options, can substantially affect responses. The underlying mechanisms causing order effects – in vignettes or single item questions – are however not well understood. Drawing on the literature in cognitive psychology and survey methodology, we examine two main research questions: Does the order in which dimensions are presented impact the vignette evaluations and change substantive conclusions? Under which conditions are order effects mostly likely to occur? Using data from a factorial web survey of 300 students we analyze several possible moderators: features of the vignette design (number of dimensions, complexity of the evaluation task, sequence of vignettes, dimension importance), characteristics of respondents (knowledge of the topic, strength of attitudes, cognitive ability), and interactions between these features. Our results show that strong order effects can occur, but only when the vignettes are of a minimum complexity.
Conference Homepage (abstract)
Web Survey Bibliography - Jaeckle, A. (14)
- Going online with a face-to-face household panel: initial results from an experiment on the Understanding...; 2013; Jaeckle, A., Lynn, P., Burton, J.
- Effects of visual and aural communication of categorical response options on answers to survey questions...; 2012; Lynch, P., Hope, S., Jaeckle, A., Campanelli, P., Nicolaas, G.
- Understanding Society Innovation Panel Wave 4: Results from Methodological Experiments; 2012; Burton, J., Budd, S., Gilbert, E., Jaeckle, A., Kaminska, O., Uhrig, S.C. N., Brown, M., Calderwood,...
- Going online with a face-to-face household panel: Initial results from an experiment on the UK Household...; 2012; Jaeckle, A., Lynn, P., Burton, J.
- ISER working paper 2011-31. Is it a good idea to optimise question format for mode of data collection...; 2011; Nicolaas, G., Campanelli, P., Hope, S., Jaeckle, A., Lynn, P.
- A classification of question characteristics relevant to measurement (error) and consequently important...; 2011; Campanelli, P., Nicolaas, G., Jaeckle, A., Lynn, P., Hope, S., Blake, M., Gray, M.
- Understanding Society Innovation Panel Wave 3: Results from Methodological Experiments; 2011; Burton, J., Budd, S., Gilbert, E., Jaeckle, A., McFall, S., Uhrig, S.C. N.
- First Equals Most Important? Order Effects in Vignette- Based Measurement; 2011; Auspurg, K., Jaeckle, A.
- Is it a good idea to optimise question format for mode of data collection? Results from a mixed modes...; 2011; Nicolaas, G., Campanelli, P., Hope, S., Jaeckle, A., Lynn, P., Nandi, A.
- The role of the interviewer in producing mode effects: results from a mixed modes experiment; 2011; Hope, S., Campanelli, P., Nicolaas, G., Lynn, P., Jaeckle, A., Nandi, A.
- The role of visual and aural stimuli in producing mode effects on answers to survey questions; 2011; Lynn, P., Hope, S., Campanelli, P., Nandi, A., Nicolaas, G., Jaeckle, A.
- Causes of Mode Effects on Survey Measurement ; 2011; Lynn, P., Campanelli, P., Nicolaas, G., Hope, S., Nandi, A., Jaeckle, A.
- Dependent Interviewing: Effects on Respondent Burden and Efficiency of Data Collection; 2008; Jaeckle, A.
- The Use of Web Surveys in the UK Household Longitudinal Study; 2007; Jaeckle, A., Laurie, H., Lynn, P.