Notice: the WebSM website has not been updated since the beginning of 2018.

Web Survey Bibliography

Title Web Survey Design: Effect of Layout on Measurement Error
Author Peytchev, A.
Source RC33 6th International Conference on Social Science Methodology: Recent Developments and Applications in Social Research Methodology, 2004
Year 2004
Access date 14.09.2004
Abstract Literature on context effects has demonstrated that placing related attitude questions together could greatly affect responses, altering their means and associations. This typically results in increased interitem correlations, which has been interpreted as higher reliability by some, and as lower between-item discrimination by others. Paradoxically, both contradictory interpretations assert less measurement error based on the same statistic. In web surveys, placing multiple questions per page is common, yet no scientific guidelines have been developed with regard to best practices. It is crucial to identify the causal mechanism for this effect of visual layout and test competing theories with analytic methods that go beyond descriptive statistics and allow evaluation of measurement error. The main objectives of this paper are: (1) to provide a theory guided framework for evaluating the quality of attitude measures in different survey layout designs, and (2) to show how data is affected by presenting fewer questions per page and whether it is preferable. Analysis is based on an experiment varying the number of questions presented per page, administered to participants in web survey panels. Eight attitudinal questions were asked in one, two, or eight separate pages. Behavior frequency questions on the same topic were also asked in a different part of the survey. Approximately 2,700 respondents were randomly assigned to one of the experimental conditions upon reaching the first of the eight manipulated questions. The association between the attitude and the behavior constructs (validity coefficient) was expected to attenuate when the attitude questions were presented together, as primarily short-term memory and avoidance of incongruent responses would result in enhanced context effects and subsequently, greater measurement error. No significant differences in the means were found. Covariance matrices were significantly different, in a manner consistent with the existing literature – covariances were higher when more questions were presented on the same page. Using SEM, a theoretically simple model that limits the likelihood of model misspecification was fitted to the data. Placing the attitudinal questions on separate pages had produced data with preferable properties, as relationships between the latent constructs were stronger and overall model fit was improved. Multi-group invariance analysis revealed that the attitude questions were measuring the same latent constructs under all three conditions, but the designs differed significantly in terms of measurement error. Since correlated measurement error results in higher reliability estimates for statistics like Cronbach’s Alpha, some researchers are led to believe that presenting multiple questions per page is preferable. This study demonstrates that presenting related attitudinal questions together causes lower validity and reliability, and discusses the likely underlying processes.
Year of publication2004
Bibliographic typeConferences, workshops, tutorials, presentations
Print

Web survey bibliography - Peytchev, A. (13)