Notice: the WebSM website has not been updated since the beginning of 2018.

Web Survey Bibliography

Title Estimating Measurement Effects of Survey Modes From Between and Within Subject Designs
Year 2013
Access date 30.05.2013

Measurement effects are a major problem in mixed-mode surveys suggesting that the same respondent potentially provides different answers under different modes. Mixed-mode researchers therefore often need to know the average size of measurement effects (AME) for the questions of their interest. The present paper discusses estimation of AME using two different data collection approaches: a between subject and a within subject (repeated measures) design. Real-world data from an experiment with N=8,800 subjects in The Netherlands are presented. In the ‘between design’, subjects were randomly allocated to one mode only (Face-to-Face, Telephone, Mail, or Web). In the ‘within design’ subjects were first allocated as in the ‘between design’ and subsequently re-approached after some weeks in a reference mode (Face-to-Face) repeating a large number of questions. Unit nonresponse in both designs represents a threat to full randomization and thus to unbiased estimation of the AME, if confounders relate to the selection mechanism into mode conditions and the outcome variable. Statistical adjustment of missing data is a possible solution to this problem, but it is based on assumptions. Adjustment in ‘between designs’ assumes that the selection mechanism is ignorable given auxiliary variables. This is often contestable in practice, because some important confounders might not be observed. An advantage of ‘within designs’ is that it is more plausible to ignore the selection mechanism when conditioning on the repeated measurements. Thereby it is not problematic whether time-related changes of outcomes between measurement occasions occur, because these can be controlled using subjects who are allocated to the reference mode on both occasions (i.e., Face-to-Face). However, ‘within designs’ need to assume that measurements can be taken  independently across time. We compare AME estimates from both designs for questions from the Dutch Crime Victimization Survey applying regression adjustment with propensity score strata as covariates or propensity score weighting.

Access/Direct link

Conference Homepage (abstract)

Year of publication2013
Bibliographic typeConferences, workshops, tutorials, presentations

Web survey bibliography - Hox, J. (23)