Web Survey Bibliography
Relevance and research question: Cost pressures continue to necessitate switching single-mode designs, such as face-to-face (F2F), to inexpensive mixed-mode (MM) designs. This, however, involves the risk of finding different estimates in the MM and single-mode designs, if total survey error is affected by the redesign strategy. Differences in measurement and non-observation error are particularly relevant candidates for design differences in total error, also called measurement and selection effects (MEs and SEs). Knowledge about these effects is relevant to evaluate the usefulness of a MM redesign. We demonstrate the estimation of MEs and SEs for the case of the Dutch Crime Victimization Survey (CVS).
Methods and Data: We administered a split-ballot design, in which four independent samples (n=2,200 each) were drawn from the Dutch population register and assigned either to one of three sequential mixed-mode surveys (web, mail, and telephone, followed by F2F, respectively) or to a single-mode F2F condition, which served as benchmark. Additionally, the respondents to web, mail and telephone were approached a second time in F2F after four weeks. This step made available ‘within-subject’ estimates of MEs, which were exploited to disentangle MEs and SEs in the split-ballot design.
Results: Largest design differences in estimates were found for the mixed-mode mail-F2F design and, with smaller magnitude, also for web-F2F. The telephone-F2F survey showed mainly insignificant differences against the benchmark. In evaluating MEs and SEs, we found that MEs were the predominant cause of the differences between both, mail-F2F and web-F2F, and the benchmark, whereas SEs were generally very small. MEs and SEs were absent when comparing telephone-F2F against F2F.
Added Value: In the CVS case, the large MEs for the mail-F2F and web-F2F designs would require further redesign of questionnaires (e.g. by unimode strategies) to balance measurement error in mail and web towards F2F. Telephone-F2F yielded comparable estimates vis-à-vis the benchmark due to similar measurement error properties and could be implemented directly. The absence of SEs might suggest that the MM designs were successful in mitigating non-observation error differences between designs. More generally, our method could be used by other researchers to evaluate MM redesigns for other surveys.
Web survey bibliography - Schouten, B. (20)
- Adaptive survey designs to minimize survey mode effects – a case study on the Dutch Labor Force...; 2016; Calinescu, M.; Schouten, B.
- Tailored fieldwork design to increase representative household survey response: an experiment in the...; 2015; Luiten, A.; Schouten, B.
- Coding Surveys on their Item Characteristics: Reliability Diagnostics; 2015; Bais, F.; Schouten, B.; Toepoel, V.
- Selection error in single- and mixed mode surveys of the Dutch general population; 2015; Hox, J., Klausch, L. T., Schouten, B.
- Evaluating mixed-mode redesign strategies against benchmark surveys: the case of the Crime Victimization...; 2014; Klausch, L. T., Hox, J., Schouten, B.
- The impact of contact effort on mode-specific selection and measurement bias; 2014; Schouten, B., van der Laan, J., Cobben, F.
- The use of within-subject experiments for estimating measurement effects in mixed-mode surveys ; 2014; Klausch, L. T., Schouten, B., Hox, J.
- The impact of contact effort and interviewer performance on mode-specific nonresponse and measurement...; 2014; Schouten, B., Cobben, F., van der Laan, J., Arends, J.
- Measurement Effects of Survey Mode on the Equivalence of Attitudinal Rating Scale Questions; 2013; Klausch, L. T., Hox, J., Hox, J., Schouten, B.
- Adaptive survey designs to minimize survey mode effects. A case study on the Dutch Labour Force Survey...; 2013; Calinescu, M., Schouten, B.
- Optimizing quality of response through adaptive survey designs; 2013; Schouten, B., Calinescu, M., Luiten, A.
- Mode Effects in Mixed-Mode Surveys: Prevention, Diagnostics, and Adjustment 1; 2013; de Leeuw, E. D., Dillman, D. A., Schouten, B.
- Estimating Measurement Effects of Survey Modes From Between and Within Subject Designs; 2013; Klausch, L. T., Hox, J., Schouten, B.
- Mixed-mode including web: Recent developments at Statistics Netherlands; 2013; Luiten, A., Schouten, B.
- Disentangling Mode-Specific Selection and Measurement Bias in Social Surveys; 2012; Buelens, B., van der Laan, J., Schouten, B., Klausch, L. T., van der Brakel, J., Burger, J.
- Assessing Measurement Equivalence and Bias of Questions in Mixed-mode Surveys Under Controlled Sample...; 2012; Klausch, L. T., Hox, J., Schouten, B.
- The Representativity of Web Surveys of the General Population compared to Traditional Modes and Mixed...; 2012; Klausch, L. T., Schouten, B., Hox, J.
- Disentangling relative mode effects for the web survey mode in the Safety Monitor; 2011; Schouten, B., van de Brakel, J., Buelens, B., Klausch, L. T., van der Laan, J.
- Indicators for the representativeness of survey response; 2009; Schouten, B., Cobben, F., Bethlehem, J.
- Nonresponse Bias in Surveys; 2009; Bethlehem, J., Vehovar, V., Stoop, I., Schouten, B., Shlomo, N., Skinner, C., Montaquila, J.