Relevance and research question: Cost pressures continue to necessitate switching single-mode designs, such as face-to-face (F2F), to inexpensive mixed-mode (MM) designs. This, however, involves the risk of finding different estimates in the MM and single-mode designs, if total survey error is affected by the redesign strategy. Differences in measurement and non-observation error are particularly relevant candidates for design differences in total error, also called measurement and selection effects (MEs and SEs). Knowledge about these effects is relevant to evaluate the usefulness of a MM redesign. We demonstrate the estimation of MEs and SEs for the case of the Dutch Crime Victimization Survey (CVS).
Methods and Data: We administered a split-ballot design, in which four independent samples (n=2,200 each) were drawn from the Dutch population register and assigned either to one of three sequential mixed-mode surveys (web, mail, and telephone, followed by F2F, respectively) or to a single-mode F2F condition, which served as benchmark. Additionally, the respondents to web, mail and telephone were approached a second time in F2F after four weeks. This step made available ‘within-subject’ estimates of MEs, which were exploited to disentangle MEs and SEs in the split-ballot design.
Results: Largest design differences in estimates were found for the mixed-mode mail-F2F design and, with smaller magnitude, also for web-F2F. The telephone-F2F survey showed mainly insignificant differences against the benchmark. In evaluating MEs and SEs, we found that MEs were the predominant cause of the differences between both, mail-F2F and web-F2F, and the benchmark, whereas SEs were generally very small. MEs and SEs were absent when comparing telephone-F2F against F2F.
Added Value: In the CVS case, the large MEs for the mail-F2F and web-F2F designs would require further redesign of questionnaires (e.g. by unimode strategies) to balance measurement error in mail and web towards F2F. Telephone-F2F yielded comparable estimates vis-à-vis the benchmark due to similar measurement error properties and could be implemented directly. The absence of SEs might suggest that the MM designs were successful in mitigating non-observation error differences between designs. More generally, our method could be used by other researchers to evaluate MM redesigns for other surveys.