Survey researchers, who must choose between mixed mode and single mode surveys, currently have limited evidence regarding implications for measurement and estimation. While many studies have investigated measurement effects associated with the main survey modes, and the overall effect of conducting a survey in one mode rather than another, few studies assess the overall effect of a mixed mode design versus a single mode.
Yet the question of equivalence between single and multi-mode surveys is important in many practical situations. In cross-national surveys, for example, only some countries may have the capacity to carry out a mixed-mode survey. In this paper, we discuss the evidence survey researchers need to help them to decide whether a mixed mode survey can provide data equivalent to a single mode survey, and we offer some tentative conclusions based on a case study.
We present an experiment conducted in the Netherlands parallel to Round 4 of the European Social Survey (ESS4). Two different mixed-mode designs – “concurrent” and “sequential”, both involving web, telephone, and face-to-face interviews – were compared with the regular single-mode ESS4 survey. We discuss practical difficulties encountered in the design and management of the mixed mode surveys, and offer explanations for the surprising finding that both mixed mode designs achieved a significantly lower response rate than the single-mode face-to-face survey.
We also gauge the equivalence of univariate and multivariate analyses across the three designs. While most variables yield equivalent point estimates across all three samples, there are notable and statistically significant differences on some types of variables, especially those with social desirability connotations. We also explore the more complex effects that the mixed mode designs had on patterns of bivariate correlations, and on the results of multivariate analyses.