Mode effects are difficult to separate from nonresponse effects in mixed-mode surveys. Mode effects include acquiescence and social desirability effects due to the presence of an interviewer, recency effects in audio surveys and primacy effects in visual surveys. However, in everyday mixed-mode surveys we often cannot distinguish mode-effects from effects due to differences in nonresponse and coverage that coincide with the use of multiple survey modes.
This paper provides a framework for separating mode-effects from nonresponse- bias by matching similar respondents from different survey modes using propensity score matching. We use data from a multi-mode survey conducted in the Netherlands in 2007. About 14.000 people in our sampling frame were randomly assigned to a survey mode: either Internet or telephone. The internet-respondents were invited by mail. To assess the technique of propensity score matching, respondents from the Internet-survey are matched to participants in an access-panel who completed an identical survey. After matching, we find that differences in the composition of the Internet sample and access-panel can account for differences in our dependent variables; a set of items that ask respondents about perceived environmental hindrance.
In the second part of the paper, we assess the occurrence of mode effects in our survey experiment and match the respondents from the Telephone to the Internet-survey. After matching, we can account for differences due to nonresponse effects. The differences we found before matching however persist after matching. This indicates the occurrence of mode-effects in our telephone and Internet survey for the environmental hindrance questions. We conclude with a discussion how future mixed-mode survey designs can give more insight in the situations in which mode-effects occur.