As more survey modes have become available, it has ever been an aspiration in survey methodology research to test and assure comparability of measurements obtained in different modes. Such research has often found 'mode effects’, which are measurement differentials between modes beyond sampling variation. Moreover, with the rise of the mixed-mode survey, which combines more than one survey mode in the data collection design , the question of comparability has become even more pressuring.
The reason why estimates based on different survey modes are known to differ is bias introduced by mode-specific survey nonresponse and mode-specific measurement error. The major problem in such research has remained, however, that these errors are confounded in studies experimentally comparing two or more survey modes. Comparative studies therefore should strive to control for mode-specific nonresponse before observing residual measurement differentials.
We review commonly used techniques useful for this goal and then present results from an experiment, in which the same survey project was administered in four different modes (CAPI, CATI, CAWI, and PAPI). About 8000 respondents were randomly sampled from the Dutch population and assigned to one of the modes. In the analysis we use structural modeling to test measurement invariance across survey modes on a set of items taken from round five of the European Social Survey (ESS). We control for mode-specific survey nonresponse using auxiliary information from various registries . The adjustment is expected to be imperfect due to the typically low explanatory power of registry data; however, the adjustment situation is comparable to that encountered in practical fieldwork. Hence if measurement invariance between some modes is rejected , this will have strong implications for measurements realized in mixed-mode surveys and the practical researcher’s means to adjust for bias in mixed-mode surveys.