Relevance & Research Question: More and more surveys use multiple modes, which supplement or replace traditional interviewer modes by web. In multi-mode questionnaire design, usually some consideration is given to mode-specific measurement error. Despite this consideration, however, these measurement effects are frequently unexpectedly large and hamper publication. For this reason, there is a strong incentive to better predict measurement effects. Measurement effects are determined by the interplay between characteristics of the questionnaire and characteristics of the respondents. In our research, we investigate the existence and utility of so-called questionnaire and respondent profiles, in which these characteristics are summarized, for predicting measurement effects. As a first research question, we ask whether questionnaires can be coded reliably on item characteristics that are suggested in the literature as influential in mode-specific measurement effects.

Methods & Data: We constructed a typology of item characteristics from the literature and applied it to a wide range of surveys; the Dutch Labour Force Survey of Statistics Netherlands and the core studies of the LISS panel of CentERdata. For all surveys, 16 item characteristics are coded by two main coders, while 7 of these 16 item characteristics that are assumed to be relatively influential in evoking measurement error are also coded by a third coder. Reliability diagnostics are derived for the various item characteristics.

Results: Analyses of the survey coding scores indicate a relatively low reliability for characteristics that literature suggested as influential in mode-specific measurement effects: Sensitivity to social desirable answers, potential presumption of filter question, emotional charge, centrality, and language complexity.

Added Value: It is investigated to what extent coding of questionnaires on its item characteristics is reliable and to what extent questionnaire profiles can be constructed based on this coding. Along with process data and register data that are linked to individual respondents who have filled out multiple questionnaires, the questionnaire and respondent profiles might shed light on the occurrence and scope of measurement effects for specific respondents and specific questionnaire characteristics over different survey modes.