Relevance & Research Question:

Previous research on clarification features in Web surveys has shown that they are an effective means of improving response quality in open-ended questions. However, little is known about their influence on response quality in closed-ended questions. Results from the literature indicate that respondents use the range and content of response categories as relevant information when generating an answer (scale effects). Given the findings concerning clarification features in open-ended questions we assume (1) that they are similarly effective in closed ended-questions and (2) that they may have a stronger effect on the response process than the range of the response categories potentially reducing scale effects.

Methods & Data:

Experiment 1 and 2 were conducted in two randomized field experimental Web surveys (n=4,620; n=944). Using a between-subjects design we assessed the effectiveness of clarification features in closed-ended frequency questions. Two types of clarification features were tested that aim at either clarifying the question meaning (definitions) or motivating respondents to search their memories for relevant information (motivating statements). Questions and clarification features were designed in a way that respondents in the experimental groups with the clarifications features were expected to provide either higher or lower frequencies than respondents in the control groups with no clarification features.

Experiment 3 was conducted in a randomized field experimental Web survey (n=944). A between-subjects design was implemented in closed-ended questions to test low and high frequency scales without clarification features, with definitions and motivating statements (2 x 3 factorial design). We assessed the magnitude of scale effects as a dependent variable.

Results:

Overall clarification features are effective in influencing responses provided. Results indicate that definitions yield stronger effects than motivating statements. Furthermore, scale effects are lower for respondents receiving clarification features than for respondents of the control group. Again, definitions are more effective in reducing the scale effect than motivating statements.

Added Value:

The use of definitions in closed-ended questions has a positive effect on survey responses and helps improve data quality. Definitions seem to have the potential to counteract scale effects, whereas motivating statements do not show any effect.