Key goals of effective online survey design are to minimize missing data and to optimize “flow”, through the instrument. This paper builds on methodological research on missing data and visual design theories to optimize the design of online instruments.
We will present results from two experiments. In the first, we explore strategies for dealing with non-substantive responses, such as “don’t know’s” or skipped questions. In market research, it is common to require respondents to answer each question before proceeding (restrictive approach). In academic research, it is common to let respondents skip questions they would rather not answer (permissive approach). Between these extremes is a strategy that encourages respondents to provide an answer by prompting them if they skip a question (encouraging approach). We tested these three alternatives, along with the presence or absence of a “don’t know” option in a fully-crossed (3*2) design. The instrument contains a variety of different question types including grid questions and items known to elicit high levels of nonresponse. Both attitudinal and behavior questions are included. We examine the effect of these design formats on breakoffs, speed of completion, item non-response, non-substantive responses, and response distributions.
In the second experiment, we explore the placement of the navigation (next, back) buttons in Web surveys. Some surveys (typically those that require answers) include only a “next” button and do not permit respondent to go back. Where both buttons are included, the logical placement is “back” on the left and “next” on the right, but this puts the “back” button in the main visual field below the response options (left-aligned) and makes it the default (on-focus) button after selecting a response. We evaluate several different design options for button placement, with a focus on their effect on speed of completion, backwards movement, and changed answers.
Both experiments are included in a survey of opt-in panel members (n=7200). The purpose of these experiments is to explore optimal design solutions for common problems faced by designers of online surveys, with the goal of maximizing data quality and enhancing the respondent experience. Both extend previous work we have done on these issues.