Relevance & Research Question:
In online surveys the questionnaire is the only relevant stimulus due to the absence of an interviewer. Accordingly, each element included in the questionnaire may have a crucial impact on the survey results. In order to gain insights on such influences, SevenOne Media conducted a complex split ballot. Three aspects were varied: the background colour of the pop-up with which the participants were recruited for onsite surveys (green, white, red and blue), the background colour of the questionnaire (same colours) and the forwarding mechanism (automatic vs. manual). The hypotheses tested in this study were built based on former research results both regarding methodological studies and research in other areas, such as website usability and behavioural science.
Methods & Data:
The invitation to participate in the survey – using the different experimental designs – was sent out toroughly 150,000 randomly selected users of four German internet pages (ProSieben.de, wetter.com, N24.de and MyVideo.de). About 7,200 persons responded to this invitation, of which 5,600 answered at least some of the questions. This enormous data basis enabled various analyses considering response rates and data quality.
Results:
The analyses show that colour has a surprisingly large impact on the willingness to participate. By using a green pop-up instead of a white one the number of accepted invitations was increased by 42 percent. The drop-out rate remained the same, leading to a higher return rate over all.
The forwarding mechanism had a much smaller impact on the return rate but the quality of the data varied strongly. In the group in which respondents were forwarded automatically, the number of missing values was reduced. However, several indicators show that at the same time, the number of randomly given answers increased, leading to more data noise.
Added Value:
Considering the fact that the implementation of aspects such as the background colour and the forwarding mechanisms are simple, this study adds an important value to online survey research as its results offers some easy options to improve quality without producing additional costs.