Relevance & Research question
Mobile phones are replacing the PC as key devices in social science data collection. In daily life, mobile phones are used for short interactions. Successful data collection strategies over mobile phones should therefore also be brief for respondents.
Questionnaires for attitude research are often very long. We argue that there is a trade-off to be made. Should questionnaires on mobile devices remain long, risking dropout, or should such questionnaires be split up (from here on called chunks) to optimize data quality?

Methods and data

We report on an experiment conducted in the probability-based LISS panel in the Netherlands, carried out in December 2015. We used a ‘within’ design of data chunking. Panelists who owned a mobile phone with Internet connection were randomly assigned to either:

a) The normal survey (about 20 min)

b) The same survey cut into three chunks, with each chunk offered after a week

c) The same survey cut into ten chunks, with each chuck offered every other day.

Results

First, we investigated the number of complete and incomplete responses and looked at indicators for data quality (straightlining, primacy effects, survey length). We find that more respondents are completing the questionnaire when it is offered in chunks (condition b, and especially c), but also that chunking results in more item missings. We find little evidence for effects on data quality.

Finally, we report on the differences we find in the factor structure when the questionnaire was split into chunks, or was completed in one go.

Added value

The idea of data chunking is not new. ‘Planned-missingness’ designs have been implemented in web surveys successfully in the past. This study is however the first to study data chunking in the setting of mobile phone surveys. We believe that more and more data will be collected using mobile phones (already 5-25% of all web surveys are taken on mobile phones), and that understanding how to design questionnaires for mobile phones is of vital importance to both survey researchers, market researchers, and anyone using such data for substantive reasons in the future.