Relevance & research question:

As ownership and use of smartphones grows, we need to adapt survey design to enable people to complete questionnaires on their chosen device, without impacting negatively on respondent experience or data quality.

Previous research highlights issues with grid questions on smartphones (e.g. McClain & Crawford, 2013). The ‘traditional’ grid format can appear cluttered on a smartphone screen, and this in turn can cause respondent burden and risks miscoding responses and higher drop out rates.

In this paper we consider alternatives to traditional grid formats and present experimental data to show the relationship between grid format and data quality.

Methods & data:

We present the results of an experiment comparing traditional grids with three alternatives: item by item scrolling, item by item paging and dynamic grids.

The experiment was run on the Kantar TNS online omnibus. The achieved sample size for each format was c. 1,200, including c. 250 respondents completing on a smartphone in each cell.

Alongside this experiment we have conducted usability testing with respondents to provide qualitative feedback on ease of use and perceptions of different grid formats.

Results:

We compared results from the formats over a number of analysis dimensions, including substantive responses, missing response levels, ‘Don’t know’ rates, question timings, flatlining, and respondent assessments. Results are also compared across device types and screen dimensions.

The results were broadly similar over the four formats, suggesting that for this sample/questionnaire the format did not substantially affect responses. However, there were differences in relation to question timings and missing response levels, with dynamic grids performing positively.

In the usability testing respondents found the dynamic grids to be engaging and intuitive, with many expressing a preference for this format.

Added value:

As social studies increasingly move online we need to ensure that questionnaires are optimised for mobile devices. This paper adds important evidence to one of the challenges associated with this shift: how to deal with grids. Through our experiment and usability testing we assess the pros and cons of alternative formats, including an interactive dynamic grids approach, and consider implications for adapting existing surveys.