Notice: the WebSM website has not been updated since the beginning of 2018.

Web Survey Bibliography

Title Professional Web Respondents and Data Quality
Year 2010
Access date 28.06.2011
Abstract

There is concern that professional respondents, motivated more by monetary incentives than the opportunity to provide thoughtful answers, may compromise data quality in on-line surveys. The worry is that such respondents primarily take actions on the “critical path” toward their monetary goal even though valid answers may require additional, “off-path” actions. Our main questions are whether professional respondents are quicker to find the critical path and adhere to it more closely than "non-professionals." We explored this in a web survey in which 2404 volunteer respondents answered two constant-sum questions, i.e., multi-part items whose answers must add to 100% (fourteen internet use categories) or 24 hours (eleven daily activities). Respondents could not advance unless their answers were well-formed (i.e., equal to the target sum). About one third of the respondents could click for a running tally to facilitate well-formedness, placing this action on the critical path. We considered respondents who belonged to more than 3 volunteer panels to be professional (58% had completed more than 30 web surveys). This distinction mattered. Professionals were more likely than non-professionals to request a tally for the first item but the quality of both groups’ responses benefited from the tally: more non-zero, non-missing answers and fewer rounded responses with than without the tally. However, by the second item, professional respondents seemed to learn that providing valid answers was not on the critical path. While professionals again requested the tally more often, the tally only improved the quality of answers for nonprofessionals: more non-zero, non-missing answers and less time spent on “other” activities with than without the tally; professionals’ answers showed no such improvement from the tally. The results suggest that professional respondents are more likely to provide well-formed than valid answers because validity is not on the critical path.

Access/Direct link

Conference Homepage (abstract)

Year of publication2010
Bibliographic typeConferences, workshops, tutorials, presentations
Print

Web survey bibliography - The American Association for Public Opinion Research (AAPOR) 65th Annual Conference, 2010 (30)