Web Survey Bibliography
Concern about data falsification is as old as the profession of public opinion polling. However, the extent of data falsification is difficult to quantify and not well documented. As a result, the impact of falsification on statistical estimates is essentially unknown. Nonetheless, there is an established approach to address the problem of data falsification which includes prevention, for example by training interviewers and providing close supervision, and detection, such as through careful evaluation of patterns in the technical data, also referred to as paradata, and the substantive data.
In a recent paper, Kuriakose and Robbins (2015) propose a new approach to detecting falsification. The measure is an extension of the traditional method of looking for duplicates within datasets. What is new about their approach is the assertion that the presence of respondents that match another respondent on more than 85% of questions, what we refer to as a high match, indicates likely falsification. They apply this threshold to a range of publicly available international survey datasets and conclude that one-in-five international survey datasets likely contain falsified data.
The claim that there is widespread falsification in international surveys is clearly concerning. However, an extensive investigation conducted by Pew Research Center and summarized in this report finds the claim is not well supported. The results demonstrate that natural, benign survey features can explain high match rates. Specifically, the threshold that Kuriakose and Robbins propose is extremely sensitive to the number of questions, number of response options, number of respondents, and homogeneity within the population. Because of this sensitivity to multiple parameters, under real-world conditions it is possible for respondents to match on any percentage of questions even when the survey data is valid and uncorrupted. In other words, our analysis indicates the proposed threshold is prone to generating false positives – suggesting falsification when, in fact, there is none. Perhaps the most compelling evidence that casts doubt on the claim of widespread falsification is in the way the approach implicates some high-quality U.S. surveys. The threshold generates false positives in data with no suspected falsification but that has similar characteristics to the international surveys called into question.
This paper proceeds as follows. First, we briefly review the problem of data falsification in surveys and how it is typically addressed. Second, we summarize Kuriakose and Robbins’ argument for their proposed threshold for identifying falsified data and discuss our concerns about their evidence. Third, we outline the research steps we followed to evaluate the proposed threshold and then review in detail the results of our analysis. Finally, we conclude with a discussion of the findings and other ways the field is working to improve quality control methods.
Web survey bibliography (4086)
- Sample Representation and Substantive Outcomes Using Web With and Without Incentives Compared to Telephone...; 2016; Lipps, O.; Pekari, N.
- Effects of Data Collection Mode and Response Entry Device on Survey Response Quality; 2016; Ha, L.; Zhang, Che.; Jiang, W.
- The Dynamic Identity Fusion Index: A New Continuous Measure of Identity Fusion for Web-Based Questionnaires...; 2016; Jimenez, J.; Gomez, A.; Buhrmester, M.; Whitehouse, H.; Swann, W. B.
- Recommended Practices for the design of business surveys questionnaires; 2016; Macchia, S.
- Navigation Buttons in Web-Based Surveys: Respondents’ Preferences Revisited in the Laboratory; 2016; Romano Bergstrom, J. C.; Erdman, C.; Lakhe, S.
- Collecting Data from mHealth Users via SMS Surveys: A Case Study in Kenya; 2016; Johnson, D.
- “Money Will Solve the Problem”: Testing the Effectiveness of Conditional Incentives for...; 2016; DeCamp, W.; Manierre, M. J.
- Effects of Incentive Amount and Type of Web Survey Response Rates; 2016; Coopersmith, J.; Vogel, L. K.; Bruursema, T.; Feeney, K.
- Effect of a Post-paid Incentive on Response to a Web-based Survey; 2016; Brown, J. A.; Serrato, C. A.; Hugh, M.; Kanter, M. H.; A.; Spritzer, K. L.; Hays, R. D.
- Web-based versus Paper-based Survey Data: An Estimation of Road Users’ Value of Travel Time Savings...; 2016; Kato, H.; Sakashita, A.; Tsuchiya, Tak.
- Reminder Effect and Data Usability on Web Questionnaire Survey for University Students; 2016; Oishi, T.; Mori, M.; Takata, E.
- Feasibility of using a multilingual web survey in studying the health of ethnic minority youth.; 2016; Kinnunen, J. M.; Malin, M.; Raisamo, S. U.; Lindfors, P. L.; Pere, L. A.; Rimpelae, A. H.
- Respondents of a follow-up web-based survey; 2016; Stoddard, S. A.; Amparo, P.; Popick, H.; Yudd, R.; Sujeer, A.; Baath, M.
- An Examination of Opposing Responses on Duplicated Multi-Mode Survey Responses; 2016; Djangali, A.
- Is One More Reminder Worth It? If So, Pick Up the Phone: Findings from a Web Survey; 2016; Lin-Freeman, L.
- Reducing Underreports of Behaviors in Retrospective Surveys: The Effects of Three Different Strategies...; 2016; Lugtig, P. J.; Glasner, T.; Boeve, A.
- Dropouts in Longitudinal Surveys; 2016; Lugtig, P. J.; De Leeuw, E. D.
- Participant recruitment and data collection through Facebook: the role of personality factors; 2016; Rife, S. C.; Cate, K. L.; Kosinski, M.; Stillwell, D.
- What drives the participation in a monthly research web panel? The experience of ELIPSS, a French random...; 2016; Legleye, S; Cornilleau, A.; Razakamanana, N.
- When Should I Call You? An Analysis of Differences in Demographics and Responses According to Respondents...; 2016; Vicente, P.; Lopes, I.
- Evaluating a New Proposal for Detecting Data Falsification in Surveys; 2016; Simmons, K.; Mercer, A. W.; Schwarzer, S.; Courtney, K.
- Quantifying Under- and Overreporting in Surveys Through a Dual-Questioning-Technique Design. ; 2016; de Jong , M.; Fox, J.-P.; Steenkamp, J. - B. E. M.
- The use and positioning of clarification features in web surveys; 2016; Metzler, A., Kunz, T., Fuchs, M.
- Take the money and run? Redemption of a gift card incentive in a clinician survey. ; 2016; Chen, J. S.; Sprague, B. L.; Klabunde, C. N.; Tosteson, A. N. A.; Bitton, A.; Onega, T.; MacLean, C....
- Online Surveys are Mixed-Device Surveys. Issues Associated with the Use of Different (Mobile) Devices...; 2016; Toepoel, V.; Lugtig, P. J.
- Mail merge can be used to create personalized questionnaires in complex surveys. ; 2016; Taljaard, M.; Chaudhry, S. H.; Brehaut, J. C.; Weijer, C.; Grimshaw, J. M.
- Electronic and paper based data collection methods in library and information science research: A comparative...; 2016; Tella, A.
- Stable Relationships, Stable Participation? The Effects of Partnership Dissolution and Changes in Relationship...; 2016; Mueller, B.; Castiglioni, L.
- Identifying Pertinent Variables for Nonresponse Follow-Up Surveys. Lessons Learned from 4 Cases in Switzerland...; 2016; Vandenplas, C.; Joye, D.; Staehli, M. E.; Pollien, A.
- The 2013 Census Test: Piloting Methods to Reduce 2020 Census Costs; 2016; Walejko, G. K.; Miller, P. V.
- A Technical Guide to Effective and Accessible web Surveys; 2016; Baatard, G.
- The Validity of Surveys: Online and Offline; 2016; Wiersma, W.
- Methods can matter: Where Web surveys produce different results than phone interviews; 2016; Keeter, S.
- Computer-assisted and online data collection in general population surveys; 2016; Skarupova, K.
- Sunday shopping – The case of three surveys; 2016; Bethlehem, J.
- Solving the Nonresponse Problem With Sample Matching?; 2016
- Will They Stay or Will They Go? Personality Predictors of Dropout in Online Study; 2016; Nestler, S.; Thielsch, M.; Vasilev, E.; Back, M.
- Do Polls Still Work If People Don't Answer Their Phones?; 2016; Edwards-Levy, A.; Jackson, N. M.
- HUFFPOLLSTER: Why Reaching Latinos Is A Challenge For Pollsters; 2016; Jackson, N. M.; Edwards-Levy, A.; Velencia, J.
- A Framework of Incorporating Thai Social Networking Data in Online Marketing Survey; 2016; Jiamthapthaksin, R.; Aung, T. H.; Ratanasawadwat, N.
- Creation and Usability Testing of a Web-Based Pre-Scanning Radiology Patient Safety and History Questionnaire...; 2016; Robinson, T. J.; DuVall, S.; Wiggins III, R
- Comprehension and engagement in survey interviews with virtual agents; 2016; Conrad, F. G.; Schober, M. F.; Jans, M.; Orlowski, R. A.; Nielsen, D.; Levenstein, R. M.
- Development of a scale to measure skepticism toward electronic word-of-mouth; 2016; Zhang, Xia.; Ko, M.; Carpenter, D.
- Improving social media measurement in surveys: Avoiding acquiescence bias in Facebook research; 2016; Kuru, O.; Pasek, J.
- Psychological research in the internet age: The quality of web-based data; 2016; Ramsey, S. R.; Thompson, K. L.; McKenzie, M.; Rosenbaum, A.
- Internet Abusive Use Questionnaire: Psychometric properties; 2016; Calvo-Frances, F.
- Revisiting “yes/no” versus “check all that apply”: Results from a mixed modes...; 2016; Nicolaas, G.; Campanelli, P.; Hope, S.; Jaeckle, A.; Lynn, P.
- The impact of academic sponsorship on Web survey dropout and item non-response; 2016; Allen, P. J.; Roberts, L. D.
- Moderators of Candidate Name-Order Effects in Elections: An Experiment; 2016; Kim, Nu.; Krosnick, J. A.; Casasanto, D.
- Predictive inference for non-probability samples: a simulation study ; 2016; Buelens, B.; Burger, J.; van den Brakel, J.