Web Survey Bibliography
Title Apples to Oranges or Gala versus Golden Delicious?: Comparing Data Quality of Nonprobability Internet Samples to Low Response Rate Probability Samples
Author Dutwin, D.; Buskirk, T. D.
Source Public Opinion Quarterly (POQ); 81, 1, pp. 213-239
Year 2017
Database Oxford Journals
Access date 24.08.2017
Abstract Nonprobability samples have gained mass popularity and
use in many research circles, including market research and some political
research. One justification for the use of nonprobability samples is
that low response rate probability surveys have nothing significant to
offer over and above a “well built” nonprobability sample. Utilizing an
elemental approach, we compare a range of samples, weighting, and
modeling procedures in an analysis that evaluates the estimated bias of
various cross-tabulations of core demographics. Specifically, we compare
a battery of bias related metrics for nonprobability panels, dualframe
telephone samples, and a high-quality in-person sample. Results
indicate that there is roughly a linear trend, with nonprobability samples
attaining the greatest estimated bias, and the in-person sample, the
lowest. Results also indicate that the bias estimates vary widely for the
nonprobability samples compared to either the telephone or in-person
samples, which themselves tend to have consistently smaller amounts of
estimated bias. Specifically, both weighted and unweighted dual-frame
telephone samples were found to have about half the estimated bias
compared to analogous nonprobability samples. Advanced techniques
such as propensity weighting and sample matching did not improve
these measures, and in some cases made matters worse. Implications for
“fit for purpose” in survey research are discussed given these findings.
use in many research circles, including market research and some political
research. One justification for the use of nonprobability samples is
that low response rate probability surveys have nothing significant to
offer over and above a “well built” nonprobability sample. Utilizing an
elemental approach, we compare a range of samples, weighting, and
modeling procedures in an analysis that evaluates the estimated bias of
various cross-tabulations of core demographics. Specifically, we compare
a battery of bias related metrics for nonprobability panels, dualframe
telephone samples, and a high-quality in-person sample. Results
indicate that there is roughly a linear trend, with nonprobability samples
attaining the greatest estimated bias, and the in-person sample, the
lowest. Results also indicate that the bias estimates vary widely for the
nonprobability samples compared to either the telephone or in-person
samples, which themselves tend to have consistently smaller amounts of
estimated bias. Specifically, both weighted and unweighted dual-frame
telephone samples were found to have about half the estimated bias
compared to analogous nonprobability samples. Advanced techniques
such as propensity weighting and sample matching did not improve
these measures, and in some cases made matters worse. Implications for
“fit for purpose” in survey research are discussed given these findings.
Access/Direct link Journal Homepage (abstract) / (full text)
Year of publication2017
Bibliographic typeJournal article
Web survey bibliography (4086)
- Web Health Monitoring Survey: A New Approach to Enhance the Effectiveness of Telemedicine Systems; 2017; Romano, M. F.; Sardella, M. V.; Alboni, F.
- Gathering Opinions on Depression Information Needs and Preferences: Samples and Opinions in Clinic Versus...; 2017; Bernstein, M. T.; Walker, J. R.; Sexton, K. A.; Katz, A.; Beatie, B. E.
- Oversampling as a methodological strategy for the study of self-reported health among lesbian, gay and...; 2017; Anderssen, N.; Malterud, K.
- Device and Internet Use among Spanish-dominant Hispanics: Implications for Web Survey Design and Testing...; 2017; Trejo, Y. A. G.; Schoua-Glusberg, A.
- Utjecaj vizualne orientacije skale za odgovaranje i broja stranica web-upitnika na rezultate ispitivanja...; 2017; Malikovic, M.; Svegar, D.; Somodzi, S.
- How to Design a Web Survey Using Spring Boot With MYSQL: a Romanien Network Case Study; 2017; Bucea-Manea-Tonis, Ro.; Bucea-Manea-Tonis, Ra.
- Analyzing Survey Characteristics, Participation, and Evaluation Across 186 Surveys in an Online Opt-...; 2017; Revilla, M.
- Careless Response and Attrition as Sources of Bias in Online Survey Assessments of Personality Traits...; 2017; Meade, A. W.; Ward, M. K.; Alfred, C. M.; Pappalardo, G.; Stoughton, J. W.
- Using Mixed Methods to Research the Professional Development Needs of English Teacher Educators in PCET...; 2017; Eliahoo, R.
- Do Incentives Increase Response Rates to an Internet Survey of American Evaluation Association Members...; 2017; Wilson, L. N.
- Examining Completion Rates in Web Surveys via Over 25,000 Real-World Surveys; 2017; Liu, M.; Wronski, L.
- Data collection mode differences between national face-to-face and web surveys on gender inequality...; 2017; Liu, M.
- Improving survey response rates: The effect of embedded questions in web survey email Invitations; 2017; Liu, M.; Inchausti, N.
- An experimental comparison of web-push vs. paper-only survey procedures for conducting an in-depth health...; 2017; McMaster, H. S.; LeardMann, C. A.; Speigle, S.; Dillman, D. A.
- Demographic Question Placement: Effect on Item Response Rates and Means of a Veterans Health Administration...; 2017; Teclaw, R.; Price, M.; Osatuke, K.
- Effects of Applying Multimedia and Dialogue Box to Web Survey Design; 2017; Chen, H.
- Role of online survey tools in creating temporally accurate Environmental Product Declarations (EPD)...; 2017; Ganguly, I.; Bowers, T.; Pierobon, F.; Eastin, I.
- A test of sample matching using a pseudo-web sample; 2017; Chatrchi, G., Gambino, J.
- A Partially Successful Attempt to Integrate a Web-Recruited Cohort into an Address-Based Sample; 2017; Kott, P. S., Farrelly, M., Kamyab, K.
- Nonprobability sampling as model construction; 2017; Mercer, A. W.
- Comparing acquiescent and extreme response styles in face-to-face and web surveys; 2017; Liu, M.; Conrad, F. G.; Lee, S.
- Comparison of response patterns in different survey designs: a longitudinal panel with mixed-mode and...; 2017; Ruebsamen, N.; Akmatov, M. K.; Castell, S.; Karch, A.; Mikolajczyk, R. T.
- Mobile Research im Kontext der digitalen Transformation; 2017; Friedrich-Freksa, M.
- Kognitives Pretesting; 2017; Neuert, C.
- Grundzüge des Datenschutzrechts und aktuelle Datenschutzprobleme in der Markt- und Sozialforschung; 2017; Schweizer, A.
- Survey mode influence on patient-reported outcome scores in orthopaedic surgery: telephone results may...; 2017; Hammarstedt, J. E.; Redmond, J. M.; Gupta, As.; Dunne, K. F.; Vemula, S. P.; Domb, B. G.
- Marketing survey research best practices: evidence and recommendations from a review of JAMS articles...; 2017; Hulland, J.; Baumgartner, H.; Smith, K. M.
- Comparative analysis of a mobile device and paper as effective survey tools; 2017; Kim, K. J.; Bae, S.; Park, E.
- Enhancing survey participation: Facebook advertisements for recruitment in educational research; 2017; Forgasz, H.; Tan, H.; Leder, G.; McLeod, A.
- Determinants of polling accuracy: the effect of opt-in Internet surveys; 2017; Sohlberg, J.; Gilljam, M.; Martinsson, J.
- Virtual reality meets sensory research; 2017; Depoortere, L.
- Data chunking for mobile web: effects on data quality; 2017; Lugtig, P. J.; Toepoel, V.
- The Failure of the Polls: Lessons Learned from the 2015 UK Polling Disaster; 2017; Sturgis, P.
- Online customer journey analysis: a data science toolbox; 2017; Bonnay, D.
- Article Establishing an Open Probability-Based Mixed-Mode Panel of the General Population in Germany...; 2017; Bosnjak, M.; Dannwolf, T.; Enderle, T.; Schaurer, I.; Struminskaya, B.; Tanner, A.; Weyandt, K.
- PC, phone or tablet? Use, preference and completion rates for web surveys ; 2017; Brosnan, K.; Gruen, B.; Dolnicar, S.
- Comparing data quality and cost from three modes of on-board transit surveys ; 2017; Agrawal, A. W.; Granger-Bevan, S.; W.; Newmark, G. L.; Nixon, H.
- Web survey experiments on matrix questions; 2017; Liu, M.
- Web- and Phone-based Data Collection using Planned Missing Designs; 2017; Revelle, W.; Condon, M. D.; Wilt, J.; French, A. J.; Brown, A.; Elleman, G. L.
- Finding and Investigating Geographical Data Online; 2017; Martin, D.; Cockings, S.; Leung, S.
- CAQDAS at a Crossroads: Affordances of Technology in an Online Environment; 2017; Silver, C.; Bulloch, L. S.
- Online Focus Groups; 2017; Abrams, M. K.; Gaiser, T. J.
- Artificial Intelligence/Expert Systems and Online Research; 2017; Brent, E.
- Improving the Effectiveness of Online Data Collection by Mixing Survey Modes; 2017; Dillman, D. A.; Hao, F.; Millar, M. M.
- Online Survey Software; 2017; Kaczmirek, L.
- Online Survey Design; 2017; To, N.
- Sampling Methods for Online Surveys; 2017; Fricker, R. D.
- Research Design and Tools for Online Research; 2017; Hewson, C. M.
- Overview: Online Surveys; 2017; Vehovar, V.; Lozar Manfreda, K.
- Three Methods for Occupation Coding Based on Statistical Learning; 2017; Geweon, H.; Schonlau, L.; Blohum, M.; Steiner, St.