Web Survey Bibliography
Title Fieldwork Effort, Response Rate, and the Distribution of Survey Outcomes: A Multilevel Meta-analysis
Author Sturgis, P.; Williams, Jo.; Brunton-Smith, I.; Moore, J.
Source Public Opinion Quarterly (POQ); 81, 2, pp. 523-542
Year 2016
Database Oxford Journals
Access date 23.08.2017
Abstract This study assesses how survey outcome distributions change over repeated calls made to addresses in face-to-face household interview surveys. We consider this question for 541 survey variables, drawn from six major face-to-face UK surveys that have different sample designs, cover different topic areas, and achieve response rates between 54 and 76 percent. Using a multilevel meta-analytic framework, we estimate for each survey variable the expected difference between the point estimate for a proportion at call n and for the full achieved sample. Results show that most variables are surprisingly close to the final achieved sample distribution after only one or two call attempts and before any post-stratification weighting has been applied; the mean expected difference from the final sample proportion across all 559 variables after one call is 1.6 percent, dropping to 0.7 percent after three calls and to 0.4 percent after five calls. These estimates vary only marginally across the six surveys and the different types of questions examined. Our findings add weight to the body of evidence that questions the strength of the relationship between response rate and nonresponse bias. In practical terms, our results suggest that making large numbers of calls at sampled addresses and converting “soft” refusals into interviews are not cost-effective means of minimizing survey error.
Access/Direct link Journal Homepage (abstract) / (full text)
Year of publication2016
Bibliographic typeJournal article
Web survey bibliography - Public Opinion Quarterly (POQ) (90)
- Necessary but Insufficient: Why Measurement Invariance Tests Need Online Probing as a Complementary...; 2017; Meitinger, K.
- Nonresponse in Organizational Surveying: Attitudinal Distribution Form and Conditional Response Probabilities...; 2017; Kulas, J. T.; Robinson, D. H.; Kellar, D. Z.; Smith, J. A.
- Theory and Practice in Nonprobability Surveys: Parallels between Causal Inference and Survey Inference...; 2017; Mercer, A. W.; Kreuter, F.; Keeter, S.; Stuart, E. A.
- Is There a Future for Surveys; 2017; Miller, P. V.
- Respondent mode choice in a smartphone survey ; 2017; Conrad, F. G., Schober, M. F., Antoun, C., Yan, H. Y., Hupp, A., Johnston, M., Ehlen, P., Vickers, L...
- Effects of Mobile versus PC Web on Survey Response Quality: a Crossover Experiment in a Probability...; 2017; Antoun, C.; Couper, M. P.; G. G.Conrad, F. G.
- The Low Response Score (LRS): A Metric to Locate, Predict, and Manage Hard-to-Survey Populations; 2016; Erdman, C.; Bates, N.
- Targeted Appeals for Participation in Letters to Panel Survey Members; 2016; Lynn, P.
- Fieldwork Effort, Response Rate, and the Distribution of Survey Outcomes: A Multilevel Meta-analysis; 2016; Sturgis, P.; Williams, Jo.; Brunton-Smith, I.; Moore, J.
- Measuring Generalized Trust: An Examination of Question Wording and the Number of Scale Points; 2016; Lundmark, S.; Giljam, M.; Dahlberg, S.
- Social Media Analyses for Social Measurement; 2016; Schober, M. F.; Pasek, J.; Guggenheim, L.; Lampe, C.; Conrad, F. G.
- Do Attempts to Improve Respondent Attention Increase Social Desirability Bias?; 2015; Clifford, S.; Jerit, J.
- Response Rates, Nonresponse Bias, and Data Quality: Results from a National Survey of Senior Healthcare...; 2015; Meterko, M.; Restuccia, J. D.; Stolzmann, K.; Mohr, D.; Brennan, C. W.; Glasgow, J.; Kaboli, P.
- Respondent Screening and Revealed Preference Axioms: Testing Quarantining Methods for Enhanced Data...; 2015; Jones, M. S.; House, L. A.; Zhifeng, G.
- Exploring the Effects of Removing "Too Fast" Responses and Respondents from Web Surveys; 2015; Greszki, R.; Meyer, M.; Schoen, H.
- The Effects of the Direction of Rating Scales on Survey Responses in a Telephone Survey; 2015; Keusch, F., Yan, T.
- Assessing the Potential of Paradata and Other Auxiliary Data for Nonresponse Adjustments; 2014; Krueger, B. S., West, B. T.
- Improving Response Rates and Questionnaire Design for Mobile Web Surveys; 2014; de Bruijne, M., Wijnant, A.
- Assessing Within-Household Selection Methods in Household Mail Surveys; 2014; Olson, K., Stange, M., Smyth, J. D.
- Mobile Technologies for Conducting, Augmenting and Potentially Replacing Surveys: Report of the AAPOR...; 2014; Link, M. W., Murphy, J., Schober, M. F., Buskirk, T. D., Childs, J. H., Tesfaye, C.
- Clicking vs. Dragging: Different Uses of the Mouse and Their Implications for Online Surveys; 2014; Sikkel, D., Steenbergen, R., Gras, S.
- The Effect of Benefit Wording on Consent to Link Survey and Administrative Records in a Web Survey; 2014; Sakshaug, J. W., Kreuter, F.
- Video Content in Web Surveys: Effects on Selection Bias and Validity; 2013; Shapiro-Luft, D., Cappella, J.
- Panel Conditioning in Difficult Attitudinal Questions; 2013; Binswanger, J., Schunk, D., Toepoel, V.
- Clarifying Categorical Concepts in a Web Survey.; 2013; Redline, C. D.
- Recruiting A Probability Sample For An Online Panel: Effects Of Contact Mode, Incentives, And Information...; 2012; Scherpenzeel, A., Toepoel, V.
- Does Giving People Their Preferred Survey Mode Actually Increase Survey Participation Rates?; 2012; Olson, K., Smyth, J. D., Wood, H.
- The changing role of address-based sampling in survey research; 2011; Iannacchione, V. G.
- Measuring americans' issue priorities. A new version of the most important problem question reveals...; 2011; Yeager, D. S., Larson, S. B., Krosnick, J. A., Tompson, T.
- Questions for Surveys: Current Trends and Future Directions; 2011; Schaeffer, N. C., Schaeffer, N. C.
- The Future of Modes of Data Collection; 2011; Couper, M. P.
- The Future of Survey Sampling; 2011; Brick, J. M.
- The Impact of “Forgiving” Introductions on the Reporting of Sensitive Behavior in Surveys...; 2011; Peter, J., Valkenburg, P. M.
- Surveying the General Public over the Internet Using Address-Based Sampling and Mail Contact Procedures...; 2011; Messer, B. L., Dillman, D. A.
- Use of Cognitive Shortcuts in Landline and Cell Phone Surveys; 2011; Everett, S. E., Kennedy, C.
- An Alternative to the Response Rate for Measuring a Survey's Realization of the Target Population; 2011; Skalland, B.
- Can Verbal Instructions Counteract Visual Context Effects in Web Surveys?; 2011; Toepoel, V., Couper, M. P.
- Nonresponse Error, Measurement Error, And Mode Of Data Collection: Tradeoffs in a Multi-mode Survey...; 2011; Sakshaug, J. W., Yan, T., Tourangeau, R.
- A Method for Evaluating Mode Effects in Mixed-mode Surveys; 2011; Vannieuwenhuyze, J., Loosveldt, G., Molenberghs, G.
- Total Survey Error: past, present, and future; 2010; Groves, R. M., Lyberg, L. E.
- Research synthesis. AAPOR report on online panels; 2010; Brick, J. M., Baker, R., Blumberg, S. J., Couper, M. P., Courtright, M., Dennis, J. M., Dillman, D....
- Recruiting probability samples for a multi-mode research panel with Internet and mail components; 2010; Rao, K.
- Cell-Phone-Only Voters in the 2008 Exit Poll and Implications for Future Noncoverage Bias ; 2009; Mokrzycki, M., Keeter, S., Kennedy, C.
- Zero Banks: Coverage Error and Bias in Rdd Samples Based on Hundred Banks with Listed Numbers ; 2009; Boyle, J., Bucuvalas, M., Piekarski, L., Weiss, A.
- National Surveys Via RDD Telephone Interviewing vs. the Internet: Comparing Sample Representativeness...; 2009; Chang, L. C., Krosnick, J. A.
- Impact of T-ACASI on Survey Measurements of Subjective Phenomena ; 2009; Harmon, T., Rogers, S. M., Eggleston, E., Roman, A. M., Villarroel, M. A., Chromy, J. R., Ganapathi,...
- Open-Ended Questions in Web Surveys: Can Increasing the Size of Answer Boxes and Providing Extra Verbal...; 2009; Smyth, J. D., Dillman, D. A., Christian, L. M., McBride, M.
- Web Survey Methods: Introduction; 2009; Couper, M. P., Miller, P. V.
- Social desirability bias in CATI, IVR and Web surveys: The effects of mode and question sensitivity; 2008; Kreuter, F., Presser, S., Tourangeau, R.
- Does a Probability-Based Household Panel Benefit from Assignment to Postal Response as an Alternative...; 2008; Rookey, B. D., Hanway, S., Dillman, D. A.