Web Survey Bibliography
Title Decomposing Selection Effects in Non-probability Samples
Author Mercer, A. W.; Keeter, S.; Kreuter, F.
Year 2016
Access date 09.06.2016
Abstract
Prior studies have found that survey estimates obtained using non-probability samples are sometimes very close to estimates that use probability-based methods, while at other times they differ substantially. To date, the conditions under which non-probability methods yield comparable estimates remain poorly understood. We propose a framework for understanding differences between probability and non-probability sample estimates as a function of four separate components, each with different root causes and requiring different approaches to remedying. First, differences may be due to confounding, or the presence of unobserved factors associated with both the survey outcome and inclusion in a sample. Second, differences may be due to the absence of certain classes of respondent from non-probability samples (such as the non-internet population in web panels). Third, differences may be due to the over or underrepresentation of certain types of respondents. Finally, differences may be reduced or magnified by post-survey adjustment such as weighting or sample matching. We use machine learning and causal inference methods to evaluate the relative contribution of each of these components in explaining observed differences between estimates generated from 11 parallel, non-probability web surveys and the Pew Research Center’s probability-based American Trends Panel. We look at the degree to which these components are consistent across different sample providers over a wide range of variables. Because the point of comparison is itself a survey rather than true values, these comparisons cannot be said to measure selection bias; however they can shed a great deal of light on dynamics that produce similarities or differences between different non-probability sample providers relative to a common, probability-based point of reference.
Access/Direct link Conference Homepage (abstract)
Year of publication2016
Bibliographic typeConferences, workshops, tutorials, presentations
Web survey bibliography - Keeter, S. (9)
- Theory and Practice in Nonprobability Surveys: Parallels between Causal Inference and Survey Inference...; 2017; Mercer, A. W.; Kreuter, F.; Keeter, S.; Stuart, E. A.
- Decomposing Selection Effects in Non-probability Samples ; 2016; Mercer, A. W.; Keeter, S.; Kreuter, F.
- Methods can matter: Where Web surveys produce different results than phone interviews; 2016; Keeter, S.
- App vs. Web for Surveys of Smartphone Users: Experimenting with mobile apps for signal-contingent experience...; 2015; McGeeney, K.; Keeter, S.; Igielnik, R.; Smith, A.; Rainie, L.
- Advance Postcard Mailing Improves Web Panel Survey Participation; 2015; Bertoni, N.; Burkey, A.; Caldaro, M.; Keeter, S.; DiSogra, C.; McGeeney, K.
- A Comparison of Results from Surveys by the Pew Research Center and Google Consumer Surveys; 2012; Keeter, S., Christian, L. M.
- Assessing Cell Phone Noncoverage Bias Across Different Topics and Subgroups; 2010; Christian, L. M., Keeter, S., Purcell, K., Smith, A.
- Do Landline RDD Samples Adequately Cover the "Wireless Mostly"?; 2009; Dimock, M., Christian, L. M., Keeter, S.
- Cell-Phone-Only Voters in the 2008 Exit Poll and Implications for Future Noncoverage Bias ; 2009; Mokrzycki, M., Keeter, S., Kennedy, C.