Relevance & Research Question: Survey researchers have been forced—by time and budget constraints—to rely on a slew of sampling methods to estimate population parameters. To be valid, sampling procedures must a) have nearly 100% population coverage b) ensure equal chance of selection, and c) feature reasonable response rates. Convenience samples have long been considered inadequate for serious research but collecting opinions from probability samples now takes longer than conveniently interviewing millions of people.
Increasingly, RDD telephone sampling now features response rates below 20%. Moreover, the decrease in landline telephony and the corresponding rise in mobile have put pressure on the claim of near total coverage. Though all online interviewing relies on convenience sampling, it is arguable that people only answer their phones to take surveys (that are a part of RDD studies) when it is convenient for them—a group that continues to shrink.
Methods & Data: This paper investigates whether volume-based approaches to collecting opinions are reasonable substitutes for probability sampling. To do so, we exposed Gallup’s long-used, well documented, United States Presidential approval rating question to a random subset of people who recently completed a survey for one of SurveyMonkey’s seven million survey creators. We presented the question to roughly 10,000 survey takers per day from June 10, 2010 to July 29, 2010. In total, 87, 308 people answered the question.
Results: The results track Gallup’s daily approval numbers within the reported margin of error nearly each day that the test ran without mimicking Gallup’s use of statistical weighting. A closer look reveals that coverage and response rate were respectable relative to those of probability studies. Specifically, respondents were from 8,300 of 19,000 American cities (43%) which approaches the proportion of U.S. households with a landline telephone (60%). Moreover, the respondents yielded an average daily response rate of 46%--more than double that of a typical telephone survey.
Added Value: These findings shed light on recent interest by market researchers in gathering data from technology and social networking sources that have access to extremely large and diverse pools of people and can ask questions of them for virtually no cost.