Web Survey Bibliography
Relevance & Research Question
The market research industry is wedded to quota controls. We apply Age and Gender quotas without a second thought as to why or indeed whether they are doing any good at all. Our argument is that, in the modern online sampling world, a different set of stratifications must be applied and our old assumptions simply do not apply. Why not? The answer, in common with so many of the problems in sampling in online research, lies in the frame. The frame in traditional research was close to the population; therefore a quota controlled random sample would tend to produce samples that, within the quota strata, also contained representative numbers of all other attitudes and behaviours. This is not the case with online access panels.
Methods and Data
Our experiment uses our US panel; the topic, eye colour, is unrelated to Age and Gender but is strongly related to Ethnicity. We have conducted 2 samples. The first strictly controlled on Age, Gender and Region, the second controlled on Ethnicity alone. Our Age Gender Region ‘nat rep’ sample should underestimate the number with brown eyes. The Ethnicity we expect to estimate eye colour extremely well. At the same time a third sample will be drawn which is simply “random enough”. Our expectation is that this sample will also under-perform on eye colour but will equal the findings from “nat rep” sample 1. A second experiment will be undertaken where the variable of interest is unrelated to anything –left- or right-handedness. Our hypothesis is that all three samples will perform equally well.
The results are precisely as predicted.
Researchers, particularly in the commercial world, apply quota controls to ensure “representivity” as a matter of practice, they do it because they have been told to, it is part of the folklore of market research. This is not sustainable in a world where we are no longer dealing with essentially incomplete frames. More science and less folklore needs to be applied to make the best of an increasing unscientific world.
Conference Homepage (abstract) / (presentation)
Web Survey Bibliography - Cape, P. J. (11)
- Do I Have Your Full Attention?; 2013; Cape, P. J.
- Rewards - Money for Nothing?; 2013; Cape, P. J., Martin, P.
- Solving the Mode Mystery The Cost, Coverage and Quality Tradeoffs of Picking (and Mixing) Online and...; 2012; Cape, P. J., Phillips, K.
- Quota Controls: Science or merely Sciencey?; 2011; Cape, P. J.
- Conditioning Effects in Online Communities; 2010; Cape, P. J.
- Trial by Ordeal, a medieval approach to a modern day problem; 2010; Cape, P., Cavallaro, K.
- The pros and cons of survey routers in online research; 2010; Cape, P. J.
- Verbal Vs Visual Response Options: Reconciling Meanings Conveyed by a Computer Aided Visual Rating Scale...; 2009; Garland, P., Cape, P.
- The Opportunity for Flash Scales in Online Surveys; 2009; Cape, P. J.
- Quality matters when designing panel questionnaires; 2008; Cape, P., Lorch, J., Piekarski, L.
- How not to kill the goose that lays the golden egg A new approach to incentives in online access panels...; 2006; Cape, P. J.