Non-probability sample ("opt-in") Internet surveys are often used to measure how many people use various consumer products and to identify the characteristics of those product users. In this study, we evaluate the degree to which different opt-in panels produce the same results when used for such analyses. Seven opt-in Internet survey companies administered the same set of measures of product usage (each N~1000). We found that (1) the seven surveys varied widely in their estimates of how many people used various consumer products; (2) the market share of some brands, such as Coca-Cola, relative to other brands, such as Pepsi, were notably different depending on which survey vendor one used, and the ranking between these and other brands flipped depending on the vendor one used; and (3) the demographic characteristics of people using various products were sometimes the same across surveys and sometimes quite different. Weighting the data to yield optimal distributions of various demographic variables did not change the variability of survey results on consumer product usage across opt-in surveys. Overall, these results suggest caution before generalizing from a product usage survey
conducted with one opt-in Internet panel to the population of all such surveys.