Commercial on-line panels based on non-probability samples have begun to be widely used not only in traditional market research but also in more academic research. But is the quality and accuracy of such data comparable to that of probability based samples? The overall aim of this study is to compare the quality of probability based and non-probability based on-line panels in Sweden, a country with almost full internet coverage. We proceed in two steps. Firstly, we compare the accuracy of three survey modes all using random samples: postal survey, telephone survey, and web survey. Secondly, we compare the accuracy of two commercial non-probability based panels with two commercial probability based panels, using the traditional mail and telephone surveys as benchmark surveys. Demographics are compared to government records, and attitudes are compared to benchmark studies of high quality and high response rate.

In order to allow comparisons, seven surveys with comparable questions were run at approximately the same time. We compare the accuracy of the four commercial on-line panels both with and without weights. In contrast to previous studies, the results indicate a surprising similarity in terms of accuracy between probability panels and non-probability panels. The two non-probability based on-line panels do not seem to be less accurate than probability based on-line panels in terms of demographics, nor do their estimates of political attitudes seem to differ more from traditional methods such as a high response rate mail survey. We conclude that a larger comparison based on more demographic and attitudinal variables are needed.