Although probability-based online panels provide many advantages such as quality sample, higher response rates and potential for longitudinal design surveys, this paper discusses the limitations and drawbacks to the continued use of a probability based web panel. One limitation that hinders sample quality is when panel respondents become conditioned to taking surveys, and may start to demonstrate bad survey taking behaviors, such as non-response and satisficing.

This paper compares data from a tracking study that has historically relied on a nationally representative probability based online panel data which in recent years has reused panel members as a result of sample limitations to data obtained from an independent probability-based random sample data source. The surveys were administered parallel to each other, and were identical in appearance, content, and functionality.

Results indicate that meaningful differences in responses between the two samples were at least partially explained by the satisficing behavior of repeat panel respondents. The panel respondents were more likely than new survey respondents to select 'Not Sure’ where it was an option. While 'Not Sure’ can be a valid response, selecting 'Not Sure’ can also lower the burden associated with answering a question, especially one that requires more effort to process. These results suggest that data provided by probability based online panels should be approached with caution, as it may not be fully representative of respondents’ reactions to the questions.