Online polls and surveys using volunteer panels have grabbed a large slice of the research data collection market largely because they offer fast and cost effective research solutions. Concerns over the level of internet penetration and the use of volunteer panels have in many cases been set aside, as has research evidence showing sharp differences between the results of online and conventional telephone and face-to-face surveys.

We set out to find effective weighting systems that would bring the results of on-line surveys into line with telephone and other large scale random surveys. Disappointingly, we found that the differences could not be removed by weighting by demographics, newspaper readership or by using attitudinal weighting variables. Instead, we found evidence of significant and disturbing mode effects that seem to arise largely because panel members are primarily motivated to participate in surveys by the money they hope to earn, and in some cases seem to expend little time and energy doing so. The paper suggests treating panel members as valued employees, appropriately paid for carefully considered answers and aware they are subject to rigorous quality control procedures. These procedures are designed to weed out panel members who do not play fair, and in this regard the paper sets out some alternative strategies that might be employed.employed.Online polls and surveys using volunteer panels have grabbed a large slice of the research data collection market largely because they offer fast and cost effective research solutions. Concerns over the level of internet penetration and the use of volunteer panels have in many cases been set aside, as has research evidence showing sharp differences between the results of online and conventional telephone and face-to-face surveys. We set out to find effective weighting systems that would bring the results of on-line surveys into line with telephone and other large scale random surveys. Disappointingly, we found that the differences could not be removed by weighting by demographics, newspaper readership or by using attitudinal weighting variables. Instead, we found evidence of significant and disturbing mode effects that seem to arise largely because panel members are primarily motivated to participate in surveys by the money they hope to earn, and in some cases seem to expend little time and energy doing so.

The paper suggests treating panel members as valued employees, appropriately paid for carefully considered answers and aware they are subject to rigorous quality control procedures. These procedures are designed to weed out panel members who do not play fair, and in this regard the paper sets out some alternative strategies that might be employed.