Notice: the WebSM website has not been updated since the beginning of 2018.

Web Survey Bibliography

Title The Measurement of Consistency in Online Research
Source CASRO Journal, 2012-2013, pp. 61-63
Year 2012
Access date 03.09.2013
Full text

pdf (988 KB)


It wasn't too long ago that we relied on probabilistic models for collecting our  data.  Telephone facilities were quite adept at the pursuit of respondents. Studies with 70 percent recovery rates  from  duly harassed respondents were  accomplished with considerable effort. Recovery rate itself was an important metric in evaluating the utility of research data. Those days are go ne. Telephone has lost its luster,but this is becoming an old story now. Most practitioners only remotely recall those halcyon days when accurately representing a population through appropriate data collection methods was our calling. As online research gained traction we held  on to the  high bar that a representative sample frame implied. Our claim to a representative sample frame became our  weak point. Academics assailed us mercilessly that our samples were not  properly grounded in theory to achieve anything that might be representative. As we recoiled off the ropes some of us called for a retreat from claims that  were  indefensible.Instead,we advocated that the industry pursue consistency in its research. A consistent sample has now become the substitute for a representative one. Well,not really. That  dirty little "r" word continues to haunt us. Consistency would seem easy to measure.If your  data remains constant then you are consistent. But what  if the world is changing? Should our  data remain constant? Clients need  to know if the change  that they see in data is real or the artifact of changes in the sample frame. If the sample frame changes  without warning, then there can be little assurance that the shifts  we see are real. To confound things, there ha s bee n a considerable amount of data  that has demonstrated differences between panel sources. There  is much  for us to fear as panels change  their sourcing,merge with one another, change incenting models and go from high  to low usage periods. With all these  drivers of variability, why should we expect online panels to be capable of consistency in the data that they  collect? In fact,convenience sampling is a prescription for instability,a warning that things might change. The maintenance of a stable  sample  frame requires effort. We ask ourselves, are online panels capable  of sustaining consistency? After  all,they seem  to be standing on shifting sands.

Access/Direct link

CASRO Journal Homepage (abstract) / (full text)

Year of publication2012
Bibliographic typeJournal article