Relevance & Research Question: In the first computer-based survey by Bartram and Bayliss (1984, 221 ff.) was already recognized that “… it is probably easier to produce a slick-looking and interesting online test than it is to produce one which is valid and reliable“.
This contribution deals with the disclosure of reliability-influencing factors in online market research. In the offline market research, the influence on the research design to the reliability of a test has already been confirmed by several authors.
Closely linked with the results of offline research by Churchill Jr. and Peter (1984, 334 ff.) and Wilson (1995, 361 ff.) , the dependence of the following reliability factors such as the number of items, the sampling method in particular, and of the other offline factors in general was analyzed.
Methods & Data: Using desk research seven empirical studies were examined, of which there were four published studies and three were student projects.
In the first step, the reliability of each was determined. In the second step the dependencies were examined based on the determinants of reliability using correlation and regression analysis.
Results: The analyzed online studies showed a significant correlation between number of items and the reliability analogous to offline research. Regarding the sampling methods, especially for online panels, no correlation could be detected. Based on the already known offline results concerning different strength of the analyzed factors influencing reliability, a similar situation line could be detected.
Added Value: Despite any criticism of the classical test theory and its quality criteria in the field, the currently known offline factors that influence reliability are sufficiently applicable to determine the quality of the online survey. “Avoiding inverse phrased items” or “Using a constant scale level” are just two examples to improve the quality.
The manipulability of the level of reliability using the factor “number of items” should be judged critically. In general the level of reliability measured by Cronbach’s alpha is an appropriate tool for on-going review and improvement of tests. Further research is needed to verify and to improve the method of measurement.