Since Peytchev and Crawford described a typology for web-based survey live-validations in the Social Science Computer Review in 2005, very little has been published documenting specific cases of validations being used, and whether they are serving their intended purpose. Arguably, validations should either improve data quality or reduce respondent burden within a survey (or both), however, the potential is there for a backfire, with validations increasing respondent burden and/or contributing to error.

In this paper, we will examine several web-based surveys conducted over the past several years on social science related topics among various populations, and using paradata collected during the data collection, we will identify what we have found happens when validations are used. We will look at the rate that they are triggered, when they are triggered (early vs. late in a questionnaire), what we know about how the respondent reacted to them, and where possible we will examine if their use had an impact on the survey response. We will also hypothesize as to whether the impact was positive or negative relating to survey quality. We will focus our attention on potential nonresponse as well as measurement errors.

While this paper will not report on any experiments designed to explore web based survey live validation design, it is intended to help identify and prioritize issues for further exploration (possibly in a controlled experiment) during future data collections.