Notice: the WebSM website has not been updated since the beginning of 2018.

Web Survey Bibliography

Title Web Probing for Question Evaluation: The Effects of Probe Placement
Year 2016
Access date 09.06.2016
Though the majority of cognitive interviewing is conducted in a laboratory environment by specialized interviewers, an alternative procedure,embedded probing, was developed decades ago (Converse & Presser, 1986; Schuman, 1966). Embedded probing involves the evaluation of select items through the administration of cognitive probes within a field survey environment, relying on much larger samples than are cost-effective with standard cognitive interviewing. Embedded probing has recently resurfaced in the form of web probing, mainly prompted by the increasing availability of internet-based platforms that provide fast, inexpensive access to respondents (Behr, Braun, Kaczmirek, & Bandilla,
2013; Murphy, Keating, and Edgar, 2013). However, little guidance exists concerning specific practices. In particular, as for standard cognitive interviewing, it is unclear whether probes should be administered concurrently with the target item, or retrospectively at the end of the interview (i.e. debriefing). This study examined web-based probing in the context of questionnaire pretesting and evaluation, relying on respondents from Amazon’s Mechanical Turk (mTurk), an online labor crowdsourcing platform. A questionnaire segment on walking behavior and attitudes, selected from the National Health Interview Survey (NHIS), was presented to 1,444 mTurk respondents. Probes were embedded for 4 targeted items. For example, respondents were asked: “When answering the question -- Where you live, are there roads, sidewalks, paths, or trails where you can walk? – Please say more about what you were thinking about...”, then typed their response to the probe into an open text box. As an experimental manipulation, approximately half of respondents answered the probes concurrently, and the remainder responded retrospectively. We will present the results of an analysis that compares the responses obtained as a function of probe placement, by assessing quantitative metrics such as number of words entered, as well as qualitative measures, including thematic coding and rater judgment of usefulness of the information obtained.
Year of publication2016
Bibliographic typeConferences, workshops, tutorials, presentations

Web survey bibliography - The American Association for Public Opinion Research (AAPOR) 71st Annual Conference, 2016 (107)