Over the last decades, a tremendous increase has occurred in cross-national data production in social science research (Harkness 2008). The large-scale provision and the wide-spread use of cross-national data sets constitute a huge opportunity for the research community but also pose the challenge to develop cross-national comparable survey items (Lynn, Japec, and Lyberg 2006). At the same time, substantive researchers are increasingly aware of the necessity to understand respondents’ cognitive processes when answering a survey question (Smith et al. 2011). Recently, the method of online probing has been developed that implements probing techniques from cognitive interviewing in web surveys. In the traditional probing approach, interviewers obtain additional information by asking follow-up questions called probes (Beatty and Willis 2007). In contrast, online probing transfers probing questions as open-ended questions in the web context. It can reveal the cognitive processes of web survey participants and it helps to assess whether respondents’ interpretations of an item differ across countries (Braun et al. 2015).

The implementation of probes within web surveys offers respondents a higher level of anonymity of their answers in comparison to the laboratory situation during cognitive interviewing (Behr and Braun 2015), which potentially reduces social desirability effects in the response process (Bethlehem and Biffignandi 2012). Online probing can easily realize large samples sizes, which increases the generalizability of the results, enables an evaluation of the prevalence of problems or themes, and can explain the response patterns of specific subpopulations (Braun et al. 2015). Since all probes have to be programmed in advance, all respondents receive the same probe, and the procedure is highly standardized (Braun et al. 2015). When applied to cross-national data, online probing is a powerful tool to assess the comparability of questions. In contrast to traditional quantitative approaches to assess the equivalence of items (e.g., measurement invariance tests), online probing can explain why respondents in certain countries might misunderstand a specific item or why they adopt different perspectives when providing a response (Behr et al. 2014a).

The overarching goal of this dissertation project is to explore the potential of the method of online probing vis-à-vis other relevant methods that share similar goals (cognitive interviewing and measurement invariance tests) and as an assessment tool for single-item indicators in cross-national surveys. In particular, the dissertation addressed the following research questions: 1) Does online probing arrive at similar results than other methods? 2) Which are the strength and weaknesses of online probing in comparison to other methods? 3) How can online probing be combined with other methods in a mixed-methods approach? 3) How useful is online probing to assess the cross-national comparability of single-item indicators? Since the dissertation’s goal is to compare the methods of online probing, cognitive interviewing, and measurement invariance tests in regard to their potential to detect problematic issues at the item level, the field of national identity has been chosen as a substantive application for the method comparisons due to the existence of potentially problematic measures in a cross-national context. This dissertation focused on items from the 2013 International Social Survey Programme module on National Identity.

The first article of this dissertation (“Comparing Cognitive Interviewing and Online Probing: Do They Find Similar Results?”; published in Field Methods) analyzed whether online probing and cognitive interviewing arrive at similar conclusion with regard to error detection and themes that are mentioned by respondents when applied to the same set of items (ISSP item battery on specific national pride). The study compares data from cognitive interviews conducted with 20 German respondents in April 2013 with a web survey conducted with 532 German respondents in September 2013. The article revealed that both methods share complementary strength and weaknesses. While probing answers in cognitive interviewing show indications for a higher response quality, online probing can compensate through a larger sample size. The article also provides the researcher with guidance which method is preferable in a given research situation and advocates the combination of both methods in a mixed-methods approach.

The second article of this dissertation (“Necessary but Insufficient: Why Measurement Invariance Tests Need Online Probing as a Complementary Tool”; forthcoming in Public Opinion Quarterly, “2016 AAPOR/WAPOR Janet A. Harkness Award” and “2016 QDET2 Monroe Sirken Innovative Paper Award for Young Scholars of Question Evaluation”) provides an example for a mixed-method approach that combines online probing with quantitative measurement invariance tests. With the examples of the concepts of constructive patriotism and nationalisms, this study explains how the combination of both methods can reveal incomparable items and countries but also explain issues related to cross-national comparability. By analyzing data from the 2013 ISSP and a web survey with 2,685 respondents from five countries, online probing discovered the reasons for missing comparability (varying lexical scope and silent misunderstanding of a key term) that was also detected during the measurement invariance tests.

Finally, the third article showed the potential of online probing for the assessment of the cross-national comparability of single-item indicators with the example of the general national pride item. Online probing provides a unique solution for the decision whether single-item indicators are equivalent because the traditional approach of measurement invariance tests presupposes multiple-indicator measures and is, therefore, inapplicable for single-item indicators. This study analyzed 2,685 probe responses from a web survey that was conducted in five countries. Online probing uncovered several potentially problematic issues and the fact that respondents in all countries associate various concepts with the general national pride item.

Therefore, the contribution of this dissertation is:

1. The insight that online probing arrives at similar results than cognitive interviewing and measurement invariance tests.

2. A clear understanding of the method’s strength and weaknesses vis-à-vis cognitive interviewing and measurement invariance tests.

3. An explanation of optimal implementations of online probing in a mixed-methods approach.

4. A demonstration of the usefulness of online probing to assess the cross-national comparability of single-item indicators.

5. But also, an assessment of the cross-national comparability of measures of national identity for substantive researchers.