Web Survey Bibliography
Title Human vs. artificial intelligence: Are software solutions already able to replace human beings?
Author Koch, M.
Year 2016
Access date 29.04.2016
Full text PDF (537MB)
Abstract
Relevance & Research Question: Human-based coding of open-ended answers is a time-consuming and tedious procedure – especially in terms of large sample sizes. Therefore, in recent years, several software solutions have been developed in order to enable an automatic process of coding. The objective of this study is to investigate if these software solutions can already replace human-based coding or if they should only be used as supportive tools.
Methods & Data: In the first step, verbatims were coded by two different people in order to calculate a basis value for inter-rater reliability: therefore, the coders independently assigned the answers to 14 predefined categories. In the next step, three software solutions (one freeware and two commercial software) were used for coding the open-ended answers: All three software programs were applied to cluster the open-ended answers based on the semantics (no predefined categories), while one of the two commercial software was also able to automatically allocate the answers to the 14 predefined categories. For analysis, the different types of coding were compared with each other (human – human, human-machine and machine-machine).
Results: Human-human: The human-based coding achieved the highest matching rate (M=86%; average Cohen’s kappa к=0,8). Human-machine: The clustering of the open-ended answers by their semantics delivered different thematic codes. However, one code was evident with all three software solutions and the human-based coding. Yet, the level of average agreement was rather low (M= 25%; Kappa к=0,3). In contrast to that, the machine-based allocation to the predefined categories performed better: the average inter-rater reliability was к=0,4 with 38% percent of agreement. Machine-machine: In terms of the code that was evident with all three software, an average agreement of 71% could be achieved between the three different programs (average Kappa к=0,6).
Added Value: In consideration of the results, it seems that human-based coding is much more precise than machine-based coding. Yet, software solutions can facilitate the tedious procedure of coding: They can be applied before coding in order to get an overview of important keywords and thematic aspects within seconds. Besides, the categorization via predefined categories is also promising.
Methods & Data: In the first step, verbatims were coded by two different people in order to calculate a basis value for inter-rater reliability: therefore, the coders independently assigned the answers to 14 predefined categories. In the next step, three software solutions (one freeware and two commercial software) were used for coding the open-ended answers: All three software programs were applied to cluster the open-ended answers based on the semantics (no predefined categories), while one of the two commercial software was also able to automatically allocate the answers to the 14 predefined categories. For analysis, the different types of coding were compared with each other (human – human, human-machine and machine-machine).
Results: Human-human: The human-based coding achieved the highest matching rate (M=86%; average Cohen’s kappa к=0,8). Human-machine: The clustering of the open-ended answers by their semantics delivered different thematic codes. However, one code was evident with all three software solutions and the human-based coding. Yet, the level of average agreement was rather low (M= 25%; Kappa к=0,3). In contrast to that, the machine-based allocation to the predefined categories performed better: the average inter-rater reliability was к=0,4 with 38% percent of agreement. Machine-machine: In terms of the code that was evident with all three software, an average agreement of 71% could be achieved between the three different programs (average Kappa к=0,6).
Added Value: In consideration of the results, it seems that human-based coding is much more precise than machine-based coding. Yet, software solutions can facilitate the tedious procedure of coding: They can be applied before coding in order to get an overview of important keywords and thematic aspects within seconds. Besides, the categorization via predefined categories is also promising.
Access/Direct link Conference Homepage (presentation)
Year of publication2016
Bibliographic typeConferences, workshops, tutorials, presentations
Web survey bibliography - Germany (361)
- Does the Use of Mobile Devices (Tablets and Smartphones) Affect Survey Quality and Choice Behaviour...; 2015; Glenk, K.; Liebe, U.; Oehlmann, M.
- Does Personalized Feedback Increase Respondent Motivation?; 2015; Kroh, M.; Kuhne, S.
- Direction of Response Format in Web and Paper & Pencil Surveys; 2015
- Nonresponse and Measurement Bias in Web surveys ; 2015; Metzler, A.; Fuchs, M.
- Deep impact or no impact, evaluating opportunities for a new question type: Statement allocation on...; 2015; Schmidt, S.
- Approaches for Evaluating Online Survey Response Quality; 2015; Gluck, N.
- Positioning of Clarification Features in Open Frequency and Open Narrative Questions; 2015; Fuchs, M.; Metzler, A.
- A Systematic Generation of an Email Pool for Web Surveys; 2015; Silber, H.; Leibold, J.; Lischewski, J.; Schlosser, S.
- 640 Current trends in management of high-risk prostate cancer in Europe: Results of a web-based survey...; 2014; Briganti, A., Isbarn, H., Ost, P., Ploussard, G., Sooriakumaran, P., Van Den Bergh, R.C.N., Van Oort...
- Disclosure of sensitive behaviors across self-administered survey modes: a meta-analysis; 2014; Gnambs, T., Kaspar, K.
- Open-ended questions in Web Surveys-Using visual and adaptive questionnaire design to improve narrative...; 2014; Emde, M.
- Query on Data Collection for Social Surveys; 2014; Blanke, K., Luiten, A.
- Why Do Respondents Break Off Web Surveys and Does It Matter? Results From Four Follow-up Surveys; 2014; Rossmann, J., Blumenstiel, J. E., Steinbrecher, M.
- The Effectiveness of Mailed Invitations for Web Surveys and the Representativeness of Mixed-Mode versus...; 2014; Bandilla, W., Couper, M. P., Kaczmirek, L.
- Post-endodontic treatment of incisors and premolars among dental practitioners in Saarland: an interactive...; 2014; Mitov, G., Doerr, M., Nothdurft, F. P., Draenert, F., Pospiech, P. R.
- Mixed-Mode Designs bei Erhebungen mit sensitiven Fragen: Einfluss auf das Teilnahme- und Antwortverhalten...; 2014; Krug, G., Kriwy, P., Carstensen, J.
- Mining “Big Data” using Big Data Services ; 2014; Reips, U.-D., Matzat, U.
- Instant Interactive Feedback in Grid Questions: Reminding Web Survey; 2014; Kunz, T., Fuchs, M.
- What Does the Satisfaction with Democracy Measure Mean to Respondents in Different Countries? How Cross...; 2014; Behr, D., Braun, M.
- Determinants of the starting rate and the completion rate in online panel studies; 2014; Goeritz, A.
- Assessing representativeness of a probability-based online panel in Germany; 2014; Struminskaya, B., Kaczmirek, L., Schaurer, I., Bandilla, W.
- The Influence of the Answer Box Size on Item Nonresponse to Open-Ended Questions in a Web Survey; 2014; Zuell, C., Menold, N., Koerber, S.
- Does the Choice of Header Images influence Responses? Findings from a Web Survey on Students’...; 2014; Barth, A.
- Using Paradata to Predict and to Correct for Panel Attrition in a Web-based Panel Survey; 2014; Rossmann, J., Gummer, T.
- Offline Households in the German Internet Panel; 2014; Bossert, D., Holthausen, A., Krieger, U.
- Which fieldwork method for what target group? How to improve response rate and data quality; 2014; Wulfert, T., Woppmann, A.
- Switching the polarity of answer options within the questionnaire and using various numbering schemes...; 2014; Struminskaya, B., Schaurer, I., Bosnjak, M.
- Improving cheater detection in web-based randomized response using client-side paradata; 2014; Dombrowski, K., Becker, C.
- Interest Bias – An Extreme Form of Self-Selection?; 2014; Cape, P. J., Reichert, K.
- Increasing data quality in online surveys 4.1; 2014; Hoeckel, H.
- Moving answers with the GyroScale: Using the mobile device’s gyroscope for market research purposes...; 2014; Luetters, H., Kraus, M., Westphal, D.
- Confirmation Bias in Web-Based Search: A Randomized Online Study on the Effects of Expert Information...; 2014; Schweiger, S., Oeberst, A., Cress, U.
- Undisclosed Privacy: The Effect of Privacy Rights Design on Response Rates; 2014; Haer, R., Meidert, N.
- The Effect of Benefit Wording on Consent to Link Survey and Administrative Records in a Web Survey; 2014; Sakshaug, J. W., Kreuter, F.
- GESIS Panel: Sample and Recruitment; 2014
- The Use of Paradata to Predict Future Cooperation in a Panel Study; 2014; Funke, F., Goeritz, A.
- Incentives on demand in a probability-based online panel: redemption and the choice between pay-out...; 2014; Schaurer, I., Struminskaya, B., Kaczmirek, L.
- Responsive designed web surveys; 2014; Dreyer, M., Reich, M., Schwarzkopf, K.
- Extra incentives for extra efforts – impact of incentives for burdensome tasks within an incentivized...; 2014; Schreier, J. H., Biethahn, N., Drewes, F.
- Innovation for television research - online surveys via HbbTV. A new technology with fantastic opportunities...; 2014; Herche, J., Adler, M.
- Asking Sensitive Questions: An Evaluation of the Randomized Response Technique Versus Direct Questioning...; 2013; Wolter, F.; Preisendoerfer, P.
- Respondent Choice of Survey Mode; 2013; Fuchs, M.
- Development and validation of a single- item scale for the relative assessment of physical attractiveness...; 2013; Lutz, J.; Kemper, C. J.; Beierlein, C.; etc.
- Accounting for the Effects of Data Collection Method Application to the International Tobacco Control...; 2013; Thompson, M. E.; Huang, Y. C.; Boudreau, C.; Fong, G. T.; van den Putte, B.; Nagelhout, G. E.; Willemsen...
- The Short-term Campaign Panel of the German Longitudinal Election Study 2009. Design, Implementation...; 2013; Steinbrecher, M., Rossmann, J.
- Too Fast, Too Straight, Too Weird: Post Hoc Identification of Meaningless Data in Internet ; 2013; Leiner, D. J.
- The Digital Divide in Europe; 2013; Zillien, N.; Marr, M.
- The Recruitment of the Access Panel of German Official Statistics from a Large Survey in 2006: Empirical...; 2013; Amarov, B.; Rendtel, U.
- Online, face-to-face and telephone surveys—Comparing different sampling methods in wine consumer...; 2013; Szolnoki, G., Hoffmann, D.
- Where does the Fair Trade price premium go? Confronting consumers' request with reality; 2013; Langen, N., Adenaeuer, L.