Cognitive interviewing is a well-established method for evaluating and improving a questionnaire prior to fielding. However, its present implementation brings with it some shortcomings: small sample sizes, considerable time and cost investment, and the possibility of interviewer effects. In this study, we test non-probability online access panels as a means to conduct cognitive interviewing, in particular category selection probing, in order to avoid the above mentioned shortcomings. The study reports on the results of 1,007 respondents from Germany. In order to identify implementation features that lead to a high number of meaningful answers, we explore the effects of (1) different panels, (2) different probing variants, and (3) different numbers of preceding probes on the answer quality. The overall results suggest that category selection probing can indeed be used in web surveys. Using data from two access panels – a community panel where members can actively get involved, e.g. by creating their own polls, and a ‗conventional‘ panel where answering surveys is the members‘ only activity – we find that high community involvement does not increase the likelihood to answer probes or produce longer statements. Testing three probing variants that differ in wording and provided context, we find that presenting the context of the probe (i.e., the probed item and the respondent‘s answer) produces a higher number of meaningful answers. Finally, the likelihood to answer a probe decreases with the number of preceding probes. However, the word count of those who eventually do answer the probes does not decline with an increasing number of preceding probes.