Relevance and Research Question:

A common method used to efficiently obtain data in online studies is the Yes-No Grid. Elements in Yes-No Grids are endorsed at higher rates than when they occur in a Multiple Response Format (i.e., select all). Prior research (Smyth et al. 2005; Thomas & Klein, 2005) suggested that this may be due to increased consideration of each element. Asking for a response for each element to be considered will increase the access of less proximal memories (the salience hypothesis). Alternatively, Callegaro et al. (2015) proposed that acquiescence bias most likely explained heightened endorsement. Acquiescence bias results from socialization that encourages people to be agreeable leading to a greater tendency to endorse ‘agree’ with an agree-disagree response format or select ‘yes’ in a yes-no choice format. Our research question was a test of the viability of the alternative explanations.

Methods and Data:

We present two studies, with 1,127 and 1,449 respondents respectively, testing the divergent predictions of the salience and acquiescence bias hypotheses. Each study had a 2 X 5 factorial design. The difference between the studies was in the brands used for evaluation. Respondents were randomly assigned to a rating dimension (either descriptive or agreement ) and to one of five response formats. Each participant rated four brands with which they had some familiarity along six different attributes (e.g., ‘Is distinctive’, ‘Is expensive’, etc.). Those assigned the descriptive dimension were asked if the attributes described each brand, with response formats being either a multiple response format, dichotomous format (‘Describes, Does not describe’ or ‘Yes-No’), or trichotomous format. The agreement dimension had comparable response formats (e.g., ‘Agree, Do not agree’; ‘Yes-No’, etc.).

Results:

We found that trichotomous formats had the highest endorsements while the multiple response formats had the lowest endorsements regardless of responses used, supporting the increased consideration-salience hypothesis. In addition, the selection of ‘Describes’ or ‘Yes’ or ‘Agree’ were not significantly different, for either the dichotomous or trichotomous formats, disconfirming the acquiescence bias explanation.

Added Value:

Contrary to recommendations by Krosnick & Presser (2010) both yes-no and agree-disagree are efficient and valid response formats.