To measure what determines people’s attitudes, definitions or decisions, social science surveys increasingly use vignettes. A vignette typically describes a situation or object as having various attributes (dimensions). Respondents are asked to form a judgment of the hypothetical situation or object described by the vignette. The values (levels) of dimensions are experimentally varied, so that their impact on respondents’ judgments can be estimated. Previous studies on related methods (conjoint analyses and choice experiments) have shown that the order in which dimensions are presented in object descriptions can affect respondents’ judgments. Similarly, it is well known that the order of single item questions, or response options, can substantially affect responses. The underlying mechanisms causing order effects – in vignettes or single item questions – are however not well understood. Drawing on the literature in cognitive psychology and survey methodology, we examine two main research questions: Does the order in which dimensions are presented impact the vignette evaluations and change substantive conclusions? Under which conditions are order effects mostly likely to occur? Using data from a factorial web survey of 300 students we analyze several possible moderators: features of the vignette design (number of dimensions, complexity of the evaluation task, sequence of vignettes, dimension importance), characteristics of respondents (knowledge of the topic, strength of attitudes, cognitive ability), and interactions between these features. Our results show that strong order effects can occur, but only when the vignettes are of a minimum complexity.