The quality of governmental websites is often measured with questionnaires that ask users for their opinions on various aspects of the website. This article presents the Website Evaluation Questionnaire (WEQ), which was specifically designed for the evaluation of governmental websites. The multidimensional structure of the WEQ was tested in a controlled laboratory setting and in an online real-life setting. In two studies we analyzed the underlying factor structure, the stability and reliability of this structure, and the sensitivity of the WEQ to quality differences between websites. The WEQ proved to be a valid and reliable instrument with seven clearly distinct dimensions. In the online setting higher correlations were found between the seven dimensions than in the laboratory setting, and the WEQ was less sensitive to differences between websites. Two possible explanations for this result are the divergent activities of online users on the website and the less attentive way in which these users filled out the questionnaire. We advise to relate online survey evaluations more strongly to the actual behavior of website users, for example, by including server log data in the analysis.