The increase in internet penetration coupled with smartphone and tablet use makes the web surveys an attractive mode; however, there exist questionnaire design issues, as well as device-specific visual design concerns, that are poorly understood. Historically, researchers have assessed measurement error caused by survey mode effects through experimental field tests where several versions of a question are randomly assigned to a subsample of respondents with different modes. As more web and multimode surveys have been used in the general population, ways of uncovering potential areas of measurement error have been developed. One such technique is usability testing. Usability testing focuses on the human and computer interaction. Thus, it is not simply about the respondent’s personal opinion or thoughts about the web survey or any given item, but instead focuses on objective results including, for instance, the time taken to answer a question, whether respondents change their answer, and whether respondents make mistakes in navigation or usage of the web survey. As Couper (2011, p. 385) explains, usability testing, or user-centered design “is more than user friendliness.” In this paper, we assess the findings from a study where a complex telephone survey was redesigned for web and mail. We conducted 90 usability tests as well as fielding a large mode effects experiment using separate samples for web, mail, and telephone. Our research sheds light on the impact of visual complexity of rostering questions, matrices, and utilization of question grouping in web surveys, as well as the comparability of different visual groupings among multiple devices with different screen sizes, while also highlighting what issues are uncovered by usability tests, field tests, and both.