Despite important advances in mobile advertising research, most surveys have used self-reported questionnaires without ensuring that the survey respondents regularly receive and access mobile ads, and are therefore able to evaluate this new medium. This paper fills this research gap, by proposing a mobile-based survey method and conducting a pilot study. It focuses primarily on critical issues associated with online survey methodology, such as cost, target respondents, questionnaire format, response rate and data quality. A pilot study of mobile advertising is then designed and executed, in collaboration with a Japanese interactive advertising agency. 'Pseudo' mobile campaigns were created using a real brand (handheld audio player) as a real stimulus. One week after transmitting the mobile advertising, a structured questionnaire was sent directly via mobile device. A confirmatory factor analysis reveals that the multichotomous response formats applied in the questionnaire produced an acceptable scale reliability and validity. The methodological implications of the findings are discussed. In closing, limitations are recognised, while future research directions are suggested.