Aims and objectives  This study aimed to determine the discriminant validity and the test–retest reliability of a questionnaire testing the impact of evidence-based medicine (EBM) training on doctors' knowledge and skills.

Methods  Questionnaires were sent electronically to all doctors working as residents and chief residents in two French speaking hospital networks in Switzerland. Participants completed the questionnaire twice, within a 4-week interval. The discriminant validity was examined in comparing doctors' performance according to their reported EBM previous training. Proportion of agreement between both sessions of the questionnaire, Cohen's kappa and ‘uniform kappa’ determined its test–retest reliability.

Results  The participation rate was 9.8%/7.1% to first/second session. Performance increased according to the level of doctors' previous training in EBM. The observed proportion of agreement between both sessions was over 70% for 14/19 questions, and the ‘uniform kappa’ was superior to 0.60 for 15/19 questions.

Conclusion  The discriminant validity and test–retest reliability of the questionnaire were satisfying. The low participation rate did not prevent the study from achieving its aims.