Skip to main content
In Search of Black Swans: Identifying Students at Risk of Failing Licensing Examinations
Academic Medicine
  • Cassandra Barber, Western University
  • Robert Hammond, Western University
  • Lorne Gula, Western University
  • Gary Tithecott, Western University
  • Saad Chahine, Western University
Document Type
Publication Date
URL with Digital Object Identifier


To determine which admissions variables and curricular outcomes are predictive of being at risk of failing the Medical Council of Canada Qualifying Examination Part 1 (MCCQE1), how quickly student risk of failure can be predicted, and to what extent predictive modeling is possible and accurate in estimating future student risk.


Data from five graduating cohorts (2011-2015), Schulich School of Medicine & Dentistry, Western University, were collected and analyzed using hierarchical generalized linear models (HGLMs). Area under the receiver operating characteristic curve (AUC) was used to evaluate the accuracy of predictive models and determine whether they could be used to predict future risk, using the 2016 graduating cohort. Four predictive models were developed to predict student risk of failure at admissions, year 1, year 2, and pre-MCCQE1.


The HGLM analyses identified gender, MCAT verbal reasoning score, two preclerkship course mean grades, and the year 4 summative objective structured clinical examination score as significant predictors of student risk. The predictive accuracy of the models varied. The pre-MCCQE1 model was the most accurate at predicting a student's risk of failing (AUC 0.66-0.93), while the admissions model was not predictive (AUC 0.25-0.47).


Key variables predictive of students at risk were found. The predictive models developed suggest, while it is not possible to identify student risk at admission, we can begin to identify and monitor students within the first year. Using such models, programs may be able to identify and monitor students at risk quantitatively and develop tailored intervention strategies.


Article available at Academic Medicine, Vol. 93(3).

© 2018 by the Association of American Medical Colleges
Citation Information
Cassandra Barber, Robert Hammond, Lorne Gula, Gary Tithecott, et al.. "In Search of Black Swans: Identifying Students at Risk of Failing Licensing Examinations" Academic Medicine Vol. 93 Iss. 3 (2018) p. 478 - 485
Available at: