Do Canadian Medical Licensing Exam Scores Correlate with Physicians’ Future Performance in Practice? A Response to the MCC by ITPO

Download the Response

January, 2025

Background/Context

As an advocacy organisation for internationally trained physicians (ITPs), ITPO has never advocated for the removal of Canadian medical licensing examinations. As an organisation led by and serving doctors, the appreciation of the importance of medical standards is instinctual. In our mission to support the comprehensive inclusion of ITPs as practicing physicians in Canada, to serve Canadian patients, evidence, sensibility, fairness and collaboration have been our strongholds. What we have cautioned against, is reducing the competence of a doctor to an exam score; and far worse, the extent to which that exam score surpasses what has been set as standard and thus acceptable. Elements of this argument can be found in our previous research and policy briefs. 

After reading the newly released research paper (Bartman et al. 2024) which has been promoted by the Medical Council of Canada, ITPO must re-emphasise its position. Caution must be exercised when using an exam score and the extent that it surpasses an established standard to speak to the performance of a physician in practice. 

Commentary on the Paper

As with all frequentist statistics that rely on the general linear model, the results of a logistic regression are only as good as the extent to which the underlying theory may be considered true. While we acknowledge the effort to narrow the type of complaints used as compared to previous research, the theoretical link remains tenuous. A large majority of ITPs have written the MCCQE1 and with over 2,700 members, ITPO has a diverse population of ITPs that have written varying versions of the MCCQE1. This exam can draw on the knowledge from a range as wide as medical ethics to medical biochemistry. Physician complaints can also span a wide range of concerns from inadequate care to sexual harassment. Therefore, a correlation between high exam scores and non-dismissed complaints is hardly interpretable without more detailed data. Did those who had high scores on medical ethics questions have complaints filed against them for sexual harassment? Did those who had high scores on neurology then turn out to have complaints filed against them for inadequate care? If so, this would be completely contradictory to the current conclusions being drawn.

Additionally, non-dismissed complaints were also said to include “acknowledgements of receipt” of complaints. The logic that followed removing complaints undergoing investigation should also have been applied in this case. Complaints can be filed for many reasons, including malicious ones. What should be sought, to enable interpretable results, is complaints that were ruled on and ended in remediation for the physician.

Given the issues with the underlying theoretical basis, it is difficult to see how these results can inform how exams scores are used in the future.

Why it Matters

At a time when a multitude of health system stakeholders are striving to integrate ITPs into the system as doctors as a part of the solution to help with the health human resource crisis, it is important that the value of ITPs be accurately represented. The bias against international training is so pervasive that despite ITPs passing Canadian standardised exams, they are still viewed as incompetent. A review of the exclusion of ITPs (Grez et al. 2023) from researchers of Simon Fraser University can be found on British Columbia’s Office of the Human Rights Commissioner. The vast clinical experience that they come with is overlooked and made to seem lesser than because they practiced internationally. The invaluable knowledge that ITPs have of diverse diseases that are not currently endemic in Canada but can rise due to immigration is ignored. The multilingualism and cultural and ethnic sensitivities that can improve the patient experience is discounted. 

Stimulating the thought that a 100 point increase on the MCCQE1 can be used as a basis for determining future performance is the sort of action that leads to the current reality of residency programs automatically filtering out candidates with less than a selected-behind-closed-doors cut off score. Hundreds of excellent candidates who have sat and passed the MCCQE1 and thus have met the Canadian standard never get their file reviewed because of the perpetuation of this rigid metric thinking. Thinking that robs Canadian patients of skilled, safe and experienced doctors who come with so many assets but are reduced to the extent to which the set standard is exceeded. 

Conclusion

ITPO:

1. supports upholding medical standards and acknowledges that examinations are a part of that.

2. cautions against the use of the research paper in question to draw conclusions or make definitive statements. 

3. does not support the use of exam scores to exclude rather than include.