How Predictive Are Med School Exams?

It's been evaluation week on my clerkship block (don't worry, passed everything), and it brought up an interesting discussion - just how predictive are tests in evaluating students' likelihood to do well in practice? My block evaluations were quite fair, I certainly can't complain about them, but they did contain some odd elements that probably aren't too meaningful to future practice.

When we talk about testing in medicine, the sensitivity and specificity come up a lot. In medical education, those concepts could easily be applied - that is, how likely is it that a good future practitioner passes a given evaluation, and how likely is it that a bad future practitioner fails it? Outside of Quebec, Canadian medical schools use a pass/fail system with pass levels that are not particularly high. It's not too hard to pass the formal evaluations. And, in the event of an "off" day that results in a single failed evaluation, a repeat is typically granted without punishment provided it's not a regular occurrence (and you pass the repeat of course).

These tests are generally designed to identify unacceptably poor students and while individual tests might mistakenly call a good student inadequate, on the whole they're probably letting through students with trouble more so than they're failing otherwise good students. They have a high specificity, in other words - a true fail likely means the student has true issues to be addressed. They're probably not that sensitive though, so a pass doesn't mean much.

Predictive values of major tests have been studied in the past, but I've yet to see an analysis along the sensitivity/specificity lines. Mostly previous literature on the subject has demonstrated correlations, with the strongest correlations usually occurring between similar assessment tools (eg GPA correlates reasonably well with pre-clerkship grades, while MCAT correlates with the USMLE Step 1, but the correlation is weaker for other medical school metrics).

Anyway, not super useful right now, just another time to note that while we're busy developing evidence-based medicine, we're pretty far behind when it comes to evidenced-based medical education.