By: Patrick Antonacci, M.A.Sc., Data Scientist
The academic admissions process is generally broken down into two categories: 1) academic performance, and 2) personal and professional characteristics.
Academic performance for medical school admissions is measured through GPA and MCAT, which have both been proven to be valid assessment tools. But the process for assessing personal and professional characteristics is not as robust. These qualities are typically measured by reference letters, personal statements and, the most widely used method, interviews.
Nearly every North American medical school requires applicants to participate in an interview as part of the admissions process. These interviews are intended to assess characteristics like work ethic, ambition, creativity, empathy and kindness, along with motivation and problem-solving skills.
Most schools consider the interview an important portion of their admissions process, hence a considerable amount of faculty time is dedicated to both arranging and conducting the meetings. But the time cost doesn’t necessarily mean that the interviews are effective. To be effective, the interview results must demonstrate both high reliability and validity. As it turns out, they rarely do.
Back in 1981, medical schools hadn’t established a way to evaluate the effectiveness of their own interviews. Instead, they relied on academic literature to demonstrate their reliability and validity — literature that didn’t offer conclusive evidence. A study by Puryear and colleagues from that same year couldn’t even describe the factors that interviews specifically measure.
Results from another study in 1991 added to the evidence that interviews have questionable reliability and validity. The authors advised that unless schools were willing to spend a substantial amount of time and resources to train interviewers and construct highly structured interview formats, they should consider dropping interviews as an admission tool all together. And, they warned, even if schools did decide to take this course of action, they could not be assured that the interviews would make a difference in the admissions process.
In these studies, reliability of interviews has varied considerably (though more structured interviews demonstrate more reliable judgments). There tends to be low agreement between interviewers in their evaluation of candidates. Additionally, several studies have shown that applicants are not consistent in their performance across multiple interview sessions. If interviews for medical schools aren’t reliable, then they’re not valid, and their utility in assisting the medical admission process is being greatly compromised.
Reliability is the ability for the measurements of a tool to be reproducible; for interviews, this means that an applicant’s scores should remain similar if the interview is repeated. If an interview is repeated however, there are many extraneous factors that could affect whether an applicant’s scores are similar to those from their first interview. These factors include having a different interviewer, using different interview questions in each interview, whether the candidate is being interviewed earlier or later in the lineup of candidates, if they’re interviewed before or after a high- or low-performing candidate, and so on.
Kreiter and colleagues found that despite the carefully developed structure of the interviews used in their study, only low to moderate reliability was found for interviews, and they lack the psychometric precision necessary to be used as a highly influential admission tool in the academic admissions process.
Over time, using well-trained interviewers and more structured interviews has increased the reliability of interviews, but its increase has not been sufficient enough to increase their validity. In fact, interviews have consistently failed tests of validity, and they’ve failed to predict performance on objective structured clinical examinations (OSCE) for medical schools and licensure exams. One notable exception is the structured Multiple Mini-Interview (MMI), which has been shown to predict performance on Part I and Part II of the Medical Council of Canadian Qualifying Examination (MCCQE), but because it requires a circuit of several short assessments with multiple interviewers, it’s better suited for smaller applicant sizes.
By comparing the medical school admissions process of 1986 with that of 2008, Monroe and colleagues found that interviews had become the most important factor in deciding who to accept. Yet, they’re the costliest and most time-consuming portion of the admissions process, and they offer low to moderate reliability, and even lower validity. And because scores from interviews mostly reflect biases and variations in the interviewer and the context of the interview, their use as a predominant element of the admissions process calls into question the fairness of decisions made by schools.
Casper supplements and can potentially replace many traditional and outdated admissions tools, including the interview. Rigorously tested, administered online to an entire applicant pool, and focused on measuring personal and professional characteristics, it’s modern and, most importantly, it’s reliable and valid.