The Utility of Multiple Observations in Multiple Mini-Interviews and CASPer®
Since its creation in 2004, the multiple mini-interview (MMI) has become the gold standard of interviews for medical education admissions. The MMI offers excellent psychometric properties and strong predictive validity. But what makes it so much more powerful than a traditional interview? To answer that question, we need to look at the flaws of the traditional format.
Traditional interview formats are typically comprised of a single interviewer interviewing a candidate in a single setting. They offer one observation and that’s problematic, as it essentially leaves the candidate’s interview performance to chance. Despite our best efforts to remain unbiased when conducting interviews, research has shown, time and time again, that bias still influences our judgment of candidates.
Sometimes, in an effort to dilute the effects of a single interviewer’s bias, a panel interview will be used to gather multiple observations of the candidate. That can also be problematic, as the opinions of a group can easily be swayed by persuasive individuals or by those in positions of power. A candidate who gets randomly paired with an easygoing but persuasive interviewer, for example, will likely score higher than an identical candidate who gets paired with a more critical interviewer. In fact, researchers have found that candidate interview performance is commonly attributed to the interviewer and not the interviewee, suggesting that outcomes are more dependent on the person conducting the interview, rather than the quality of the candidate being evaluated.
Another issue with traditional interview formats is context specificity. A candidate’s interview performance can be highly contextual, excelling or faltering based on the interview’s focal topic, which might not even be generalizable to their future job or academic performance. For example, a student’s ability to work effectively in groups is unlikely to predict the student’s ability to handle stressful situations; restricting the interview to a single context may not provide much useful information about how the candidate will actually perform on the job.
The range of interview topics can be increased in an effort to assess different knowledge, skills, and abilities (KSAs) but the interviewer’s impression of each response is not necessarily independent of the other. If the interviewer gets a negative impression of a candidate based on their answers to the first few questions, it’s unlikely that the rest of the interview will change that opinion. Research shows that job interview success, for example, can be accurately predicted within the first few minutes.
Similar to the Objective Structured Clinical Examination (OSCE) used for the assessment of healthcare professionals in a clinical setting, the multiple mini-interview tackles these challenges by having multiple interviewers observe a candidate in multiple situations. In a typical MMI, candidates go through a series of 10 stations, each lasting 10 minutes, each evaluated by a different interviewer. The stations feature different scenarios designed to assess different competencies, such as ethical decision-making, communication skills, and critical thinking. For example, in one station from a sample MMI, students are asked to describe why they want to become a physician. In another station, students are asked how they would deal with a physician who is known to prescribe medicine with no scientific evidence.
Having multiple interviewers in multiple contexts generates a more accurate evaluation of a candidate’s “non-academic” abilities that are imperative to becoming a successful physician. Since the inception of MMIs in 2004, multiple research studies have demonstrated their superior reliability and predictive validity compared to the traditional interview formats.
CASPer® is an online Situational Judgement Test (SJT) that incorporates the advantages of MMIs by assigning different raters to each of its 12 sections. In addition, the sections span a variety of different situations and contexts, such as dealing with a difficult supervisor in school, working with an unmotivated colleague at work or supporting a friend struggling with financial problems.
By aggregating the evaluations from 12 separate raters, CASPer® minimizes the individual rater biases that may distort scores in superficial ways. It also reduces the impact of context specificity by looking at the candidate’s responses across 12 different situations.
With these primary sources of error significantly reduced, CASPer® tests obtain more reliable and more accurate assessments of candidates’ personal and professional competencies. CASPer® and MMIs both take advantage of multiple observations in multiple settings and are in fact moderately correlated with one another – demonstrating that there are both similarities and differences between the two (more on the differences later…). Both of these tools can provide medical education and other programs with unique insight into their applicants that are not being captured by the traditional assessment tools.
Published: November 7, 2017
By: Christopher Zou, Ph.D.
Education Researcher at Altus Assessments