We’ve talked about the multiple mini-interview (MMI) quite a bit on in previous posts. Namely, about the number of ways in which the format outperforms traditional interviews. MMIs have become the gold-standard interviewing format for medical admissions, and their popularity has spread to other disciplines such as dentistry, pharmacy, and occupational therapy, as well as to employee selection in the corporate world.
The success of an MMI in accurately evaluating personal and professional characteristics of applicants is due not just to its format, but also to how it’s constructed.
A recent paper by Dr. Kevin Eva (one of the original founders of the MMI) and colleagues — published in the academic journal Medical Teacher — outlines 12 evidence- and experience-based tips for constructing an effective MMI.
In this post, we’re providing an actionable summary of those 12 tips. Whether you develop your own MMI or work with a company to do so, these tips can be a helpful resource in making sure you are getting the most reliable and valid information about your candidates.
#1: Remember — there is no such thing as the MMI
When people refer to the MMI, what they’re likely referring to is the MMI format of interviewing. For instance, we don’t say that students take the multiple-choice exam. Like an MMI, multiple-choice is a test format; the format is standard, but the content is not. The effectiveness of the test will vary.
That also applies to MMIs. Just because one school found that their MMI correlated with clerkship evaluations, doesn’t mean your school’s MMI will have the same correlation. Similarly, if MMIs at one school found large subgroup differences across demographic groups, it does not mean that your MMI will also adversely affect your minority applicants.
#2: MMI creation is a way to promote an integrated educational system
Admissions tools should align with the goals of the individual institution. The content of the MMI should reflect stakeholder values, and should communicate the program’s priorities to its candidates, as well as to faculty, current students, and community. Selection methods should be closely integrated with the program curriculum to ensure that both elements are maximizing the odds of graduating the “best” students with the highest potential for success.
#3: Assemble a diverse team
One of the primary advantages of the MMI is that it lessens the impact of biases and random errors that negatively impact the usefulness of traditional interview methods.
To truly diminish the presence of bias, it’s essential that the interviewers come from diverse backgrounds. This creates a healthy variety of perspectives, and reflects the different viewpoints of the stakeholders. Interviewers who come from just one subset of the population are more likely to make the same judgements about candidates, essentially negating the advantages of collecting multiple evaluations.
#4: Write stations purposefully
If you’re developing your own stations, it’s key to:
- Recruit a diverse team of station writers, who can draw upon their unique backgrounds and experiences to design stations that assess the targeted competencies.
- Give your station writers a blueprint to work from. This ensures that each station is tapping into one or more competencies that stakeholders feel is important for the program.
- Prioritize stations that don’t have obvious right or wrong answers. This keeps applicants from answering in what is obviously a socially desirable manner.
#5: Submit your stations to peer review
Once your station writers have submitted their stations, a panel of reviewers should closely examine each station to decide if the focus is clear, if the content could put particular candidates at a disadvantage, if adjustments are needed, and if any one station should be omitted entirely. This panel should also be diverse, in order to provide diverse perspectives.
#6: Consider the implicit messages your stations might communicate
It’s important to consider the possibility that certain stations might send an implicit message to candidates and stakeholders. For example, stations that prompt candidates to pretend to be a doctor will be biased toward candidates who have medical training or who come from a family with a background in a health discipline. That’s why it’s recommended that stations reflect roles that all candidates are likely to have experience with, such as the role of student or friend.
It is also important for stations to be face valid (transparent and understandably relevant) in order to avoid undermining the process. Asking a question like “if you were a color, what color would you be?” can cause candidates to take the MMI less seriously, negatively impacting the reputation of the program.
#7: Maximize the number of stations (within the limits of what’s feasible)
It’s recommended that an MMI include 10-12 stations. Reliability of the test increases with each additional station, but tends to plateau around eight to 10 stations. The additional two stations can be used for pilot testing, or as substitutes when another station has failed because of a technical glitch, or other reason.
Twelve stations would mean that the MMI would last about two hours — not feasible for programs with fewer resources. In these cases, remember that independent sampling is even more important than the careful construction of scenarios, so maximizing the number of independent samples (stations) should be a priority.
For example, if your program can only find three interviewers for 30 minutes, it’s better to have each interviewer assess the candidate separately, in separate stations, for 10 minutes, than to include all the interviewers in a single 30-minute interview.
#8: Don’t fret about the rating form
Reliability of the MMI can be increased by asking interviewers to judge candidates on a few criterion, such as communication skills, professionalism, and the quality of the discussion content.
But once programs decide which criteria raters should focus on, there’s no need to put further effort into specifying the exact details of what these dimensions look like. Small details about the rating form (such as whether to use a seven- or 10-point rating scale and what kind of adjectives to use) have been shown to make very little difference, as raters generally formulate an overall impression of the candidate.
#9: Tell your interviewers what you’re trying to accomplish
Because each interviewer will, subconsciously, bring their own biases and perspectives, it’s important that interviewers are provided with detailed instructions about what they are trying to accomplish. In a full MMI station, interviewers are given: the instructions provided to candidates; what the station is meant to assess; background information about the basic principles underlying the station; and a series of questions the interviewers might use to prime conversation.
Formal interviewer training isn’t essential, as teaching people about their biases doesn’t eliminate them. Additionally, formal training may also inadvertently force every interviewer to rate the exact same way, which would eliminate the value of diversity in perspectives that are so important to the MMI’s success.
#10: Give interviewers a chance to interact with many candidates
It’s hard to predict how candidates will perform in a particular station. That’s why it’s helpful for interviewers to interact with as many candidates as possible, so that they gain a better sense of the distribution of responses and can spread their ratings out accordingly.
During (minimal) training, encourage interviewers to use the full range of the scale to better differentiate students from one another. It’s also helpful for raters to use the first few responses to establish a benchmark. The first three to five responses should be treated as tentative and adjustable before submission, once interviewers get a better feel for the range of responses.
#11: Be transparent
The process of selection decisions in a high-stakes context should be transparent, so that candidates understand the process to be fair and that each candidate has an equal opportunity to succeed. Providing examples of MMI stations to candidates prior to interviewing can improve their perceptions (although scores don’t seem to improve with prior knowledge of station content). Similarly, it’s also helpful to provide general feedback about the candidate’s interview performance to help them decide whether they should re-apply if unsuccessful.
#12: Engage in continuous quality improvement
No matter how many resources and how much effort is dedicated to the development of an MMI, the test isn’t going to be perfect. MMI stations are meant to be an approximate reflection of how a candidate may act in real life, but that reflection will never be a perfect prediction, as we cannot account for all aspects of human behaviour.
Programs should engage in continuous quality improvement to help evolve their MMI, and to adapt to the changing needs of stakeholders. It is helpful to keep up with relevant research and literature, to examine how the process can be better refined. The data from each round of MMIs should also be used to inform the development of future MMI stations, to ensure that your MMI is improving with each successive year.
Published: February 16, 2018
By: Christopher Zou, Ph.D.
Education Researcher at Altus Assessments