With the vast majority of disciplinary action by medical boards being attributed to issues of professionalism, it’s critical that medical programs consider applicants’ non-academic qualities, like empathy and compassion, when making their selections. However, traditional tools like personal statements and reference letters are unreliable, and interviews are too time-consuming to be used for a large number of applicants. All of this contributes to the need for a revamped admissions process that uses more evidence-based screening tools to assess non-cognitive competencies and support existing efforts to increase diversity in medical programs.
In ongoing research projects with three MD programs in the United States, we’re looking to see how Casper can help evolve their admissions processes so they make decisions with confidence while saving time. These studies specifically seek to answer three questions:
- Does Casper measure the non-academic skills that are most important to the program?
- Are Casper scores associated with in-program performance?
- What demographic differences are seen in Casper test performance and how do they compare with other assessments?
Here is what the preliminary results show so far:
Casper’s impact on measuring key non-academic skills in the admissions process
In a joint study with a mid-sized medical program in the South-West, we evaluated the association and overlap between competencies assessed in the admissions process and Casper constructs and scores. We found that two key competencies the program measured – maturity and critical thinking – had a significant association with Casper scores. Based on these results, we’re building a process that will help the program use Casper earlier in the process to select the most promising applicants to invite to the MMI.
Interested in finding out who these programs are and getting the final results of the studies?
Subscribe to our newsletter to receive research updates and more content about how our programs are using Casper
Casper and in-program performance
One large program in the South-East US worked with us to conduct a quality assurance assessment focused on the link between Casper and clerkship grades. Initial results show that at a given GPA and MCAT score, applicants with the highest casper scores have a 5 percent higher probability of receiving the highest grade in clerkship. We are now working with the program to build a composite program-specific professionalism metric that incorporates clerkships, course grades, and administrative remarks. We’re also looking to create an admissions model to predict applicant performance in interviews based on screening measures like Casper.
We also worked with a mid-sized program in the same part of the US to evaluate the association between admissions metrics and course grades. The results showed that higher Casper scores were significantly associated with non-academic course grades, including clinical skills and doctoring, after accounting for MCAT scores and demographic characteristics (sex, age, and ethnic group). No significant associations were observed between Casper and more academically-oriented course grades, such as anatomy and biochemistry. Based on these initial results, the program will now work with Altus to add more cohorts of data and expand the study to build predictive models that identifies students who are likely to have issues, fail, or require multiple course attempts.
Demographic differences in Casper and other assessments
The large program in the South-East did not find any statistically significant differences in Casper scores between sex, age, or race group for current students. Meanwhile, in the MCAT, males averaged 5 percentile points higher than females, and black students averaged 13 to 14 points lower than all other groups. GPA for students older than 25 was about 0.17 lower than younger students, and black students had a GPA that was on average 0.29-0.33 lower than all other groups.
The mid-sized program in the South-West also saw major demographic differences in MCAT scores. Differences in GPA and Casper were generally small between demographic groups, with the exception of performance differences between genders and rural and urban applicants. To illustrate this effect size, a disadvantaged applicant would score an average of 505.1 on the MCAT, while others would score an average of 508.9. A disadvantaged applicant would have an average GPA of 3.57, while others would average 3.68. In Casper, a disadvantaged applicant would have an average z-score of 0.00, while others would have an average z-score of 0.11.
What does all of this tell us?
Although preliminary, the results show that Casper can reliably measure key non-academic skills that programs assess during admissions and in-program, making it a useful pre-screen tool to prioritize applicants for limited interview spots. All of this helps to save admissions committees valuable time in the admissions process. Based on the experiences of two of these programs, demographic differences in Casper test performance appear smaller than traditional cognitive assessments, although our own research has identified opportunities to further minimize those differences. By balancing academic and non-academic competencies in the admissions process, and using more effective and less-resource intensive screening tools to do so, programs can bring in more well-rounded students and support existing diversity-building efforts.