By: Christopher Zou, Ph.D., Education Researcher
Last year at Altus Assessments, we focused our research efforts on addressing the two questions we hear most often from our academic partners:
- What does Casper measure?
- How should we incorporate Casper into our existing medical admissions process?
In this blog post, we’re using our research from the 2017-2018 admissions cycle to answer both questions.
What does Casper measure?
The answer to this question is still up for debate, as there isn’t a clear consensus in the research community on what it is that SJTs actually measure.
Some researchers argue that SJTs are a method of assessment that allow for the measurement of a number of different constructs, such as general mental ability and personality variables. Other researchers argue that SJTs are a measure of multiple dimensions that tap certain competencies such as leadership and teamwork skills. Yet another group of researchers view SJTs as an assessment of candidates’ abilities to effectively navigate specific situations.
To tackle this question in the context of medical admissions, we partnered with New York Medical College. We collected 2015-2016 admissions data, which included scores on both the old and the new 2015 MCAT subsections, undergraduate GPA, Casper scores, and performance on the multiple mini-interview (MMI). We then conducted a factor analysis to examine how these various scores group together.
The factor analysis revealed two general factors: cognitive skills and professionalism. While GPA and the science subsections of the MCAT clustered to form the cognitive domain, Casper and MMI scores clustered to form the professionalism domain. These results suggest that Casper measures the non-cognitive competencies of candidates that are distinct from GPA and MCAT, but more closely related to the MMI.
How should Casper be incorporated into the existing admissions processes (i.e. medicine)?
Programs gather lots of information about their applicants, including but not limited to: their undergraduate GPA, the types of courses they took, volunteer hours, reference letters, MCAT scores, personal statements, and research experience. So how does Casper fit in the sea of various admission metrics?
This year, we conducted a simulation study with a large dataset of over 9,000 applicants to examine the different ways in which Casper can be incorporated into the existing admissions process, and how it would impact the quality and diversity of the incoming medical students.
Our results revealed that the more heavily Casper was weighted in the applicant rankings, the greater the demographic diversity, with only a very slight decrease in the academic metrics (MCAT and uGPA). A heavier weighting of Casper also led to better performance on the MMI, which research has shown is more predictive of clinical performance than the cognitive metrics.
How Casper can be incorporated should depend on the institutional goals:
- If the goal is to increase diversity, Casper should be more heavily weighted
- If the goal is to improve the academic standings, then perhaps Casper should be less heavily weighted
- If the goal is to select applicants with the highest potential for success as a physician, then a heavier emphasis on Casper might be warranted
While the simulation study is not able to definitively say what the “best” method is for each program, our study reveals that Casper gives programs the flexibility to adjust their admissions criteria to be better tailored to their individual needs.
Want to collaborate with Altus?
This year, Altus Assessments has collaborated with New York Medical College, Université de Montréal, McMaster University, and the Council of Ontario Nursing Group to conduct a variety of research projects for internal review, submission to conferences, and journal publications. Several programs have also expressed interest in conducting research on CASPer and are in the early stages of building a research project.