What happened when Stanford University tested Casper for surgery residency selection

Selecting applicants for surgery residency is, in a word, challenging.

Take this, for example. During the 2015-2016 application-match cycle for surgery residency, 7,838 applicants submitted 17,813 total ranks for only 1,241 post-graduate year-one residency positions.

Filling those positions requires a substantial dedication of time and resources on behalf of the program director, and other department faculty and staff. And despite all their work, attrition rates for general residency programs have remained steady at around 20%, indicating that there’s room for improvement in the selection process.

So why the drop-out after successfully obtaining a coveted spot? It could have to do with how residents are selected.

Residency selection has traditionally relied on the contents of the Electronic Residency Application Service (ERAS) application, as well as the personal interview. But these sources don’t provide enough information about the applicant’s personal qualities — such as resiliency, empathy, and communication skills — which, alongside cognitive qualities, help maximize the chances of an applicant completing the residency program.

Unfortunately, the tools often used to assess these qualities, such as personal statements, letters of recommendation, and the Medical Student Performance Evaluation (MSPE)/Dean’s Letter, have been criticized for being highly subjective. Amongst admission officers and researchers, there are a number of concerns about the tools’ reliability and validity.

That’s why a group of researchers at Stanford University in the Department of Surgery decided to try Casper during their surgery resident selection process, and evaluate its feasibility and validity in assessing applicants’ personal characteristics. Their experience was outlined in a recently published paper in the Journal of Surgical Research. Here’s a summary of their findings.

How the evaluation worked

On the day of their in-person interview for general surgery residency, applicants were invited to complete a voluntary and anonymous survey. The survey was designed to assess their attitudes about using the traditional method of having a faculty panel evaluate their ERAS application and interview, versus using the Casper test method of assessing applicant personal characteristics.

Faculty members were also invited to participate in a survey about their perceptions of Casper. Although they did not see or take Casper, they were given general information about the structure of the test.

Both the surveys for the applicants and the faculty members asked for perceptions of how strongly personal characteristics are currently emphasized in the screening process, how strongly personal characteristics should be emphasized, and whether the requirement to complete Casper during the residency application would affect the likelihood of an applicant applying to that residency program.

Following the survey, applicants were invited to take Casper after the interview was completed. Those who took Casper were asked to complete a post-test survey.

The survey results

  • Applicants felt that the program’s current residency selection process did not emphasize personal characteristics enough
  • Applicants thought that the traditional method (faculty panel evaluation of ERAS and the personal interview) was slightly more accurate but less objective than Casper
  • Faculty members thought Casper was more accurate than the traditional method, but no more or less objective
  • 85% of applicants reported that a requirement to complete Casper would have no bearing, or would make them more likely to apply to the program
  • 62.5% of faculty members reported that the requirement to complete Casper would have no bearing, or would make applicants more likely to apply to the program

Casper results

Thirty percent of applicants (23 out of 77 applicants) voluntarily completed Casper, and 16 applicants completed the post-survey, so the sample size was quite small. Preliminary results suggested that applicant perceptions of Casper became less positive after taking the test — just nine of the 16 applicants who completed a post-Casper survey (56%) reported that completing Casper would have no bearing on submitting a residency application — but it’s difficult to say whether these results are generalizable due to the small sample size.

Based on the results from the 23 applicants who did complete Casper, the authors found preliminary evidence that Casper does equally as good of a job in predicting applicant rank list position as the traditional faculty assessment. This suggests both methods are viable tools for residency selection.

A major advantage of Casper is that it allows residency programs to substantially reduce costs by outsourcing personal characteristics assessment and utilize their resources more effectively in other aspects of the selection process. “More importantly,” the researchers state, “adding an additional requirement like the Casper test to the general surgery residency application does not appear to deter a large percentage of applicants from applying to a program.”

The researchers conclude that “With the information currently known, the Casper test is best understood as an adjunct to traditional methods of assessing personal characteristics, and a potential alternative to improve the chances of selecting residents who will succeed in residency programs.”

Photo by Javier Matheu on Unsplash

Share via
Copy link
Powered by Social Snap