By: Patrick Antonacci, M.A.Sc., Data Scientist
In the world of academic admission tools, the reference letter is an old favourite. For the better part of a century, college and university applicants have been asked to supply a letter written by a mentor, boss or teacher that testifies to the applicant’s intelligence, work ethic, kindness, ambition and creativity. It’s a nice tradition, but is it an effective one?
Academic admission assessment tools change infrequently, for two reasons: newer tools are rarely developed; and there’s institutional inertia that exists within admissions departments. Because of the lack of innovation, there are often few opportunities to examine current legacy assessment tools, like reference letters, and ask: is this a tool that we should continue using?
The literature suggests no. Research indicates that when it comes to letters of recommendation, the interrater reliability — the degree of agreement amongst those doing the rating or assessing — is only about 0.40. That’s below the recommended threshold of 0.7-0.9 for high-stakes testing. A low reliability rating means that two different people looking at the same reference letter are unlikely to have the same rating of the applicant — not exactly helpful when it comes to academic admissions.
Low reliability tools like reference letters provide almost no truly useful information about the applicant. In fact, one study found that there’s more agreement between two letters of recommendation written for two different applicants by the same referee than there is for two different referees penning recommendations for one applicant. This suggests that the review process has more to do with the referee than the applicant. As one researcher summarized: “Put another way, if letters were a new psychological test they would not come close to meeting minimum professional criteria (i.e., Standards) for use in decision making (AERA, APA, & NCME, 1999).”
Despite this evidence, letters of recommendation remain popular in the academic admissions world because there’s a desire and need to evaluate more than just academic performance. Personal and professional characteristics, like communication, ethics and empathy, are, in many ways, just as important as cognitive measures that predict for the academic performance of an applicant. Those are the characteristics that Casper reliably identifies, through rigorously tested and refined methods — not a nice letter from an old teacher.