Are Situational Judgement Tests Coachable?Situational judgement tests (SJTs) have recently gained popularity in educational and employment settings, as their high reliability, strong predictive validity, and relatively low cost of administration have made SJTs an attractive tool to assist in selection decisions. Their popularity has also given rise to a growing number of test preparation businesses that attempt to coach people on how to respond to SJTs more effectively. CASPer® is no exception. A handful of test preparation companies offer CASPer® coaching courses in an attempt to improve applicant scores. As we talked about in our previous blogpost, this can be a problem. If their coaching programs are actually successful in improving applicant scores, it would cause an unfair advantage towards those students who do not have access to these programs. As these CASPer® preparation classes typically cost money, these coaching effects can be particularly detrimental for low-income applicants who are already disadvantaged in the medical selection process. Due to these concerns, a lot of resources have been devoted to investigating the effects of coaching on most high-stakes admission tests, such as the Scholastic Aptitude Test (SAT), Graduate Record Examination (GRE), Graduate Management Admission Test (GMAT), Medical College Admissions Test (MCAT), and Law School Admission Test (LSAT). An overview of these tests have found that coaching effects on these standardized tests are fairly small (0.25 standard deviation). Additionally, test preparation and retaking does not seem to affect the predictive validity of standardized tests. When focusing on SJTs in particular, the general literature suggest that coaching effects remain fairly small, with an increase in only 0.5 standard deviation after coaching and an increase in 0.32 standard deviation for students who retook the test in the following admission cycle. However, not all SJTs are equally susceptible to coaching effects. When SJTs are more complex and difficult, coaching does not seem to improve subsequent performance. This suggests that tests are only coachable when test-taking strategies are simple and straight-forward, but are ineffective when the strategies are more challenging. A majority of SJTs use a closed-response format, where a series of pre-constructed answers are presented to respondents, but CASPer® uses a constructed-response format where respondents are required to come up with their own answers to the hypothetical scenarios. Compared to the closed-response format, the constructed-response format tends to be more complex and challenging, requiring a deeper level of processing by applicants, which potentially makes CASPer® a much harder test to coach compared to other SJTs. Therefore, it would be quite difficult to coach students to do better on CASPer®, especially within a short period of time. While there doesn’t seem to be very much students can do to improve their CASPer® scores, they can implement a number of strategies to calm their test anxiety and familiarize themselves with the test. We provide a number of sample questions on our website and offer a number of suggestions to ensure a smoother test-taking experience for students.
Published: August 18, 2017 By: Christopher Zou, Ph.D. Education Researcher at Altus Assessments