Student Doctor Network

2022 Situational Judgment Test (SJT) Experience Survey – Part II

Situational Judgment Test Experiences Survey

In Part 1, I introduced you to some of the results from the 2022 SJT Experience Survey, sponsored by the Health Professional Student Association. This anonymous survey captured input from 86 respondents who took a situational judgment test (SJT) and were willing to reflect on their experience. For this article, we focus on their self-disclosed results to see if there are any inferences on how the SJTs have been used with admissions decisions in the past.

How well did the examinees do?

Respondents self-disclosed their exam results for Acuity Insights (formerly Altus) Casper and AAMC PREview. The breakdown of results from the SJT Experience survey is summarized in Table 2-1.

Table 2-1: Self-disclosed SJT results

Altus (Acuity Insights) Casper
 Quartile 1Quartile 2Quartile 3Quartile 4Quartile Average
All (n=68)141518212.6
Postbac non-student (n=25)74952.5
Postbac/ Graduate student (n=23)36592.9
Professional students (n=12)04443.0
Professional MD/DO (n=9)04322.8
Professional not MD/DO (n=3)00123.7
URM (n=13)34422.4
AAMC PREview
 Quartile 1Quartile 2Quartile 3Quartile 4Quartile Average
All (n=29)357143.1
Postbac non-student (n=13)32352.8
Postbac/ Graduate student (n=12)11373.3
Professional students MD (n=4)02112.8
URM (n=5)11302.4

68 respondents disclosed their Casper quartile. The distribution of Casper scores was fairly even across all quartiles (the average quartile is 2.6). While postbac non-student applicants also showed an even distribution of results (average quartile 2.5), postbac/graduate student applicants showed a slight skew towards higher quartiles (2.9). Among the professional students who disclosed their Altus Casper results, no professional students reported a first-quartile result. Interestingly non-medical professional students only reported scores in the upper two quartiles. A larger number of results from current professional students would be needed to see if this observation is just coincidental, especially as Acuity Insights does not recommend that schools use Casper results as a cutoff.

20 respondents disclosed AAMC PREview quartile results. Unlike the Acuity Insights Casper results, we observed a skew towards the higher quartiles (average quartile 3.1). Although postbac non-students reported a slight skew (2.8), postbac/graduate students showed a higher likelihood for 4th-quartile scores (3.3). Results from four professional students also showed a similar skew (2.8). These results contrast with AAMC pilot data that show a normal performance curve among test-takers, consistent with well-designed psychometric exams.

How well do Casper and PREview assess your competencies?

27 respondents disclosed both Casper and PREview results in the survey (Table 2-2). While Casper scores were evenly distributed (2.4), PREview scores were skewed (3.1). These data further confirm that most MD/DO applicants and matriculants should be able to perform well on PREview. In contrast, applicants with low Casper scores may be relegated for additional screening or placed on hold.

  AAMC PREview (Average Quartile 3.1)
Altus Casper (Average Quartile 2.4)
  Q1 Q2 Q3 Q4 Total
Q1 0 0 5 2 7
Q2 0 2 1 4 7
Q3 2 2 0 3 7
Q4 0 1 1 4 6
Total 2 5 7 13 27

How do underrepresented minority applicants perform?

Most SJTs show attenuated differences in scoring due to racial differences, so we were interested in any differences in Altus Casper and AAMC PREview performance among underrepresented minority respondents (Black/African/Caribbean and Latinx). While 13 URM applicants’ Casper results showed an even distribution comparable to the entire cohort (average quartile 2.4), five URM applicants’ PREview results showed a skew towards lower quartiles (average quartile 2.4) compared to the overall cohort (3.1). Four took both Casper and PREview: two had Q1 Casper results but Q3 PREview results; one had a Q3 Casper and a Q2 PREview, and one had a Q4 Casper and a Q3 PREview. We hope a subsequent survey can provide more insights.

What are some takeaways?

While a larger survey that encompasses multiple health professions (including residency selection) and more diverse applicants on the SJT experience needs to be developed, there are a few insights the SJT Experience survey provided about the strategies applicants have taken to prepare for these SJTs and possible connections with admissions decisions.

How can you prepare for the “unpreparable”?

At the time of this article’s publication, programs are evaluating the impact of Casper and PREview exams on the admission of future classes of students. More programs are expected to use these exams on a “research” basis to gain confidence that fewer students with academic or professional issues are admitted while still meeting goals for a diverse and inclusive learning community. We have compiled some tips to help you strengthen your situational awareness and knowledge of professional expectations of conduct.

  1. Through your life experiences, develop your empathy and advocacy with social justice issues affecting healthcare.

    As one of our posters, acstylin, points out: “The best way to prepare is to have some life experiences with people that disagree with you, debate a heated/sensitive topic and defend the side you don’t agree with, immerse yourself in cultures that make you uncomfortable, etc.” As more programs seek applicants with strong advocacy/leadership skills and a track record as a change agent toward better health equity or social justice, gaining a better understanding of ethical and empathetic behavior in the context of diversity and belonging will help all aspiring professionals. Students should commit themselves to such activities to hone one’s awareness of bigger-picture issues and external influences on social determinants of health.

    Lior Krimus of PrepMatch agrees: “One thing I always recommend is to write down a list of scenarios in your life involving the following: conflicts (professional/personal), scenarios reflecting your strengths, scenarios reflecting your weaknesses, list of failures, times you showed good communication, times you showed good collaboration, and times your collaboration didn’t go as planned. Having these scenarios written beforehand really helps with Casper and the MMI.”

    For example, as more restrictions to provide healthcare access to LGBTQ+ community members have been implemented, applicants should pursue opportunities to become stronger allies. HPSA has developed a free Google Classroom curated resource on Treating Transgender Patients that includes over nine hours of audiovisual materials from transgender patients and health professionals on making the healthcare system more inclusive, as well as quality journalistic articles that address issues of concern.

  2. Reflect on how you approach – or avoid – conflicts.

    By understanding your own tendencies to react to situations, you can better understand how to evaluate effective or ineffective outcomes to your responses. This insight will help you as you further hone your preparation for the question formats of each SJT or MMI question you could face. HPSA has developed a situational awareness workshop that debuted at the 2022 University of California Davis Prehealth Conference, and more is available in the HPSA Becoming a Student Doctor course.

  3. Ask schools about how they factor SJT and online MMI results into admissions decisions.

    Find out if the schools you are considering will require an SJT or rely on an online platform such as Kira Talent or SortSmart to interview candidates. By the spring, programs should finalize their participation with Casper, PREview, or another system for the upcoming cycle. Frequent contact with admissions counselors during spring recruitment will help determine whether you need to prepare for these exams during the summer or fall.

  4. Schedule extra time to prepare for different SJT formats.

    Most test-takers will take a few hours to get oriented to the exam format, review practice questions, and secure appropriate technology and support to schedule and take the exam. Seek out information about processes to request accommodations or fee assistance before scheduling your exam. Note that most respondents did not go out of their way to take the exams and most likely prepared for no more than a month before the exam date. 

    Scheduling the exam will depend on the applicant’s school list and any other test preparation. Many only think about taking this exam after they receive results from high-stakes exams like the MCAT, DAT, or GRE. For many premedical students, while Casper dates may be available as early as April, most will likely take the exam between July and September as they discover a school of high interest requires Casper or PREview. Hundreds of students passively withdraw from applying to schools requiring these tests by not taking these exams, especially after they receive sub-optimal MCAT/DAT score results. The only advantage to having Casper scores in hand (taking the exam before July) is to have a complete file for early screening and consideration for an interview. Still, one must also submit all supplemental information (secondary application, high-stakes exam result, possibly letters of recommendation) early to take full advantage. There is little information on whether taking any of these exams near the end of the testing window affects your chances for an interview or admission.

    For those pursuing residency, you should prepare to take the Altus Suite if you are interested in anesthesiology, internal medicine, OB/GYN, ophthalmology, and surgery (more may be added after this article has been published, so check the Take Altus website). One should follow up with specific programs requiring you to take the Acuity Insights assessments and submit your responses concurrent with your ERAS application so that all your materials are received by the program directors promptly. 

  5. Use free practice resources.

    PrepMatch, a student/trainee-run organization that provides peer-assisted Casper preparation, has provided free resources for the last few years. Both written and video scenarios are available through the PrepMatch website, and some sample written scenarios can be accessed on the SDN forums. We also have an extensive article archive on Casper with strategic tips from other influencers or admissions consultants.

  6. Stop overthinking and be yourself.

    While you can view dozens of influencer videos on how to “crush” these SJTs, remember that the best answer is often the answer you come up with. Aside from suggestions to improve your typing speed on Casper or to practice responding to behavioral/situational questions in mock interviews, the best performance you can have is the one that reflects your authenticity when handling the conflict. Based on what the survey suggests, PREview may include more situations you have experienced within an academic setting. In comparison, Casper may present more “real world” situations when it comes to working with others; this is shown by the more diverse representation of its raters [https://altusassessments.com/rating-casper-infographic/p/1/]. As long as you are familiar with your own timing and the testing formats, answer the questions like a supplemental interview, and hope for the best.

    For residency programs that require completion of the Acuity Insights Duet and Snapshot assessments, a thorough self-reflection of your purpose in pursuing the specific specialty and expectations of what you need to be successful should guide your preparation.

    Finally, the utilization of SJTs with health professional education needs to become more transparent. Specifically:

  7. Admissions, student affairs administrators, and residency administrators must publish objective studies on the reliability of SJT assessments throughout professional training to confirm competency mastery.

    Because all health professions programs assess professionalism as a condition of student promotion or graduation from residency, each program should be able to map specific professional competencies with results from the SJTs, including tracking any gradual improvement on attributes that contribute to EI. Many published studies co-authored by Acuity Insights (formerly Altus) suggest that there is a relationship between Casper results and student success outcomes [https://altusassessments.com/resources/case-studies/]. Similarly, residency directors should be able to measure satisfactory completion of entrustable professional activities to show improvement of EI from the “baseline” SJT. There should ultimately be a determined threshold for SJT score with a candidate’s ability to complete residency training and manifest the ACGME (or similar professional) competencies successfully.

    To date, no known long-term objective studies have been conducted that show that the SJTs truly measure competency mastery. Many faults with the aforementioned studies have been pointed out. True reproducibility and validity have not been demonstrated on an individual basis, which ultimately is how the scores are presented for candidate selection. A well-designed meta-analysis would also inform residency program directors and professional school student advisors of the usefulness of these SJTs in their selection process, especially if their own programs have used SJTs to admit and enroll students and trainees.

    What is more remarkable is that there is little known about how Casper has been used for student selection at non-medical health professions programs after so many years of using it. How Casper results influence student selection in Canadian programs that have had years of using Acuity Insights assessments remains a carefully guarded secret. Furthermore, no independent peer-reviewed articles have published insights into how Casper is used in US non-medical health professions programs; this includes over 50 physician assistant programs, 26 physical therapy programs, 12 dental programs, 11 veterinary medical programs, 8 speech-language pathology programs, and 3 anesthesiology assistant programs. Most successful examples are digested in old Altus blog posts or archived presentations.

    Furthermore, claims that one cannot be “coached” to perform better on these SJTs contradict the consensus that emotional intelligence (EI) – consisting of self-awareness, self-regulation, social awareness, and social skills – can be improved. SJTs could be used as an assessment tool that rewards those who benefit from effective interventions or remediations to improve these qualities throughout professional education. Designing studies that correlate SJT performance with validated instruments that assess EI throughout the educational and training continuum should give greater insight into the effectiveness of the SJTs in measuring important professional competencies.

With the impending ban on race-factored admissions anticipated and the onset of more “pass/fail” scoring of board exams, the adoption of these SJTs will likely become widespread as a measure of non-cognitive attributes that are desired in holistic review. Future applicants – including those seeking graduate professional training – should see these assessments as a regular part of their future. Removing any doubt that these exams truly measure competency mastery will give confidence to admissions committees, residency program directors, employers, and thousands of future health professionals that an unfortunate low score will not derail one’s career trajectory.

NOTE: Since the original writing of this article, Altus Assessments (which runs the Suite of Casper, Duet, and Snapshot assessments) has rebranded to become Acuity Insights. This article has been updated, but many resources still reference Altus.

For inquiries on the SJT Experience survey, contact Dr. Emil Chuck (emilchuck@hpsa.org).

About HPSA

HPSA is a non-profit 501(c)3 organization that has supported individuals striving for health professional careers with donor-supported tools, resources, and community for over 20 years. HPSA is the publisher of Student Doctor Network articles, courses, and webinars. We welcome partnerships and collaborations with other non-profit organizations that share our mission. 

Exit mobile version