Can AI tools help train a more effective physician?

A patient arrives with a persistent, rattling cough. The novice clinician runs through a series of hypothesis-driven questions and conducts their physical exam until they identify the cause and decide on the best treatment plan. All the while, the ambient listening capabilities of artificial intelligence (AI) record the encounter and analyze their clinical reasoning skills.

That’s the vision of CRISP (Clinical Reasoning Insights for Shaping Performance), a new Penn Medicine initiative that will use AI-enabled systems to deliver robust, data-driven feedback to medical students and postgraduate trainees, accelerating their growth as expert clinicians. The project is led by Perelman School of Medicine faculty Jessica Dine, MD, MSHP; Janae Heath, MD, MSCE; Jennifer Kogan, MD; and Ilene Rosen, MD, MSCE, with support from the Department of Informatics, Biostatistics and Epidemiology.

“Clinical reasoning is a pivotal piece of excellent patient care,” said Heath, associate program director for the Internal Medicine Residency Program and assistant professor of Pulmonary, Allergy and Critical Care. “And so an improvement in those skills, resulting in higher levels of expertise, is going to directly benefit our patients.”

CRISP was one of 11 innovative projects recently selected for a four-year, $1.1 million grant through the American Medical Association’s Transforming Lifelong Learning Through Precision Education Grant Program. Precision education aims to tailor training to meet each learner’s needs instead of the traditional “one size fits all” approach to medical education. It’s also part of Penn Medicine’s goal to individualize learning across the medical education continuum.

“It’s incredibly exciting because not only is this the first true precision education at Penn, but I think it’s a really unique way to do precision education,” said Dine, associate dean of Assessment, Evaluation and Medical Education Research and a professor of Pulmonary, Allergy and Critical Care.

This new initiative in medical education and training is one of many ways that Penn Medicine is harnessing AI to improve the way that clinicians work and ultimately strengthen patient care, from a new tool that can quickly synthesize a patient’s medical history to an AI-enabled system that assists with replies to patient messages to an AI “scribe” that helps with note-taking during patient visits.

Working together to strengthen clinical reasoning

While clinical reasoning is at the heart of effective, high-quality medicine, it can be tricky to teach—it’s shaped by chance patient encounters and subjective assessments. Faculty can’t observe every patient interaction in real time, so they often rely on students’ and trainees’ presentation of a case to retroactively assess their clinical reasoning skills.

“Clinical reasoning is a really hard skill to capture well,” said Heath, drawing on her work as part of the project team. “But over the last two years, there’s been an explosion of new technologies available, including ones that Penn Medicine has embraced, that have created this new opportunity to capture clinical reasoning in different domains.”

The new initiative will include undergraduate and graduate medical education learners in four specialties: Internal Medicine, Emergency Medicine, Surgery and Radiology. The goal is to create a scalable, precision education tool that supports individualized coaching and competency-based progression across medical specialties.

Penn Medicine is an ideal environment to explore novel assessment models because of its educational structure—which is integrated under shared leadership with the health system’s clinical care—and a culture where people regularly collaborate across medical specialties and academic disciplines, project leaders said.

“Penn Medicine is thinking about education across the continuum and through a really innovative lens,” Heath said. “We have team members across the institution, including the informatics team, who are pivotal to this project. That’s allowing us to tackle this issue, thinking about it differently than anyone’s thought about it before.”

The team will spend the first year of the grant developing a prototype for studying clinical reasoning in each specialty, designed for each area’s unique workflows. The team plans to use AI to collect and analyze ambient audio from clinician-to-clinician conversations in Internal Medicine and clinician-to-patient interviews in the Emergency Department. For Radiology, researchers will use AI to analyze clinician documentation, and for Surgery, they ’ ll track clinician interaction patterns within the electronic health record.

As part of that work, the team will create profiles of diagnostic and therapeutic reasoning for learners across different stages in their skill development. Direct observation will continue to be a key part of training, but the goal is to measure clinical reasoning in a more data-driven way, providing a clearer picture of a learner’s progress and where they still need to grow.

“The idea is to really make a difference in how learners progress through their clinical education and think about their own growth over time, including when they’re finished with their training,” said Rosen, associate dean for Graduate Medical Education (GME) and vice president for GME and an associate professor of Sleep Medicine.

Knowing that more seasoned clinicians tend to summarize cases succinctly and precisely, part of the analysis of skill development will consider the number of words and semantic richness of learners’ speech and clinical documentation. The project will also look at clinical reasoning in context, considering factors like time of day and team dynamics. Has a trainee been working for 16 hours straight? Is it their first day or third month with a team?

The use of ambient audio won’t be constant. “It could feel very Big Brother pretty quickly,” said Kogan, vice dean for Undergraduate Medical Education and William Maul Measey President’s Distinguished Professor in Medical Education. “And so how do you deploy this in a way that is meant to give people useful information as opposed to feeling like you’re in a surveillance system?”

Introducing system change can be challenging, and researchers expect it will take some time for clinicians to adjust. The team also wants to make sure that AI-enabled systems don’t perpetuate systemic biases. “We’ve built in a lot of equity monitoring,” Heath said. “We’re being really intentional about that piece of it because we don’t want to contribute to problematic assessments.”

Creating the future of precision education

The project is co-designed with learners, educators and health system stakeholders. To kick off the initial stage, the CRISP team will host an “education innovation hackathon” March 13-15 to bring together clinical educators, medical students and residents, informatics experts and students from across the University of Pennsylvania.

“The learner voice is really critical,” Heath said. “Residents and medical students have a role in the development of each of those prototypes. They’re instrumental in the development and refining.”

In 2027, the team plans to test their early prototypes with several hundred medical students and residents across the four specialty areas. The researchers will use the initial feedback to make any necessary adjustments before rolling out the initiative to a larger group.

As part of the grant, the CRISP team joined the American Medical Association’s precision education consortium, connecting with other grant recipients through regular Zoom meetings and biannual in-person gatherings to share progress, challenges and solutions. The ongoing collaboration is designed to strengthen all 11 funded projects, with a vision of transforming medical education nationwide.

“By having those conversations, we can accelerate that growth more than if we were all working in silos, which I think is incredibly exciting,” Kogan said.

Penn Medicine faculty hope that CRISP becomes a model for improving clinical reasoning skills in other medical schools and health systems.

“If our goal is to have a true learning health system, then this is a great prototype to do that,” Dine said. “Trainees receive feedback in real time, and they can learn from it, but then the system also learns from it and adapts, which I think is really exciting.”