Penn Study Pinpoints Accuracy of ICU Doctors’ and Nurses’ Predictions of Patient Outcomes

WASHINGTON, D.C. — Physicians in intensive care units (ICUs) routinely consider their patients’ chances of survival and recovery when guiding patients and family members in making important decisions about care plans. A new study is shedding light on the accuracy of those judgments — and for the first time also reveals the accuracy of ICU nurses’ predictions of patient outcomes. For example, the study shows that ICU physicians are better at predicting whether patients will be alive in six months than they are at predicting patients’ cognitive function in six months, and the more confident doctors are when making predictions, the more accurate those predictions tend to be. The study, led by researchers at the Perelman School of Medicine at the University of Pennsylvania, is published online today in JAMA and will also be presented at the American Thoracic Society’s 2017 International Conference in Washington, DC.

“Our study showed that doctors’ accuracy in predicting what will happen to patients six months after leaving the ICU was quite good when the doctors were confident in those predictions, and even better when nurses agreed with them,” said study senior author Scott D. Halpern, MD, PhD, an associate professor of Medicine, Epidemiology, and Medical Ethics and Health Policy at Penn and Director of Penn’s Palliative and Advanced Illness Research (PAIR) Center. “When deciding how to frame their prognostic judgments to patients or family members, ICU physicians should consider their own confidence level and their agreement, or lack thereof, with bedside nurses.”

The study included 47 physicians, 128 nurses, and 303 patients across three hospitals and five ICUs within the University of Pennsylvania Health System. All patients included in the study had been in the ICU for three to six days and had required either a ventilator or medication to keep their blood pressure high enough for them to live. Because only brief periods of such life support were required to be in this study, the results apply to many patients admitted to ICUs.

At the time of enrollment, attending doctors and bedside nurses were asked to predict whether each patient would die during their hospital stay or within six months. If clinicians predicted that the patient would survive for at least six months, they were then asked to predict whether the patient would be unable to: return to his or her original residence, use the toilet independently, climb ten steps independently, or function well cognitively (“remember most things, think clearly, and solve day-to-day problems”).

The accuracies of clinicians’ predictions were mixed. The ICU doctors performed well at predicting patients’ in-hospital mortality, six-month mortality, and six-month inability to use the toilet independently. For example, doctors’ predictions for most of these outcomes were as or more accurate than many medical tests, such as blood tests or x-rays. And when doctors were confident about these predictions, their accuracy in predicting whether or not a patient would be alive six months later became nearly perfect. The one important exception was in predicting cognitive dysfunction, where doctors’ predictions were less accurate than most medical tests.

Nurses’ predictions had a similar pattern across outcomes, but were on average moderately less accurate than doctors were. And their abilities to predict cognitive dysfunction were no better than chance.

Halpern noted that estimating hospital patients’ long-term cognitive ability has never been easy to do accurately. “Prior studies have also highlighted the difficulties, in some cases, of predicting future cognitive function. What this study suggests is that doctors can reveal their uncertainty to families about patients’ abilities to think clearly, while also confidently guiding families as to whether or not a patient will be able to go to the bathroom or even be alive,” he said.

The results may impact practice for the tens of thousands of Americans in ICUs each year for whom clinicians and family members must make crucial decisions about whether to use life support and for how long. “These data are important,” Halpern said, “because nearly all families want to know what the doctors believe will happen, but doctors often fail to share their prognoses because they know that there’s no way to be 100 percent certain.”

The results of this study confirm that doctors are not 100 percent accurate, but also show that when doctors are confident in their predictions, they get quite close to predicting things like whether a patient will ever get back to living in the place he or she lived before the ICU. “My hope is that the results of this study will reinforce to clinicians that their predictions, while not perfect, are still likely to be helpful to the families of critically injured or ill patients who are faced with making extraordinarily difficult decisions about long-term care for their loved ones.” Halpern said.

The research team says in the future, studies should focus on better understanding the factors that influence doctors’ and nurses’ predictions of ICU patient outcomes.

The lead author on the study is Michael E. Detsky, MD, MSHP, an assistant professor at the University of Toronto, who was a Masters of Science in Health Policy research student at Penn when the study was conducted. Other co-authors from Penn are Michael O. Harhay, Elizabeth Cooney, Nicole B. Gabler, Sarah J. Ratcliffe, and Mark E. Mikkelsen.

Funding for the study was provided by the National Heart, Lung, and Blood Institute (T32-HL098054, F31-HL127947) and the Otto Hass Charitable Trust.