PHILADELPHIA – Almost 10 percent of the nation’s entire population live with substance use disorder, but many struggle to find the right help – a task which is made more difficult because there is no standardized rating system to ensure the quality of care within specialized drug treatment facilities. Even the efforts that do exist to evaluate these entities don’t seem to be aligned with the central concerns of patients, according to a new study from Penn Medicine researchers which was published today in the Journal of General Internal Medicine.
Using a machine learning application, researchers uncovered the most common themes associated with facilities’ reviews on Google and Yelp, then compared them to a survey run by the Substance Abuse and Mental Health Services Administration (SAMSHA) — the most widespread evaluation method currently used for these facilities. However, that survey is intended to be an inventory of services provided and not patient satisfaction, so only 7 percent of its codes aligned themes from the patient reviews.
“It would be prudent of us to have a comprehensive method of defining quality, capturing data and adjusting care to be both meaningful and patient-centered at treatment facilities,” said lead author Anish Agarwal, MD, an assistant professor of Emergency Medicine. “Without these considerations, we’re attempting to tackle a component of the opioid epidemic without taking into account the voices of those who are actually seeking help.”
Online reviews are considered to be unfiltered sources that come directly from patients and their support systems, a resource that the team, led by Agarwal and Raina Merchant, MD, the director of the Penn Medicine Center for Digital Health and an associate professor of Emergency Medicine, consider to be relatively untapped. The team analyzed 7,823 online reviews for 539 Pennsylvania facilities posted between the summers of 2010 and 2018. Both Yelp and Google use a five-star rating system, and the researchers found that 43 percent of the reviews fell into the five-star category, while another 34 percent were one-star, leaving just 23 percent for the middle ratings.
“We’ve seen this polarized trend more than once in online reviews involving patient satisfaction,” said Agarwal. “People tend to be motivated to post a review if they feel that they had either a great — or a terrible — experience, and this usually manifests as either clicking one- or five-stars. We’ve seen a similar trend in online reviews of hospitals, emergency departments, and urgent care centers.”
Almost 70 percent of the reviews were accompanied by text explaining the reviewers’ thoughts. But reviews with text were significantly more likely to be lower rated (averaging 2.9 stars) than the ones without text (3.8 stars).
“People with strong opinions tend to want to share their experiences online. I imagine, the ones without text likely score higher because people have less of a strong opinion to voice,” Agarwal explained. “When people leave a more negative review, like a one-star, they often want to follow it up with a ‘why.’ This is very helpful to researchers like us because we can begin to understand more about the person’s experience and why it may have been subpar, and then we can take steps to address it.”
To dig into the text of the online reviews, the researchers fed the review text into an automated machine learning tool to identify and group instances of the same words across reviews. They then applied a technique known as Differential Language Analysis to pull out themes that correlated to certain online reviews. When it came to five-star reviews, there was a correlation with mentions of a focus on recovery, staff helpfulness, compassionate care, experiencing a “life-changing moment,” and general professionalism. However, the themes most directly tied to one-star reviews included long wait times, poor accommodations, poor phone communication, the types of medications offered, and appointment availability.
The researchers identified three of the 14 services categories in the SAMSHA survey that aligned with the top themes derived from patients’ online reviews. Moreover, 12 of the service codes from the survey, out of 162, total, addressed top themes from online. Seeing the differences between the survey and what patients post about provides a new source of information to help guide quality improvement within these centers.
Moving forward, Agarwal, Merchant and their fellow researchers hope that their discoveries can help build the yard stick by which drug treatment facilities are measured to better reflect the actual feelings of those living with substance use disorder.
“Future work will focus on evaluating reviews across the United States and use this to collaborate with organizations creating national quality measures that encompass patient feedback and other performance measures,” Merchant said. “We would also like to work with online review platforms to add targeted questions to these portals that seem most important to patients and caregivers.”
Other Penn authors on this study included Viven Wong, Arthur M. Pelullo, Sharath Guntuku, Daniel Polsky, David A. Asch, and Jonathan Muruako.
Penn Medicine is one of the world’s leading academic medical centers, dedicated to the related missions of medical education, biomedical research, and excellence in patient care. Penn Medicine consists of the Raymond and Ruth Perelman School of Medicine at the University of Pennsylvania (founded in 1765 as the nation’s first medical school) and the University of Pennsylvania Health System, which together form a $8.9 billion enterprise.
The Perelman School of Medicine has been ranked among the top medical schools in the United States for more than 20 years, according to U.S. News & World Report's survey of research-oriented medical schools. The School is consistently among the nation's top recipients of funding from the National Institutes of Health, with $496 million awarded in the 2020 fiscal year.
The University of Pennsylvania Health System’s patient care facilities include: the Hospital of the University of Pennsylvania and Penn Presbyterian Medical Center—which are recognized as one of the nation’s top “Honor Roll” hospitals by U.S. News & World Report—Chester County Hospital; Lancaster General Health; Penn Medicine Princeton Health; and Pennsylvania Hospital, the nation’s first hospital, founded in 1751. Additional facilities and enterprises include Good Shepherd Penn Partners, Penn Medicine at Home, Lancaster Behavioral Health Hospital, and Princeton House Behavioral Health, among others.
Penn Medicine is powered by a talented and dedicated workforce of more than 44,000 people. The organization also has alliances with top community health systems across both Southeastern Pennsylvania and Southern New Jersey, creating more options for patients no matter where they live.
Penn Medicine is committed to improving lives and health through a variety of community-based programs and activities. In fiscal year 2020, Penn Medicine provided more than $563 million to benefit our community.