New metric more accurately evaluates physician performance

• Link to: Northwestern Now Story

Inspired by the advanced analytics used in sports, a Northwestern University-led research team has developed a new approach for measuring physician adherence to best practices. The study also shows that high-performing individuals are more likely to be aware of system-level challenges. 

The data-driven method takes into consideration the characteristics of the patients seen by a physician (how sick a patient is, for example) in order to estimate an adjusted performance that more accurately captures the know-how of a set of physicians. The researchers’ case study focused on patients with acute respiratory distress syndrome (ARDS), a form of lung failure.

“Our new metric could help change behavior more broadly, beyond our case study, and increase adoption of medical innovations,” said Luís A. Nunes Amaral, a data science expert and a corresponding author of the study. “Our method also is very flexible — it can be used on individual physicians, on groups of physicians, on entire hospitals or on entire hospital groups.”

Amaral is the Erastus Otis Haven Professor of chemical and biological engineering at the McCormick School of Engineering. He also is co-director of NICO (Northwestern Institute on Complex Systems).

The study was published recently by the journal BMC Medical Research Methodology. 

Dr. Curtis H. Weiss, a pulmonary and critical care physician at NorthShore University HealthSystem and an adjunct assistant professor at McCormick, also is a corresponding author of the study. Amaral has long collaborated with him, using data science to improve critical care medicine.

Amaral, Weiss and their research team evaluated the ventilator management by 48 critical care physicians of 362 patients with ARDS at a large academic hospital. Because the lungs of patients with ARDS are very fragile, accepted best practice is to mechanically deliver lower breath volumes to these patients than is standard for patients on ventilators. Proper ARDS diagnosis is critical but can be challenging. 

The team examined the relationship between recognition of ARDS and physician characteristics, such as demographics, social network position and attitudes toward innovation. They found the only factor associated with ARDS recognition was training background; pulmonary and critical care medicine training was associated with higher recognition. Once a difference is identified, interventions can be designed to address the problem.

“ARDS is a pulmonary disease, and so it makes sense that physicians with advanced pulmonary training would be more primed to spot ARDS,” said Meagan A. Bechel, the paper’s first author and a recent graduate of Northwestern’s Medical Scientist Training Program.

“Even after we account for the number of ARDS patients seen, the severity of disease and the number of years in clinical practice, specialty training still has the largest impact on ARDS recognition,” Bechel said. “This is important because ARDS is a diverse syndrome and is treated by physicians from several specialties — in fact, the majority of our ARDS patients were cared for by non-pulmonary physicians.” 

 As an analogy, consider the number of home runs hit by a baseball player as a measure of performance. Given two hitters with the same skill level, the one playing in a smaller field would be expected to hit more home runs. Paying more money to the one that hit more home runs without accounting for the difference in conditions would be a mistake.

“Similarly, physicians see patients with different characteristics,” Amaral said. “Some patients are easier to diagnose than others. If a physician sees a group of easy-to-diagnose patients while another sees patients who are very hard to diagnose, then it would be easier for the former to appear to be performing better, even though that is not the case.”

Another interesting result of the study is related to the so-called “squeaky wheel.” When the researchers asked physicians how easy it was to find all the information for correctly diagnosing ARDS, they found that most physicians — except the higher-performing ones who recognized ARDS — had no complaints about the difficulty of the task.  

“This suggest that users of a system might not identify problems with it because they are not engaging deeply with the system,” Amaral said. “That is, the squeaky wheel is actually telling us where the problems are. It’s a challenge for physicians to get the disparate medical information they need for an ARDS diagnosis.”

Weiss noted that severe COVID-19 can lead to ARDS, but that the data used in this study was from before the pandemic.

The title of the paper is “The first step is recognizing there is a problem: a methodology for adjusting for variability in disease severity when estimating clinician performance.” Meagan Bechel, a postdoctoral fellow in Amaral’s lab and a medical student at Northwestern University Feinberg School of Medicine, is the first author of the paper. Other authors are Adam R. Pahand Stephen D. Persell of Northwestern University.

The research was supported by the National Institute of General Medical Sciences (grant T32GM008152), the National Heart, Lung, and Blood Institute (grants K23HL118139 and R01HL140362-01A1), the Francis Family Foundation, the Department of Defense Army Research Office (grant W911NF-14-1-0259) and the National Center for Research Resources (grant 5UL1RR025741), which is now at the National Center for Advancing Translational Sciences (grant 8UL1TR000150).