Machine learning model predicts physician turnover

Physician turnover is disruptive to patients and costly to health care facilities and physicians alike. In a new study, Yale researchers have used machine learning to reveal the factors—including the length of a physician's tenure, their age, and the complexity of their cases—that can increase the risks of such turnover.

Evaluating data from a large U.S. health care system over a nearly three-year period, they were able to predict, with 97% accuracy, the chances of physician departure. The findings, researchers say, provide insights that can help health care systems intervene before physicians decide to leave in order to reduce turnover.

The study was published Feb. 1 in PLOS ONE.

While health care facilities typically use surveys to track physician burnout and job satisfaction, the new study used data from electronic health records (EHRs), which are used by the majority of U.S. physicians to track and manage patient information.

The problem with surveys, said Ted Melnick, associate professor of emergency medicine and co-senior author of the new study, is that physicians often feel burdened to respond. Consequently, response rates are often low. "And surveys can tell you what's happening at that moment," he added, "but not what's happening the next day, the next month, or over the following year."

Electronic health records, however, which in addition to collecting clinical patient data also generate work-related data continuously, offer an opportunity to observe physician behavioral patterns moment to moment and over long periods of time.

For the new study, the researchers used three years of de-identified EHR and physician data from a large New England health care system to determine whether they could take a three-month stretch of data and predict the likelihood of a physician's departure within the following six months.

"We wanted something that would be useful on a personalized level," said Andrew Loza, a lecturer and clinical informatics fellow at Yale School of Medicine and co-senior author of the study. "So if someone were to use this approach, they could see the likelihood of departure for a position as well as the variables contributing most to the estimate in that moment, and intervene where possible."

Specifically, data were collected monthly from 319 physicians representing 26 medical specialties over a 34-month period. Data included how much time physicians spent using EHRs; clinical productivity measures, such as patient volume and physician demand; and physician characteristics, including age and length of employment. Different portions of the data were used to train, validate, and test the machine learning model.

When tested, the model was able to predict whether a physician would depart with 97% accuracy, the researchers found. The sensitivity and specificity of the model, which show the proportion of the departure and non-departure months that were correctly classified were 64% and 79%, respectively. The model was also able to identify how strongly different variables contributed to turnover risk, how variables interacted with each other, and what variables changed when a physician transitioned from low risk of departure to high risk.

The details about what's driving the prediction is what makes this approach particularly useful, researchers said.

"There have been efforts to make machine learning models not black boxes wherein you get a prediction but it's not clear how the model came to it," said Loza. "Understanding why the model produced the prediction it did is particularly useful in this case as those details are going to identify issues that may be leading to physician departure."

Through their approach, the researchers identified several variables that contributed to departure risk; the top four factors, they found, were how long the physician had been employed, their age, the complexity of their cases, and the demand for their services.

Whereas previous work enabled only analyses of linear relationships, the machine learning model allowed researchers to observe the challenges facing physicians with more nuance. For example, risk of departure was highest for physicians more recently hired and those with longer tenures but lower for those with middling tenure lengths. Similarly, risk of departure was higher for those up to 44 years old, lower for physicians aged 45 to 64, and higher again for those 65 years old or older.

There were also interactions between variables. For instance, more time spent on EHR activities decreased the risk of departure for physicians who had been on the job for less than 10 years. But for those physicians employed longer, it increased the risk of departure.

"The findings highlight there's not a one-size-fits-all solution," said Loza.

The risk of physician departure shifted throughout the study period, which covered a 34-month span from 2018 to 2021 (a period that included the pandemic and the pre-pandemic world), researchers said. They also identified specific variables that changed when a physician switched from low to high risk of departure; the proportion of EHR inbox messages responded to by a team member other than the physician, physician demand, and patient volume, were the variables that changed the most when a physician's risk flipped from low to high. COVID-19 waves were also linked to change in departure risk.

"I think this study is an important step in identifying factors that contribute to clinician turnover, with the ultimate goal of creating a sustainable and thriving work habitat for our clinicians," said Brian Williams, a medical informatics officer with Northeast Medical Group and an author of the study.

Towards that goal, the researchers created a dashboard that can display this information. Health care leaders see the value in the type of analysis this approach can provide.

"As physician burnout is an increasingly recognized problem, health care systems, hospitals, and large groups need to figure out what they need to do to ensure the emotional and physical health and well-being of the physicians and other clinicians who do the actual caring for patients," said Robert McLean, New Haven Regional Medical Director of Northeast Medical Group.

"Many health care systems already have wellness officers and wellness committees who could have the responsibility of collecting and analyzing this data and developing conclusions, which then would lead to implementation plans for changes and hopefully improvements."

Melnick added, "We're excited about the possibility of what this might look like in practice. And we're continuing to work on ethical implementation as this is really about how best to foster physician well-being and a thriving workforce."