Major measurement issues found in emergency department patient experience data

There are major measurement issues in patient experience data collected from U.S. emergency departments, including high variability and limited construct validity, according to an analysis published by researchers at the George Washington University (GW) and US Acute Care Solutions.

Patient experience data is becoming increasingly important in healthcare. The data is incorporated into the U.S. Centers for Medicare and Medicaid Services public reporting and value-based purchasing models for inpatient hospital care and will be used in the implementation of the Medicare Access and CHIP Reauthorization Act, known as MACRA. The data is also used to judge physician performance and hospital performance, often driving managerial decisions such as compensation and employment, and how a hospital is perceived in the community.

"The concept of measuring patient experience and rewarding providers who deliver a better experience is absolutely right on. No one argues with that. Yet what we found is that the data currently being gathered is not particularly reliable nor valid," said Jesse Pines, MD, MBA, director of the Center for Healthcare Innovation and Policy Research and professor of emergency medicine at the GW School of Medicine and Health Sciences.

Pines and his co-authors, including senior author Arvind Venkat, MD, chair of research at US Acute Care Solutions, looked at commercially-generated patient experience data from 2012-15 collected from a large sample of U.S. emergency departments. The data evaluated satisfaction surveys gathered from patients about their experience in the emergency department with questions on how they perceived their physician and the facility. The research team found the data varied greatly month-to-month, with physician variability considerably higher than facility variability.

"Presumably, if a physician produces a particular experience for his or her patients, then scores should be relatively stable over time. But from month-to-month, physician scores bounced around tremendously. In some cases, a physician was rated in the 20th percentile one month, then 80th percentile the next month, then in the 30th percentile. Facility scores also bounced around, but less so," said Venkat.

A major driving factor in the findings was the response rate, which was between 3-16%. "Imagine you conduct a survey, and only the very happy and very unhappy return their surveys," continued Venkat. "What you get is a very biased sample. That makes it difficult to come to any meaningful conclusions from the data."

Nevertheless, several facility factors were found to predict higher scores: departments associated with a residency program, a higher amount of older, male, and discharged patients without Medicaid insurance, lower patient volume, less requirement for physician night coverage, and shorter lengths of stay for discharged patients. Younger physician age, participating in patient satisfaction training, rising relative value units/visits, more commercially insured patients, higher CT/MRI use, working during less crowded times, and fewer night shifts were found to predict higher physician satisfaction scores. From this, the authors concluded that the survey process was marginally valid, and while some factors that predicted scores were within a hospital's control, many were not. Facility-level scores were shown to have greater construct validity - the degree to which a test is measuring what it claims to measure - than physician-level scores. Therefore, the authors recommend the use of risk-adjustment models to balance the scores to account for factors outside of a hospital's control.

"The voice of the patient is increasingly important in healthcare, particularly today with rising costs of care and increasing out-of-pocket costs for our patients. What is clear from our study is there needs to be a better process to measure, capture, and report patient experience data," said Pines.

The study, titled "Measurement under the Microscope: High Variability and Limited Construct Validity in ED Patient Experience Scores," was published in the Annals of Emergency Medicine.