Strong evidence of 'threshold effect' for NHS 18-week waiting list target

There's strong evidence of a 'threshold effect' in English hospitals' efforts to comply with the 18-week referral to treatment standard, concludes a long term data analysis of performance against the target, published online in the journal BMJ Quality & Safety.

The target focused activity on meeting the threshold requirement for patients on the waiting list after which it tailed off—the so-called threshold effect–rather than instigating pervasive improvement in practice, the analysis indicates. Clinical need may be a secondary consideration for meeting the target, suggest the researchers.

In 2012, an 18-week referral to treatment standard was introduced in England. This stipulated that at any point at least 92% of patients requiring hospital treatment should have been waiting for less than 18 weeks.

But how this target affects hospitals that are close to meeting it rather than those that have already met it or fall far short of doing so isn't clear.

To try and find out, the researchers retrospectively scrutinized publicly available monthly data on treatment waiting times for all 144 non-specialist acute NHS hospital trusts in England between January 2016 and September 2021.

Treatment covers admission to hospital; surgery; starting a course of drugs; fitting a medical device; agreeing to monitor the condition to see if further treatment is warranted; or receiving advice from a clinician on how to manage a condition.

They analyzed waiting times for all patients and then repeated this for 13 specific treatment groups: cardiology; cardiothoracic; dermatology; ear, nose and throat; gastroenterology; general surgery; gynecology; neurosurgery; ophthalmology; oral surgery; plastic surgery; trauma and orthopedics; and urology.

A threshold effect shows up as a spike in the data, known as discontinuity, which appears around the target threshold, explain the researchers.

Their analysis showed that the proportion of NHS hospital trusts meeting the target worsened over time, falling from 92% in 2015–16 to 64% in 2021–22.

Similarly, the percentage of trusts where patients waited less than 18 weeks fell gradually after the removal in 2016–17 of the financial sanction for breaching the standard. It then fell sharply during the early phase of COVID-19 in 2020–21 before gradually recovering.

There was strong evidence of a threshold effect up to 2019–20, despite the fall in the number of hospital trusts meeting the target.

The data repeatedly showed a large spike in the number of trusts exactly meeting the 92% target threshold for the 18-week standard, followed by a sharp drop after the target had been reached. This suggests that some trusts treat the minimum number of patients waiting under 18 weeks to comply with the standard, say the researchers.

The threshold effect only disappeared in the financial years 2020–21 and 2021–22, when the COVID-19 pandemic made it virtually impossible for most hospitals to meet the target.

Hospitals near the target likely act to clear it, while those further away from meeting it don't bother, on the grounds that it would be futile to do so, suggest the researchers.

They acknowledge that the study wasn't designed to investigate the behavioral triggers and motivations that might explain the threshold effect.

Nevertheless, they conclude, "Our findings suggest that hospital trusts may choose whom to treat based on the target instead of clinical need, as the target provides no encouragement to treat patients who have only been waiting for a short time or who have already passed the 18-week wait."

Performance targets are common in the NHS, they add. But they caution, "Policymakers should be circumspect in their use of targets. Second, if targets are used, then policymakers should examine for threshold effects routinely. Third, targets should be carefully designed to mitigate threshold effects."

In a linked editorial, Nigel Edwards, chief executive of the health think tank, the Nuffield Trust, London, says that while targets can be effective and are helpful for public accountability, the study "joins a long litany of examples of the unintended impact of targets."

He explains, "The most sustainable approach to meeting a target is to redesign processes and realign resources to ensure that the targets are met as a by-product of a well designed system.

"However, if there are insufficient resources—for example, in the case of waiting lists where demand exceeds capacity, or the organization lacks the skills and resources to undertake a major review of processes and ways of working—less desirable approaches may be taken."

These include demanding unsustainable levels of work from staff to meet the target, to the exclusion of almost everything else; and 'gaming' the targets, which can lead to an arms race of increasingly complex rules designed to eliminate the practice, he suggests.

The NHS has often focused on input targets rather than outcomes and on promoting 'achieving the numbers' over developing a better understanding of the problem, and often without involving those responsible for delivery, he adds.

"The experience of the use of targets in the English NHS, as evidenced by [the study authors] and many other researchers, suggests that over-reliance on a small number of high-profile measures is risky. A richer picture of how the system being measured works and how its staff and managers behave and are motivated is needed for sustainable long-term change," he writes.

"Effective performance improvement systems cannot be built solely on targets but need a great deal of managerial judgment. The improvement system and the local teams delivering improvement both need the management capacity to make good judgments to avoid the issues of gaming, overpromising, and other perverse ways of pursuing the metric while missing the point," he concludes.