Risk adjustment method
It is common for surgical outcome information to be risk-adjusted because the characteristics of patients vary across organisations, and the outcomes of care will be influenced by these characteristics. For example, on average, outcomes are worse for sicker patients.
The postoperative mortality figures for the NHS trusts were adjusted for several patient characteristics: age at diagnosis, performance status, ASA grade, presence of significant comorbidities, tumour site, pathology T stage, number of positive nodes and receipt of neoadjuvant therapy.
We used the same patient characteristics in the risk adjustment for both the 30- and 90-day mortality rates. The risk adjustment process works by taking into account the overall average mortality rate and the distribution of deaths across the NHS trusts. Because of this, the process will occasionally produce a slightly higher estimate of the 30-day mortality rate than the 90-day mortality rate. This quirk of the risk adjustment process is related to the degree of certainty with which we can estimate surgical outcomes. As explained below, we use a control chart technique to take into account the degree of certainty in these estimates when assessing surgical outcomes.
Using control charts to check outcomes are within the expected range
Trust outcome information will always differ from the outcome figures published at a national level because of random variation – some NHS trusts will have higher values and some lower. This variation is not communicated when figures are ranked. Consequently, these NHS trust figures should not be ranked. Presenting this information in the form of a league table of outcomes would be misleading and lead to wrong conclusions about an individual trust’s performance.
The variation in postoperative mortality rates was examined using a graph known as a funnel plot. The benefit of this approach is that it shows whether the outcomes for individual NHS trusts differ from the national average by more than would be expected due to random fluctuations. Random variation will always affect outcome information like mortality rates, and its influence is greater among small samples.
On these funnel plots, each dot represents an NHS trust and they are compared against the national average. The national average for 30-day and 90-day postoperative mortality over the three year period covered by the analysis was 1.6% and 3.2%, respectively. The vertical axis indicates the outcome, with dots higher up the axis showing NHS trusts with higher mortality rate. The horizontal axis shows surgical activity with dots further to the right showing the NHS trusts that perform more operations.
The funnel plot includes two control limits to define the range within which we would expect NHS trust outcomes to lie. Following convention, we use 99.8% control limits. It is unlikely for an NHS organisation to fall beyond these limits solely because of random variation (a 1 in 500 chance). If the outcome figures for an NHS trust fell outside the outer limits, there could be a systematic reason for the higher or lower rate, and they would be flagged as an outlier for further investigation. The main cause of variation within the control limits is likely to be random variation.
The expected range of outcomes for a given number of operations (the sample size) is shown on the funnel plots by the different coloured regions. We would expect the figures of the NHS trusts to fall within the green area if their outcomes only differed from the national average because of random variation.
None of the NHS trusts included in the analysis had a risk-adjusted mortality rate above the 99.8% control limit.