Method of analysis

It is common for surgical outcome information to be risk-adjusted because the characteristics of patients treated by surgeons can vary, and the outcome of care will be influenced by these characteristics.  For example, on average, outcomes are often worse for sicker patients. 

An assessment of risk-adjusted outcomes by NHS trust for curative surgical patients was published in the NOGCA 2013 Annual Report.  We found that the risk-adjusted 30-day postoperative mortality for all NHS trusts were within the expected range. 

The lack of clinical data on the additional patients used in this report meant that we could not produce risk-adjusted outcomes for NHS trusts and surgeons at this time.  Producing risk-adjusted outcome statistics based on small number of operations can be misleading.  Having more data available for the next publication of surgeon outcomes will overcome this current limitation.

Using control charts to check outcomes are within the expected range

Surgeon outcome information will always differ from the outcome figures published at a national level because of random variation – some will be higher and some lower. This variation is not communicated when figures are ranked.  Consequently, these surgeon figures should not be ranked.  Presenting this information a league table of surgeon outcomes would be misleading and lead to wrong conclusions about an individual surgeon’s performance.

The variation in post-operative mortality rates was examined using control chart techniques. The benefit of this approach is that it shows whether the outcomes for individual NHS trusts or surgeons differ from the national average by more than would be expected from random fluctuations.  Random variation will always affect outcome information like mortality rates, and its influence is greater among small samples. 

Following convention, we use 99.8% control limit to identify “outliers”.  It is unlikely for an NHS organisation to fall beyond these limits solely because of random variation (a 1 in 500 chance).  If the outcomes of an NHS trust or surgeon fell outside the outer limits, there could be a systematic reason for the higher or lower rate, and they would be flagged as an outlier for further investigation. Variability within the control limits is likely due to random variation.

For NHS trusts, we show these control limits using a funnel plot.  On these plots, each dot represents an NHS trust. The solid horizontal line is the national average. The vertical axis indicates the outcome with dots higher up the axis showing Trusts with higher mortality rate. The horizontal axis shows surgical activity with dots further to the right showing the Trusts, which perform more operations. 

The expected range of outcomes for a given number of operations (the sample size) is shown on the funnel plots by the funnel-shaped dotted lines. These lines define the region within which we would expect the outcomes of operations to fall if outcomes only differed from the national rate because of random variation.

For surgeons we report the upper 99.8% control limit for the given number of operations in terms of the number of deaths that would lie within these limits. For example, a surgeon operating on 15 patients would have an upper 99.8% control limit corresponding to four deaths. In other words, up to four patient deaths for this surgeon could be explained by random variation. None of the surgeons included in the analysis had an observed number of patient deaths above the 99.8% control limit.