Real-time outcome monitoring following oesophagectomy using cumulative sum techniques

Abstract

AIM: To examine the feasibility of prospective, real-time outcome monitoring in a United Kingdom oesophago-gastric cancer surgery unit.

METHODS: The first 100 hybrid (laparoscopic abdominal phase, open thoracic phase) Ivor-Lewis oesophagectomies performed by a United Kingdom oesophago-gastric cancer surgery unit were assessed retrospectively using cumulative sum (CUSUM) techniques. The monitored outcome was 30-d post-operative mortality, with the accepted mortality risk defined as 5%. A variable life adjusted display (VLAD) was constructed by plotting a graph of cumulative mortality minus cumulative mortality risk on the y axis vs sequential case number on the x axis. This was modified to a zeroed VLAD by preventing the plot from crossing the y = 0 axis - essentially creating two plots, one examining trends where cumulative mortality was higher than mortality risk (i.e., worse than expected outcomes) where y > 0, and vice versa. Alert lines were set at y = ± 2. At any point where a plot breaches an alert line, it is felt that the 30-d post-operative mortality rate has deviated significantly from that expected and an internal review should be performed.

RESULTS: One hundred cases were assessed, with a mean age of 66.4 years, mean T stage of 2.1, and mean N stage of 0.48. Three cases were commenced using a laparoscopic technique and converted to open surgery due to technical factors. Median length of inpatient stay was 15 d. The crude 30 d mortality was 5% and the incidence of clinically significant anastomotic leak was 6%. The VLAD demonstrated a plot of cumulative mortality minus cumulative mortality risk (i.e., 5% per case) which remained in the range -1.4 to +0.5 excess mortalities. With the alert set at two greater or fewer than predicted mortalities, this method does not approach the point of triggering internal review. It is however arguable that a run of performance that is better than expected, causing the plot to be well below y = 0, would mask a subsequent run of poor performance by requiring a rise of greater than two excess mortalities to trigger the alert line. The zeroed VLAD removes this problem by preventing the plot that is examining above expected mortality from passing below y = 0, and vice versa. In this study period, no audit triggers were reached. It is therefore possible to independently assess runs of good, or poor performance and so target internal audit to the appropriate series of cases. It is important to note this technique allows targeted internal review, in response to both above and below average outcomes. This study has demonstrated the feasibility of prospective outcome monitoring using the above techniques, actual real-time implementation has the potential to pick up and reinforce good practices when performance is better than predicted, and provide an early warning system for when performance falls below that predicted. Further development is possible, including more patient specific risk adjustment using the oesophago-gastric surgery physiological and operative severity score for the enumeration of mortality and morbidity score.

CONCLUSION: CUSUM techniques provide a potential method of prospective, real-time outcome monitoring in oesophageal cancer surgery.

Key Words: Oesophagectomy, Cumulative sum, Variable life adjusted display, Outcome, Mortality

INTRODUCTION

High risk surgery is, in the United Kingdom, undergoing a process of centralisation to high volume centres[1-6]. This has been driven by an increased focus on short and long term surgical outcomes and evidence showing better results from high volume centres. The authors have applied a real-time, risk-adjusted measure of outcomes after Ivor-Lewis oesophagectomy, with the aim of immediately triggering internal audit following a period of worsening outcomes.

In the United Kingdom, cardiac surgery has pioneered the development of real-time outcome monitoring, followed by transplant surgery, obstetrics, surgical education and most recently plastic and burns surgery[7-16]. Cumulative sum (CUSUM) techniques, incorporating risk-adjustment using variable life adjusted displays (VLADs), have been demonstrated to provide a valid, continuous measure of surgical outcomes[7]. They allow for ongoing assessment of outcomes against the acceptable standard, both better and worse, and early assessment when they are significantly different to those expected. It is crucial to note that this can pick up on periods of time when a unit is performing above average, as well as below average, and give the opportunity to reinforce those contributing factors. For periods of below average performance, the VLAD performs as an early warning system, allowing early recognition and management of contributing factors.

Collins has examined the role of VLADs and other CUSUM techniques in monitoring outcomes after oesophagectomy, utilising the Scottish audit of gastro-oesophageal cancer services data set[17]. That retrospective study has highlighted the feasibility of VLADs in monitoring the results of individual units.

The authors present the results of applying VLADs and zeroed-VLADs to monitoring outcomes in a single United Kingdom oesophago-gastric cancer centre and a mechanism for implementing this technique as a prospective, real-time measure.

MATERIALS AND METHODS

The results of the first 100 hybrid (laparoscopic abdominal phase, open thoracic phase) were assessed using CUSUM techniques. A VLAD of 30 d post-operative mortality, using an acceptable mortality risk of 5%, was created as detailed below. The results from this analysis were used to validate a tool for monitoring prospective outcomes using zeroed-VLAD plots.

A VLAD is constructed by plotting the cumulative mortality minus predicted mortality on the y axis, against cumulative number of cases on the x axis (Figure 1A). The generated plot is extended every case and should follow the y = 0 line if the actual rate of the measured outcome matches that of the predicted rate. If the actual rate is higher than predicted, the plot rises, and if the actual rate is lower than predicted, it falls. “Alert” lines are set at y = 2 and y = -2, and when the plot crosses the alert line the outcome rate is regarded as having deviated enough to warrant review.

Figure 1 Variable life adjusted display. A: Variable life adjusted display (VLAD) of 30 d mortality; B: Zeroed VLAD of 30 d mortality (5% risk)-trends of increasing mortality-trends of increasing mortality; C: Zeroed VLAD of 30 d mortality (5% risk)-trends of reducing mortality.

It is apparent that a good “run”, as seen by the large negative deflection in Figure 1A, could mask a subsequent poor run, as that run would start below y = 0 and require more than two excess mortalities to trigger a review. This problem can be eliminated by preventing the plot from becoming negative, so y > 0 at all times (Figure 1B). This is enabled by the simple expedient defined by the equation:

= if y < 0, y = 0

Essentially, only runs of worsening outcome are examined. Likewise, the mirror image plot, where y < 0 at all times, can be used to trigger review when results are significantly better than predicted (Figure 1C).

Statistical analysis

Data were collected and analysed using Microsoft Excel 2007. Construction of the relevant plots was performed as described elsewhere in this study.

RESULTS

The patient and disease load, and non risk-adjusted outcomes were as expected for a United Kingdom upper gastrointestinal tract (UGI) cancer centre (Table 1).

Table 1 Demographics of study population.

Demographics	Data
Age, yr (mean ± SD)	66.4 ± 9.2
Mean T stage	2.1
Mean N stage	0.48
Median length of stay, d (25%-75%)	15 (13-20)
Crude 30-d mortality	5.4%
Anastomotic leak rate	6.5%

The zeroed-VLAD (Figure 1B) demonstrates two spikes in mortality, neither of which crosses the alert line at y = 2 excess mortalities. The zeroed-VLAD for better than average results (Figure 1C) demonstrates one “run” of good results which does not cross the alert line at y = -2 (i.e., two fewer mortalities than expected).

DISCUSSION

The authors demonstrate the first published implementation of VLADs to monitor oesophagectomy outcomes within an UGI cancer centre. While this technique has gained widespread acceptance in cardiac surgery, its uptake in other specialties is still taking place. The authors believe that this could provide a robust tool for monitoring outcomes in other high risk operations and are using this to monitor the outcomes in their unit.

The benefit of targeted real-time internal review, compared to the current practice of retrospective assessment, is in the opportunity to identify and address causes of poor results early. This clearly prevents potentially deleterious practices continuing longer than necessary and could improve patient outcomes. Likewise, the opportunity to flag up periods of better than expected performance allow for identification, standardisation and dissemination of exceptional practices.

This study has used a 5% mortality risk as the acceptable standard. This is however being further modified by the centre and the authors will be assessing risk on an individual basis using the oesophagogastric physiological and operative severity score for the enumeration of mortality and morbidity score[18]. It is thought that more accurate risk assessment, in a similar fashion to cardiac surgeons, will provide a further refinement to this technique.

While some within the profession may resist the concept of rigorous and open outcome monitoring, it is likely to become the standard as healthcare outcomes become more closely observed[19]. Indeed, this is not intended as a punitive measure, but as a means to identify early those random factors that have a significant impact upon outcome. This however raises the issue of how to manage the review process, particularly that when outcomes are poorer than desired. While not further elaborated here, the authors propose a checklist-based approach to review of contributing cases. This is due to recent work in healthcare checklists showing a benefit in removal of personal factors, and ensuring that all potential issues are covered in a standardized fashion[20-26].

This tool, if widely implemented, could well stimulate close monitoring and therefore improvement of oesophagectomy outcomes.

COMMENTS

Background

It is unusual in the field of healthcare for outcomes to be monitored in a prospective, real-time fashion. The potential of real-time outcome monitoring, with the early detection of both good and poor outcomes, is the improvement of medium and long term outcomes. Currently, in the United Kingdom, only cardiac surgery rigorously monitors surgical outcomes in this fashion, using cumulative sum (CUSUM) techniques. This study examines the feasibility of adapting CUSUM techniques to monitoring of 30 d mortality in oesophago-gastric surgery.

Research frontiers

CUSUM techniques are gradually being introduced to healthcare, with increasing understanding of their role in outcome monitoring. This is being explored in many fields, particularly surgery, obstetrics and skills training.

Innovations and breakthroughs

Previous work has defined the role of risk-adjusted outcome monitoring using variable life adjusted displays (VLADs). These essentially track the outcome rate compared to the acceptable, or predicted outcome rate and trigger when the plot deviates to a significant extent. Other studies are examining the role of these techniques in transplant surgery, obstetrics, surgical education and other medical fields, although to date only cardiac surgery has successfully implemented a prospective CUSUM based means of monitoring surgical outcomes.

Applications

If feasible, the techniques outlined hear could form the basis for prospective, real-time outcome monitoring in oesophago-gastric surgery. This is potentially a significant improvement on current retrospective outcome assessment.

Terminology

CUSUM is a statistical control technique which monitors cumulative outcome rate vs sequential cases, essentially creating a plot with the gradient being the outcome rate. VLADs are modifications of the basic CUSUM plot, plotting cumulative outcomes minus cumulative expected outcome rate against sequential cases. A VLAD for a process which has an outcome rate as expected should demonstrate a horizontal line, a process with above expected outcome rate is a rising line, and vice versa.

Peer review

This is a good descriptive study in which authors examine the feasibility of prospective, real-time outcome monitoring in a United Kingdom oesophago-gastric cancer surgery unit. The results are interesting and suggest that CUSUM techniques provide a potential method of prospective, real-time outcome monitoring in oesophageal cancer surgery.