London Ambulance New Year IT failure blamed on upgrade

The New Year’s Day failure of London Ambulance’s dispatch system, which took the system down for five hours on its busiest night of the year, has been blamed on a ‘historic upgrade’.

Staff had to switch to pen and paper, and some people had to wait seven hours for an Ambulance, one call took 25-minutes to answer, and other calls had to be diverted to Scotland.

One person died, possibly as result of treatment being delayed. A coroner’s inquest is pending.

LAS chief executive Garrett Emmerson said: “The system fault that occurred on New Year’s Day has been fixed and cannot now happen again.”

Emmerson added: “This review exposed some significant short-comings in our IT processes and governance, which we have taken action to address. However, it also found that the computer system that we use to respond to 999 calls is fit for purpose.”

US security company Northrop Grumman won the contract to provide the CAD system to the London Ambulance Service in 2008.  The system, called CommandPoint, was rolled out in 2012, replacing an in-house developed system.

An internal inquiry by LAS said that the cause of the IT failure was a “historic upgrade that over time caused the CAD software to consume all available system resources, slowing it down to the extent it became unusable”.

The software upgrade, made 18 months previously, had prevented a database feature known as the ‘Recycle Bin’ from being emptied effectively. While the CAD system was being heavily used on New Year’s Day, the process to manage the Recycle Bin consumed all the available system resource, slowing it down to the point it no longer worked.

At this point, LAS switched to use of a paper-based backup system in its two emergency operations centres in Waterloo and Bow.

The CAD system was restarted and available within 90 minutes, but the need to then reconcile the paper records back onto the CAD system, hampered by the increased number of calls, meant full recovery to normal operations took a further four hours.

Following the incident four separate reviews were carried out, which the London Ambulance Report summarises.  LAS has accepted the findings of the reviews.

The report praises LAS staff, “both in the call centres and on the front line, [who] deserve thanks and recognition for the skill and professionalism they showed, under exceptionally difficult conditions, during the incident”.

The report also notes that the ambulance service has since successfully managed to respond to the Westminster and London Bridge terrorist attacks, together with Grenfell Tower fire, and routinely plans for a range of contingencies and large-scale events such as the Olympic and Paralympic Games, as well as IT and other system issues.

As well as the failures in the IT system the report says the incident “also exposed gaps in the expertise and capacity of the IT function at the [London Ambulance] Trust”.

The report says that “Management and routine maintenance of the trust’s IT systems were found to be insufficient, with a lack of oversight at board level.”

Since the incident, LAS has appointed a chief information officer, Ross Fullerton, and a non-exec director, Sheila Doyle, to provide better oversight and management of the call handling system and wider IT issues.

The report also found that the New Year’s Computer Aided Dispatch (CAD) system failure was not an isolated incident.

There had been two earlier failures of the CAD in the previous 12 months, which the review said raised questions about the quality and resilience of the system.

Subsequent investigation of the New Year’s failure found that the recycle bin was not needed to operate the call handling system; it was therefore emptied and then disabled.

As part of the inquiry an independent review of the IT system, carried out by PA Consulting, examining why it failed, the steps that were taken to fix the problem and the IT systems, and what expertise and support LAS has available.

One of the findings of the review was that further simulation training and preparation for concurrent events were identified as being needed, particularly true when dealing with concurrent events during a period of very high call volumes.

As a result of the failure, NHS England commissioned an NHS-wide review looking at the response of the NHS more widely to CAD system failures, to identify what additional learning could have taken place after previous outages.

Anne Rainsberry, NHS England’s regional director for London, said: “We are satisfied with the immediate actions that the London Ambulance Service took to rectify the cause of the outage, as well as the organisation’s assessment of the impact that this had on patients. The Service has made substantial progress against the investigation’s recommendations and over coming months we will seek assurance that the outstanding recommendations have been actioned.”

Steve Russell, NHS Improvement’s executive regional managing director for London, said: “London Ambulance Service has worked openly and constructively with its regulators, suppliers and staff to fully understand and fix the problems that caused its CAD system to fail on New Year’s Day.”