Software ‘freeze’ after network failure at St George’s

A south London trust has "frozen" all software deployment as continues to recover from trust-wide IT crash in June.

St George’s University Hospitals NHS Foundation Trust suffered a ‘network failure’ that shut down computer systems for several hours on 6 June, which was further exacerbated when the disaster recovery system did not work.

However, the crash also appears to have exposed a range of IT issues in need of improvement to prevent future failures, including “inadequately supported” software such as XP, ageing computers, and insufficient data storage capacity.

In a report to the trust’s board this month, chief information officer Larry Murphy said the trust was still working on “stabilisation and recovery of ICT” but a new long-term solution was needed to reduce the risk of future failure.

Deployment of new clinical systems, such as the roll-out of Cerner Millennium to the neo-natal unit, had also been “frozen” until the underling IT infrastructure was no longer an “unstable platform”.

“In reality this freeze is a formal recognition of what has already happened as the main programmes are neither properly funded or resourced.”

During the crash, the 724 disaster recovery system supplied by Cerner, which is meant to back up clinical notes on local PCs in the event of network failure, did not work. Even four months later, the trust remained unable to fully back-up its data in the event of another crash, he said.

“One of the risks relates to back-up, that is; the current inability to fully backup trust data.”

The lack of a back-up did not affect the trust’s core clinical system, RIO and Cerner, which are hosted offsite, but did cover emails.

“Emails should not contain any patient data; however, a cautious approach is being taken on the assumption that this may not be the case.”

In the short-term, the trust will use the Microsoft Azure cloud to back-up its email Exchange.

Other specific risks to the trust’s IT infrastructure included:

* A lack of computer processing capacity. This was expected to partially improved with extra capacity being purchased this month.

* Not enough computer storage, which is expected to be partially improved by November.

* Nearly 2,000 computers still running on Windows XP, 750 of which are currently being replaced, with the rest expected to be phased out by the end of the financial year.

* Network vulnerability, with “significant reconfiguration work” in progress that should partially reduce the risk by January.

In a statement, a St George’s spokeswoman said the data "replication errors" meant the emergency back-up system did not work during the failure. However, the network was only down for a few hours and no patient data was lost.

"There have been no network failures since the network problems in June. However, we have investigated why the 724 back up-system failed, and have worked with Cerner to rectify any faults, which has meant re-installing and fully testing the system to ensure it will meet our needs in the event of another system failure."

The St George’s report comes amid increasingly central scrutiny of obsolete technology, particularly around Windows XP. The operating system has not been supported since April 2014, but remains in widespread use within the NHS.

The third Caldicott and Care Quality Commission reports into data security, released in July, both singled out old IT systems as a cyber security risk.

NHS Digital has since been working with trusts to review their unsupported and potentially vulnerable systems. 

As well as XP, Murphy said in his report that a number of other systems, such as document manager and e-triage, were “inadequately support and acting unpredictably”.

Overall a strategy would be needed to not only stabilise the IT system but make it fit for future improvements, and the trust was looking for a “strategic business partner” to help develop that strategy by December.

“Without a strategic direction the ICT department will continue to implement short term tactical repairs that will ultimately cost more, deliver less benefit and continue to drive poor user experience.”

The trust has been contacted for comment.