NHSmail ‘internal software issue’ causes NHS-wide outage over weekend

The nationally run health service email system NHSmail suffered a complete outage on Saturday which left 1.2 million staff without access to their email accounts.

From lunchtime on Saturday staff were frozen out of the accounts and prompted for their password, with many of them locked out after making multiple login attempts.

The national NHSmail system, managed by Accenture, handles email and messging such as Skype and is certified as secure to handle patient-identifiable data. The service is delivered using a version of the Office 365 platform.

Digital Health News understands staff were left unable to log-in on Saturday following what NHS Digital vaguely termed ‘internal software issues’.

NHS Digital, the agency responsible for the national run email service, blamed the ‘severity 1’ problem on a software issue and said the problem began at midday with access to all accounts being restored by 8pm on Saturday.

The CCIO for health and social care and the senior responsible officer for NHSmail, Dr Simon Eccles, said during the outage: “NHSmail is completely down on all platforms. This is exceptionally unusual. The team and supplier are investigating.”

All back up now. See @NHSDigital for details

— Simon Eccles (@DrSimonEccles) December 1, 2018

In a statement to Digital Health News, NHS Digital said: “The issue was not cyber-related and was caused by issues within the supplier’s internal infrastructure”.

The agency further added: “We are not aware of any patient harm caused by this problem”.

The supplier for the service is Accenture, which delivers NHSmail and associated messaging and collaboration tools using a limited version of Microsoft Office365.

Many in the health service have questioned why the NHS centrally pays the costs of a sub-set of Office365 while leaving trusts to pick up the costs of licensing the rest of Microsoft desktop of server products. The suggestion is that significant efficiency and productivity savings could instead be achieved through a national licensing deal.

In a statement on its website Monday morning NHS Digital said: “We can confirm that all accounts identified as having been locked out as a result of multiple login attempts were unlocked by 10.30pm last night”.

The agency said in a statement: “We apologise for the inconvenience caused and worked to resolve the issue as soon as possible, which was caused by an internal issue with the supplier’s software. A full post-incident review will be conducted to ensure appropriate lessons are learned.”

Earlier versions of the article incorrectly stated that the problem was created by switching on new spoofing controls, NHS Digital say this was not correct and in a ‘clarification’ variously said the problem was either to do with unspecified “internal software issues” or “issues with the supplier’s internal infrastructure”.

Asked to be more specific and why back-ups didn’t kick-in a spokesperson told Digital Health News “It’s far too early to tell, you can’t expect us to know within 24-hours.”

In January 2017, the NHSmail systems was crashed by #replyallgate when a member of staff used an ‘undocumented feature’ to reply-all to everyone on NHSmail, creating 500 million emails in less than two hours.

Did that lady reply all again? #nhsmail

— Kate Doughty (@londongaswoman) December 1, 2018