Exclusive: 500m emails flood NHS during #replyallgate

The NHSmail “reply-all” error created 500 million emails in less than two hour, nearly crashing a system relied on by 1.2 million NHS staff.

A NHS Digital report, seen by Digital Health News, revealed that a “software configuration error” was to blame for what became known as “#replyallgate” by tens of thousands of frustrated NHS staff.

On 14 November, 2016, a Croydon NHS IT consultant sent a group email, using a Dynamic Distribution List, to what she thought was less than 20 people and went home for the afternoon.

Instead, the software error caused the email to be sent to every NHSmail account in England, about 840,000 accounts. The resulting tens of millions of reply-all emails clogged up the entire NHSmail system, leaving some people unable to access their account for days.

The NHS Digital report shows that in a normal day three to five million emails are sent on NHSmail.

On 14 November, 500 million emails were sent in one hour and 15 minutes, roughly the equivalent to what is normally sent over three months.

“Although the service did not crash at any point, the traffic queues that had already built in the system by 09:45 resulted in significant service delays for the majority of the day.”

The error was not visible to any of the users, all of whom would have thought they were replying to a small group. The report does not ascribe any blame to the sender.

“It is very important to note that the individual who created the distribution list was not at fault, and nor were the users who unwittingly used “reply to all”.”

Once NHS Digital detected the spike in emails, the ability to “reply-all” was disabled across the service and the faulty distribution list deleted.

In 2015, Accenture won a £60 million contract to move NHSmail off its existing “burning platform” onto a new service. The switch to ‘NHSmail 2’ was made last year, with about 1.4 million accounts now running on the new system.

The NHS Digital report said when Accenture was contracted there were “clear and strict” design controls requiring a limit on the volume of emails a user could send in one instance.

These limits were never put in place and the “functionality is still to be delivered by Accenture”.

In the interim, NHSmail has disabled the ability for users to create Dynamic Distribution Lists. Until Accenture has found a way to limit the scope of these lists, any user seeking to create one will need to contact the company.

“We are continuing to work with Accenture and with Microsoft to review and enhance where necessary all areas of the service, which successfully supports more than 35,000 distribution lists, to ensure we are satisfied such an incident as described by this paper does not recur.”