Joe’s view of SUSing out IT systems

This time last year, the National Mental Health Informatics Network launched the first ever National Electronic Patient Record Usability Survey.

Much of the donkey work was done by my trust’s chief information officer, Darren McKenna who, as of this week, has replaced me as chair of the network.

We shared the results within the network at our annual meeting EHI Live 2013 last November. There was considerable interest, even though the survey was an amateur effort in the best sense of the word.

After all, we are by no means experts at running national surveys; we had no budget, no experience, and could only work on it in our spare time.

We publicised the survey via eHealth Insider, The Royal College of Psychiatrists’ Twitter feed, through our NHS Networks website, and via our individual social media accounts to try to reach as wide an audience as we could.

And we used the System User Survey, an industry standard software usability scale which gives a system a score of between 0 and 100.

Drum roll for the results

Overall, we gained a gratifying 1,200 responses from 54 of what was, at the time, 56 mental health trusts.

We also obtained usability data for 22 systems currently deployed in mental health settings. So, here are the anonymised results:

We learned that usability ratings varied with job role, with medics most likely to rate systems badly. Perhaps that makes them the most useful audience for system suppliers seeking to improve their product?

We also know that adding a mobile solution to a product will add about ten points to a usability score. People love a technology that allows them to access records out of hours without going into the office.

Why the anonymity?

So why have we anonymised the results? Well there are several reasons. Firstly, we are an organisation with no significant funding and no legal services department. In addition to which, we all have full time day jobs, which would leave us in a poor position to fight any legal challenge.

Secondly, the number of responses varied across systems from a handful to hundreds, rendering statistically valid comparisons difficult.

That particularly applies to those systems where a small number of responses were recorded. They may have been biased by a handful of either vitriolic or overenthusiastic reviews.

Thirdly, we had no methods in place to prevent gaming. So, in theory, suppliers might have been able to inflate their usability scores; although this seems unlikely, given the relatively low scores.

Or, again in theory, suppliers might have been able to invest time and energy in doing down their rivals by submitting bad reviews.

Again, this seems unlikely, given the pattern of results; but it is something that has been alleged in connection with TripAdvisor and other review sites.

Doing it all again – someone should

Bearing these caveats in mind, and being familiar with many of the systems, the results do appear to have some face validity.

A particularly interesting finding is that one system can have very different usability scores from one trust to another.

This suggests that usability isn’t all about the system; it’s also about hardware, clinical engagement, local IT capability – and even general staff morale. The IT system can act as a lightning rod for criticism at an unhappy trust.

Will we run the Great Big Mental Health System User Survey again this year? Maybe. But we’d have to learn the lessons of our first “amateur” effort.

We’d have to make it even bigger to iron out the statistical problems of small samples for some systems. We would have to have a mechanism to prevent gaming (NHSmail accounts only?)

Maybe we could buy enough lottery tickets to win big and give us deep enough pockets to “publish and be damned”! Or maybe the suppliers could agree that this opening up of the market to massive end user feedback could be good for everyone; and agree not to take legal issue with us.

Maybe this kind survey could be bundled into the Clinical Digital Maturity Index that EHI Intelligence has developed and NHS England is supporting, to add richness to the assessment of NHS IT capability?

Some or all of these things would be good. Because I suspect that if we don’t engage in this kind of feedback the current gap in the market for truly excellent and elegantly usable systems will remain…

Joe McDonald

Joe McDonald is a practising NHS consultant psychiatrist. Over the past five years he has been an NHS trust medical director and national clinical lead for IT at NHS Connecting for Health – a stint that included 18 months as medical director of the Lorenzo delivery team!

His experiences in the National Programme for IT in the NHS have left him with a passion for usability and "end user knowledge networks.” He is the founding chairman of the National Mental Health Informatics Network. Motto: we don't get fooled again. Follow him on twitter @CompareSoftware