Study highlights need for more rigorous evaluation of digital health apps
A study from Ulster University School of Computing which analysed 1,500 digital health apps, has concluded that user ratings and download rates are unreliable indicators of their quality.
The ‘Don’t judge a book or health app by its cover study‘, published in the PLOS ONE journal in March 2024, examined data from apps provided by the Organisation for the Review of Care and Health Apps (ORCHA).
Researchers assessed the quality of the apps against their user ratings and number of downloads, as part of a PhD programme in 2021-2024.
They explored the level of evidence behind the app, whether it was user-friendly and accessible, and areas including data protection and privacy.
Professor Raymond Bond, professor of human computer systems at Ulster University and chair of the research project, said in a press release published on 3 June, 2024: “Our research underscores the importance of looking beyond metrics like user ratings and the number of app downloads when evaluating digital health apps.
“While these metrics may influence user perceptions, our research shows that they do not necessarily reflect the quality, effectiveness, safety or reliability of these healthcare apps.”
He added: “Relying solely on user ratings and download counts could lead to misguided decisions and, eventually, undermine public trust in digital health technologies”.
The study highlighted the need for users and healthcare providers prescribing the apps to rethink the metrics they use to judge the quality of digital health apps.
It also advocates for the prioritisation of rigorous evaluation processes, so that stakeholders – including healthcare professionals recommending apps to patients – can ensure that digital tools meet high quality standards and make a positive contribution to patient care and wellbeing.
Liz Ashall-Payne, founding CEO of ORCHA, said: “This research emphasises the need for comprehensive evaluation to ensure that digital health apps meet the highest standards of safety, effectiveness and reliability.
“By prioritising rigorous assessments, we can guide healthcare providers and consumers towards better choices, ultimately improving health outcomes and building trust in digital health solutions.”
Professor Maurice Mulvenna, professor of computer science at Ulster University, said: “As the digital health landscape continues to evolve, studies like these serve as a vital reminder that appearances can be deceptive.
“In delving deeper into the substance of digital health apps and by prioritising evidence-based evaluation, we can help consumers to make informed decisions and harness the full potential of technology in advancing health outcomes”.
The study suggests that more could to be done to support users in recognising high-quality health apps, such as improving digital health literacy training and the provision of nationally endorsed ‘libraries’ such as the ORCHA Digital Health Formulary, which launched in March 2023.
