Enter the CCG loses its RAG

One of the many targets this year for our clinical commissioning group is to identify and reduce inappropriate variation between practices.

But note the word ‘inappropriate’: it’s all too easy to use data too simplistically and to jump to inappropriate conclusions about clinicians’ behaviour and quality.

The trouble is that if people aren’t careful they can be mesmerised by numbers. There is something about figures – especially when expressed to several decimal places – which gives a sense of accuracy and authority to any argument.

Unfortunately, this apparent precision may not be justified: it can hide bad sampling, deficient logic and a poor understanding of the wider context of the situation.

Indeed, the figures on their own may not be sufficient to reach a sensible conclusion — the background to these data may be just as important.

Referral rates

For example, it’s all too easy to assume that a high referral rate implies poor quality primary care, stemming either from clinical incompetence or laziness.

Now think again. A good quality clinician may spot more problems that need hospital treatment, so her referral rate will be high, especially if she is spotting conditions that cannot easily be treated in primary care.

Alternatively, her expertise may allow her to deal with more patients within her practice; in which case her referral rates will be low.

On the other hand, an incompetent GP may not spot all that much, so her referral rates will also be low. Or, alternatively she may diagnose patients correctly, but not know what to do with them and so refer them onwards; making her referral rates high.

The bottom line is this: simple logic suggests there will be no automatic relationship between crude referral rates and the quality of a GP or practice. The mantra of ‘high referral rates, bad; low referral rates, good’ simply doesn’t apply.

Yet how long has it taken the NHS to recognise this? And how many people – politicians, managers and doctors – at heart still believe that it’s true? (In fact the latest research suggests that the highest quality clinicians are likely to refer slightly more than the average.)

Now ask yourself this: is your CCG only concerned about excess referrals? If so, it’s only addressing half the problem.

It needs to understand that a low referral rate may also indicate poor practice – although clearly it doesn’t have to – because some of these clinicians may not be identifying or referring as much as they ought.

I’m delighted to say that our CCG has clearly acquired this balance and always talks about ‘inappropriate variation from the norm’ – in either direction.

In statistics, interpretation is vital

Far from being easy to understand and a near-universal language, statistics can be an extremely dangerous tool.

All aspects of data collection and manipulation need care: these include obtaining a truly representative sample; choosing the right mathematical tool for the job; and above all, interpreting the results wisely.

Context and understanding is key. In the complex and inter-connected world of medicine and healthcare delivery, it isn’t always possible to spot truly inappropriate variations in clinicians or practices without also being aware of the wider pressures and limitations placed upon them.

We all know that poverty can affect healthcare needs, but so will ethnic group, type of practice (urban, rural, inner-city, university), class and cultural expectations; the proximity to casualty; access to laboratory and imaging – all these can have a profound influence on the demands made on practices.

And increasingly in the coming years we will have to factor in inadequate premises, sickness among clinicians from stress, and the decreasing availability of replacement clinicians and locums.

Choice, small numbers and over-simplified conclusions

There is a dreadful tendency in some areas to measure clinicians’ quality according to outcomes which are partially or completely out of the control of the practice. This is often worsened by using over-simplistic indicators, of which the worst is often the RAG or red, amber, green rating.

Consider child immunisation rates. A CCG might decide to rate any practice achieving more than 90% child immunisation as ‘green’, anyone below 70% as ‘red’, with ‘amber’ in the middle. That sounds good, doesn’t it?

Now think about what happens in small practices, where it is quite possible to have just two children in a particular target cohort.

The only way the practice can achieve a ‘green’ rating is by immunising both of them. The practice will be ‘red’ rated if just one of them chooses not to be immunised – even if the clinicians have spent large amounts of time trying to persuade the parents to let their child be immunised. And there’s no exception coding for patient choice.

So pity the practice (often rural) with small numbers of children, some of whom are travellers – because many travellers, by belief, refuse to have their children immunised: and that is their choice, which they are fully entitled to express.

Yet however careful and conscientious that practice is, it risks being ‘red’ rated: in turn it will appear low down in the league tables and will risk being unfairly named, shamed and blamed – and all because one of their patient’s relatives has exercised choice.

At a stroke, this illustrates two vitally important points: the hugely distorting effect of small numbers; and the utter impossibility of assessing practices’ or clinicians’ abilities using measurements that are influenced by patient consent or cooperation.

The bottom line

In the interpretation of statistics, context is key. In my opinion, all interpretation of healthcare statistics needs to involve at least one clinician from the specialty under discussion.

Only someone with this level of detailed experience can understand the complexity of the clinical background and the potential limitations of the inferences that can genuinely be made from the data. I’ll be returning to this theme next month.

Dr John Lockley

Dr John Lockley is clinical lead for informatics at Bedfordshire Clinical Commissioning Group and a part-time GP.