Editorial Note: This post continues a sequence that began with Cochrane Cock-up. It centers on treatment efficacy. A third post will center on safety where Cochrane has failed even more comprehensively than on efficacy. A fourth will feature some of the protagonists.
The Cochrane Collaboration took shape around two foundational ideas. One was that randomized controlled trials (RCTs) offered a gold standard in medical evidence.
The other was the idea of a systematic review. Where RCTs had been conducted, compiling them would give the best possible fix on a treatment’s effects. Doing this involved winnowing down what might be up to 100 publications to the 2 or 3 bedrock trials from which they came and collecting all the trials that had not been published.
This seemed like a good idea when Cochrane was established in 1993 and close to 40,000 clinicians and other researchers have apparently been involved since in compiling Cochrane Reviews. Many believed they were engaged in an exercise that would help contain the pharmaceutical industry.
Flawed or Fraud?
While the idea of winnowing publications made sense and led to the discovery that one industry trial might give rise to 50 different publications, and the idea of chasing unpublished trials was also good, most of the other Cochrane ideas were fundamentally flawed from the get-go.
Meta-analysing industry or any other trials might appear to make sense but it doesn’t actually hold water. If RCTs don’t consistently give the same effectiveness result they are irredeemably flawed. The flaw might lie in the heterogeneous nature of the population being studied or it might lie in the heterogeneous effects of the drug being studied but either way if the results aren’t the same every time the claim that any finding is statistically significant doesn’t hold. The term statistical significance means – you will get the same result every time saving for the occasional intervention of chance.
Another way that Ronald Fisher, the creator of the RCT and the concept of statistical significance, put it was that when RCTs get the same result every time it demonstrates that we know what we are doing and what to recommend. If we don’t get the same result every time we do not know what we are doing and no amount of meta-analysing the data can put this right.
Goodness only knows what Fisher would have made of the trials of antidepressant.
Not everybody views trials the way Fisher did. Others are Neyman-Pearsonites; this approach, developed from studies measuring stars, works fine if you’re investigating stars. It was never designed for and doesn’t work for people exposed to medical treatments.
Finding that trials yield inconsistent results should result in a statement that we don’t know what we are doing – clinicians and the public should beware any claims to the contrary. Inconsistent trials should not result in NICE guidelines mandating treatment approaches from HPV vaccines to antidepressants – especially when the inconsistencies are based on surrogate outcomes, from scores on a Depression Rating Scale to bone densities, rather than outcomes that count to patients.
If it weren’t for Cochrane nobody would have to make blindingly obvious statements like this.
Many engaged with Cochrane, thinking it was a way to contain the pharmaceutical industry. When it began, Cochrane had a chance to bring industry to heel by saying it would only include trials and treatments where the data was available in its reviews. The Collaboration was in a better place to force industry to engage with science than anyone else was. It blew this opportunity then and has continued to sell its birthright ever since.
Cochrane helped create the Evidence Based Medicine (EBM) bandwagon. When those working in healthcare or receiving healthcare hear about EBM, most of them hear “Data Based Medicine”. But this is not what they are getting. Cochrane at present is one of the biggest obstacles there is to people getting Data Based Medicine.
Journals from the BMJ to the NEJM know this, as do academics – many on some of the listserves to which this post will go – but they nevertheless extol Cochrane and mislead others into thinking Cochrane is independent – as it would be if it had access to the data from trials.
The evidence that pretty well all the clinical trial literature on on-patent pharmaceuticals is ghostwritten has been around since 2000. Senior figures in Cochrane and others who go about meta-analysing trials know this but have ignored it.
Cochrane make a great deal of noise about applying a range of indicators to publications – from clarity of randomization procedures to conflict of interest statements – in an effort to appear rigorous or methodical. Peter Goetzsche was one of those who contributed most to the development of a range of instruments of this sort. These methods may fool some into thinking industry trials have been appropriately sterilised and can be used but rather than advancing science, they have a primary effect of excluding the laity.
In multiplying the tick-boxes around trials, these processes have created a strange Alice in Wonderland type of needle’s eye through which industry camels can swagger to publication in NEJM, JAMA or the Lancet or get into a Cochrane HPV vaccine review, but anyone attempting to report a significant adverse effect on a drug or a vaccine finds it impossible to get published.
The one thing about ghosts is that they ensure publications come with all boxes and quality indicators ticked in a way that the average clinician or researcher cannot hope to emulate.
Fake news has been in the news for the last two years. There is nowhere the news has been so Fake for the last two decades as in the medical literature and Cochrane have played a key role in ensuring this was the case.
Quite recently Tom Jefferson became one of the first members of Cochrane to publicly acknowledge this – suggesting that reviewers should be working from Company Clinical Study Reports (CSRs), which tick no quality indicators, rather than ghostwritten articles. The recent HPV review appears to have worked from ghostwritten publications rather than CSRs.
But it has to be a toss-up as to whether ghostwritten articles designed to deceive academics or company authored reports, which are prepared with sleep-at-the-wheel reviewers within the regulatory apparatus with a brief to partner industry in mind, are likely to be the more reliable. What is needed is access to the data – not ghostwritten articles or CSRs
The HealthCare Games
Several decades ago within mental health, anyone going to a doctor would have met someone playing a Freudian or analytic game. We can now see the game since Freud has been as discredited as Christian clerics (another game).
You presented yourself and found you had to engage in a game where the other person held the rule-book. At best you might play to stalemate; you could never win. You were in a game where everything you said was automatically discredited – you wouldn’t be saying it if you were okay.
RCTs and Cochrane are a similar game. Everything doctors and patients say or see or sense that leads to the administration of a drug or in response to what appear to be the effects of this drug is wrong. We who have done clinical trials know what the true effects are. We wouldn’t be doing trials if we could depend on what you saw or said or sensed. This exercise is as flawed as the Freudian one was before it or the clerical games we have played.
Cochrane Inc. began just after Archie Cochrane died in 1989. For Archie C, trials were a means to combat medical arrogance and therapeutic bandwagons – such as coronary care units when these began in the 1960s. They were not value neutral.
Archie figured the money in the NHS should go into rehabilitation facilities and a greater use of placebo options in therapy. Running trials to debunk the claims of high cost medical care was a way to ensure this happened. He would almost certainly have favored Papp smear screening programs over HPV vaccines.
Immediately after his death, RCTs in the hands of the Cochrane Collaboration, were reborn as value neutral. There was no longer any need for doctors or others in healthcare to bother their pretty little heads (the men that is) about what healthcare should look like. Their job was follow the trials and apply what worked and healthcare would take care of itself.
The Collaboration and others sold this as Evidence Based Medicine rather than Eminence Based Medicine, a version of speaking truth to power. The message was trust us not them.
There was one value that might have given this plea some validity – if the Collaboration had made clear that it valued, was passionate about, data. Value neutrality hands everything over to the most powerful player in the arena.
What the HPV vaccine story reveals is that Cochrane can’t be trusted. RCTs and Systematic Reviews have become the fuel of therapeutic bandwagons rather than a means to derail these bandwagons.
In terms of therapeutic bandwagons it is difficult to imagine a greater mismatch between the use of a man’s name, Archie Cochrane, and an enterprise, like Cochrane Inc, than this – it’s on the scale of a Mahatma Gandhi center for ethnic cleansing if there is one somewhere.
Imagining Cochrane, the man, living with a lack of access to data or having to base his conclusions on ghostwritten articles doesn’t compute.