This is the second of 3 posts laying out the philosophical basis for RxISK.org which will be live in the next few weeks. The others are Cri de Coeur & the Unbearable lightness of being.
In Cri de Coeur, I outlined a scenario in which a treatment that causes suicide when put into good trials without any manipulation of the data, any statistical artifice, or any ghostwriting might give rise to a relative risk of suicide that is less than 1.0.
Companies know exactly how to use RCTs to hide risks without any fraud at all
This poses a real problem for anyone who thinks RCTs provide evidence of cause and effect in general or that RCTs are the way to investigate adverse effects. How could a drug that does one thing in real life do exactly the opposite in an RCT?
Given what we now know about antidepressants and suicide, we can construct studies to make suicide on antidepressants appear or disappear. We could produce almost any relative risk between 0.1 and 10.0 (see Heads we win, tails you lose, Psychotic doubt, Cri de coeur).
Companies know exactly how to use RCTs to hide risks without any fraud at all. The surprise is that they got caught out in the case of antidepressants and suicide. There may be other risks they have worked out how to conceal for ever.
We can map out the dynamics in the case of antidepressants and suicide because this problem is now well understood. Comparable scenarios can be constructed for some arrhythmias on some anti-arrhythmics, or respiratory problems following beta agonists given for asthma, and of course everyone believes certain vaccines can cause brain damage but that controlled studies would show a lower incidence of brain damage in the vaccinated group.
Hide deliberately or hide accidentally
In principle companies can deliberately use RCTs to hide problems in every case in which both an illness and its treatment give rise to at least superficially similar problems. Where the problems are not understood the way the antidepressant and suicide issues are, RCTs risk accidentally becoming a means to hide rather than reveal the problem.
Impossible to have confidence, despite any number of confidence intervals
If the adverse event is not well understood, RCT results are impossible to interpret with confidence, despite any number of confidence intervals. We should only say that these are the data that emerged from this particular assay. We should also say that for adverse events it may be a serious mistake to give RCT data primacy over other data.
With adverse events that stem from both an illness and its treatment, the question is what weight to put on observations from controlled trials that have not been designed to investigate the issue versus good observations from clinicians staring the problem in the face, who have an opportunity to investigate the link by means of challenge, dechallenge and rechallenge (CDR) relationships, along with evidence of dose-responsiveness, and reversal by antidote.
We might discount a report from one doctor reporting a patient develop an adverse event on treatment who because of CDR, dose response and other relationships links a drug to the problem. But if a thousand doctors make the link (and even more so if each knows there are 999 other reports) the field will believe the outcome.
Where do we cross the credibility threshold for believing clinical reports like these? The antidepressants and suicide offer a good test case because so few people naturally believe this could happen. It is now clear that the original set of 6 cases from Teicher, Glod and Cole were spot on the mark. The later addition of reports from 5 further centers provided powerful corroboration regardless of what the RCT data might have shown. FDA and everyone else should have gone with these reports as evidence of cause and effect.
But it’s not just FDA who have dug themselves a hole on this one (see Cri de Coeur).
I wrote a version of this post for the Lancet 12 years ago. Before I got the reviews back I had feedback that the Lancet would “buckle” and the article would not be accepted for “political” reasons. (I put the article and its reviews on Healyprozac.com a decade ago).
“completely bizarre” and lacking in “any statistical sense”
The reviews of my paper were in fact longer than the paper itself. The clearest point at which a reviewer lost their cool was when the statistical reviewer was faced with the suggestion that a relative risk greater than 0.5 for a problem in a trial that could stem from both an illness and its treatment might be grounds for concern. He went into orbit, branding this as “completely bizarre” and lacking in “any statistical sense”.
He went on to say that it would be completely unethical to run a trial designed to look at the issue of suicide – or by implication any other hazard. In fact FDA and Lilly had designed just such a trial but dropped it when the public relations heat cooled down.
It is this attitude that delivers Evidence Based Medicine straight into a drug company’s pocket.
Using exactly the same thinking, GlaxoSmithKline’s Ian Hudson argued that even though in scores of cases company employees had categorized a suicide or other problem as caused by their drug, because there was no statistically significant RCT data showing an increased relative risk on the drug these judgments were meaningless (Psychotic doubt).
Our drug killed people retrospectively
In 2006, when GlaxoSmithKline’s data for suicidal acts became statistically significant, did these prior judgments of causality in individual cases magically change from wrong to right?
Now is a time for those supporters of Evidence Based Medicine who spend their time bravely challenging the charlatans of complementary medicine to step up to the plate and sort out this critical problem within orthodox medicine.
They need to say that a relative risk of 1.0 or less can be consistent with a drug causing a problem, or else explain why this is wrong.
The acknowledgement needs to be as specific as this. There are lots of generic statements to this effect in books like Rothman’s Modern Epidemiology but generic statements cut little mustard with the lawyers working for pharmaceutical companies or with doctors in general.
RCTs are not a sacrament
Why would anyone with good intentions fail to step up to the plate?
Well here’s the dilemma. Saying that RCT data from a drug that causes a problem might show a relative risk less than 1.0 concedes that RCTs are not some sacrament that purifies but are rather an assay system and that the results may have little meaning outside the assay. It also entirely undercuts FDA’s current position (see Cri de coeur).
Conceding this point concedes that the results of an RCT may be deeply misleading for variables other than the primary outcome measure (and even on this score may mislead).
There is a way forward even for secondary outcome measures that embraces RCTs – it would call for all RCTs conducted in healthy volunteers to be registered with the data made fully available (see Mystery in Leeds).
Anyone who accepts the overall argument here about the role of RCTs in determining adverse events but remains silent becomes to some extent party to drug induced deaths in people who do not deserve to die, to suicides, violence and inappropriate incarcerations on psychotropic drugs. This happens because your silence is being used every week of the year by drug companies to deny plaintiffs justice and by doctors every day of the year to deny patients recognition.
With SSRIs the suicide data says the risks outweigh the benefits – no one is saved by not warning
There is a second linked problem here. The relative risk of a suicidal act on an SSRI compared to placebo is in fact 2.0 or thereabouts. If this doesn’t mean that SSRIs cause suicide, what does it mean?
The only thing it can mean is that when it comes to suicidal acts the risk of harm in these studies exceeds the likelihood of a benefit. Regulators, doctors and others are reluctant to warn about the risks of antidepressants on the basis that the adverse publicity will mean that some who might benefit from treatment will be deterred from seeking treatment (See Pills and the man).
Regulators and others “feel” that somehow there are more people benefitting from antidepressants than being harmed. This is not an evidence based position.
Before stepping up to the plate, anyone faced with this dilemma can ask two questions, the first of which is how did we get it so badly wrong?
As Daniel Kahneman and others have shown over the years, the simple repetition of mantras like RCTs are the gold standard and case reports are just anecdotes, and “once is never”, produces a sense of familiarity that induces agreement. The Golden Rule of propaganda is that once is never – repetition is all.
It takes a critical effort to pull ourselves out of the hypnosis, to wake up, to stop being good Germans. Academics who cannot recognize propaganda are like salt that has lost its flavor. Proper science should be the antithesis of propaganda, with a Golden Rule when you hear the words the Gold Standard… wake up!
The second question is if RCTs are not the Gold Standard for determining whether a drug causes an adverse event or not, is there an evidence-based alternative?
See The unbearable lightness of being.