April Fool in Harlow: Anecdote Fishing in Harlow

This is the second of a series of six posts that began with Not So Bad Pharma and will continue with The Tragedy of Lou Lasagna, The Empire of Humbug 1, The Empire of Humbug 2 & Brand Fascism.

To celebrate May Fool’s Day last year The Scientist ran an article on Data Diving. This featured the work of Peter Doshi and Tom Jefferson and their efforts to get clinical trial data on Tamiflu from its makers Roche. Data access has become an even bigger saga in the last eleven months with BMJ leading a charge to demand access to trial data.

The other person featured in Data Diving was Robert Gibbons, who was supposedly given unfettered access to patient level data from the trials of Lilly’s Prozac and Wyeth’s Effexor (see Coincidence a fine thing, May Fools’ Day). Just as Doshi and Jefferson’s access overturned a myth – that Tamiflu has a significant clinical effect – so also Gibbons’ access to the “data” according to The Scientist seemingly overturned myths. In this case, the myths that antidepressants don’t really work and that they cause suicide. It was, it turns out, lack of access to the data in the first place that led us to these mistaken beliefs. Companies don’t engage in conspiracies, we are being told, they are masters of the cock-up, and if given a choice of feet to shoot themselves in will opt for both feet. It needs independent academics like Robert Gibbons to wade in and put a stop to their self-injurious behavior.

Gibbons in fact didn’t have access to anything the original authors of the Prozac papers hadn’t already had access to. The original authors were all Lilly personnel with even fuller access than Gibbons was later given. Gibbons also managed to steer clear of other data in the public domain that could be readily accessed that show incontrovertibly that Prozac and Effexor can not only trigger suicide but that the clinical trial data shows that they have led to a net loss of lives – more lives have been lost than saved.

By ignoring relevant accessible data in favor of data no-one else can access, Gibbons has turned the standard access to data argument on its head and made access to data that comes through companies a potential liability for all of us. It’s hard to call the Gibbons version of data access anything less than a perversion.

April fool

Keeping a straight face, last May I suggested that the give away that this was a May Fool’s Day joke was that very few articles in recent years in the psychiatric or any other literature have received such withering critique as the Gibbons’ articles to which The Scientist refers. The journal’s fact checkers would never have missed this, and by picking such an egregious piece are clearly letting us in on the joke.

The best response, I suggested, was to carry on in joke mode and invite Dr Gibbons to chair an interview panel to recruit academics to whom companies would be prepared to make data available in the manner Lilly and Wyeth had done with him.

What began as a joke took shape later in the year. In contrast to Roche’s resistance to data access, GSK facing a $3 Billion fine fought back with a press release that got them great kudos. Andrew Witty, their CEO, said they were going to make their clinical trial data available. The Gibbons experiment began to look more and more like a trial run for company efforts to manage calls for data access. And there is considerable evidence that academics and FDA explored controlled data access as a way to close down debate on treatment related adverse events. (GlaxoSmithKline and Access to Data)

This has now led GSK to set up a portal for researchers to access recent trial data (GSK data access). To do so they will have to submit an analytic protocol to an independent panel (Independent Panel) whose job is to decide if this application for access to the data is going to be sanctioned.

It is of course only responsible to put the proper research governance in place. We don’t want people able to get in and fish around in the data and start making irresponsible claims that will alarm the public. We, in GSK, are completely committed to proper analytic procedures, as advocated in Bad Pharma and we take a leaf out of Bad Science when it comes to concerns about alarming the public as Andrew W did with MMR vaccines. Ah if Andrew Wakefield hadn’t existed we’d have had to invent him.

When does Yes mean No

Here is what’s happening. In 1962, Louis Lasagna introduced the first consent form into clinical trials involving new drugs. It simply asked the patient to sign that they had been told that the drug they were being treated with either was or might be an investigational new drug. One consequence of the thalidomide crisis of 1962 was that FDA made these “informed consent” forms mandatory.

All industry trials now have informed consent forms but their purpose is just the opposite to Lasagna’s original intention. We now have slippery forms that hide the riskiness the original form conveyed. Of greater importance to companies, they tell anyone who reads them that the company will make sure that no-one gets to see their data other than those who are authorized. This may sound good but it means we only show data to the regulator – and only then when we chose to. It’s doubtful if any patients spot that they are consenting to their data being hidden for ever when they sign.

GSK decides America doesn’t exist

Just as the meaning of consent forms was turned inside out, Andrew Witty’s new proposal turns the meaning of data access inside out. If “fishing” is not allowed, and the only way to look at the data is on the basis of a predetermined protocol, then essentially outside investigators can probably only come up with the same answers that GSK came up with in the first place. Rather than being a well paid inside outsider like Gibbons, anyone applying for the data will be an unpaid outsider and even more valuable to companies accordingly.

There is no proposal to turn the data over to someone like me and let me fish. The data will be retained within GSK while I and others will be able to access the variables we pre-specify – with our every key stroke monitored (for quality control purposes). The data won’t move over to my computer where I might be able to look at it one way this week and another way next week or where it could be combined with other data from other drugs in its class.

Thirteen years ago, a US court mandated me to go into GSK’s healthy volunteer trials archive in Harlow. I was looking for evidence of suicidality in these studies, but what leapt off the page was that these volunteers were having withdrawal problems. Under the new arrangements, there won’t be any pages for anything to leap off.

The US Supreme Court has ruled on this issue in the Matrixx case stating that investors have a right to fish in the data and make their own minds up as to what it shows. The Supreme Court has ruled against the kind of data access on offer now from GSK and other companies.

More generally, as Jack Scannell puts it, Christopher Columbus didn’t reach his pre-specified primary endpoint.  If we take GSK’s recommended approach we would not now accept that America exists.

The plural of anecdote is data

Fishing as any good angler will tell you is an art. It is also critical to science. Just as they turned the meaning of Consent inside out, companies have made Fishing a term of abuse.

When you catch a fish you know what you’ve got. In contrast we don’t know what clinical trials can do to data. For instance, when the SSRI trials began first, it is quite possible that companies didn’t know they could hide suicides and suicidal acts by selecting the right patients.  Every time both a drug and the illness it treats produce the same problem, whether heart attack, homicide or diabetes, clinical trials risk hiding a drug induced problem rather than revealing it.

In just the same way, by repeating the mantra that the plural of anecdote is not data, as Dee Mangin spotted, companies have obscured the fact that the original phrase was “The plural of anecdote is data”.

If the plural of Anecdote were not Data, Google and the entire Big Data movement would be out of business. This is a good example of how conflicts of interest can be helpful – it was clearly critical for the currently intensely fashionable Big Data movement to demolish the idea that anecdotes are not useful by discovering the original version of the quotation.

From a Big Data point of view, Andrew Witty’s Data Access proposals look like efforts by the Chinese government to censor the Internet. But for his efforts, Andrew is portrayed by BMJ as the candidate of Hope and Ben Goldacre has apparently described his offer as a “cartwheel” moment.

End of the line

What happened next in the Gibbons story is a cautionary tale. Several sets of academics wrote to Archives pointing out the lethal flaws in the Gibbons’ papers that ranged from inappropriate data inclusion to incompetent mathematical calculations to biased interpretations. Letters like this would once have been published in a good journal – science depends on robust debate about what the data shows. Without debate, error is propagated and the self-correcting function of science is disabled, as Barney Carroll has said. But Archives refused to print.

Archives were forced to back down – perhaps only because it didn’t seem to be a good idea to draw too much attention to what was going on.  In future a journal editor would have a perfect defense against publishing letters from any critics of industry. Why publish when on the face of it, Gibbons or Witty or Wakefield are claiming access to data that their critics don’t have? Of course, science depends on debate about data – but it does not depend on publishing the rants of people who are unhappy about something but have not analyzed the data. From an editor’s point of view allowing letters that were not based on the data seems to be tackling the man (Gibbons, Witty or Wakefield) rather than the ball.

Tackling the man is not cricket. But provided you play the ball all is fair, even eleven lined up against one as in cricket – the English epitome of fair-play.

Illustration: Glaxo Buys Open Science (PDF), Copyleft 2013 Billiam James