April Fool in Harlow: Anecdote Fishing in Harlow

This is the second of a series of six posts that began with Not So Bad Pharma and will continue with The Tragedy of Lou Lasagna, The Empire of Humbug 1, The Empire of Humbug 2 & Brand Fascism.

To celebrate May Fool’s Day last year The Scientist ran an article on Data Diving. This featured the work of Peter Doshi and Tom Jefferson and their efforts to get clinical trial data on Tamiflu from its makers Roche. Data access has become an even bigger saga in the last eleven months with BMJ leading a charge to demand access to trial data.

Created by Billiam James

The other person featured in Data Diving was Robert Gibbons, who was supposedly given unfettered access to patient level data from the trials of Lilly’s Prozac and Wyeth’s Effexor (see Coincidence a fine thing, May Fools’ Day). Just as Doshi and Jefferson’s access overturned a myth – that Tamiflu has a significant clinical effect – so also Gibbons’ access to the “data” according to The Scientist seemingly overturned myths. In this case, the myths that antidepressants don’t really work and that they cause suicide. It was, it turns out, lack of access to the data in the first place that led us to these mistaken beliefs. Companies don’t engage in conspiracies, we are being told, they are masters of the cock-up, and if given a choice of feet to shoot themselves in will opt for both feet. It needs independent academics like Robert Gibbons to wade in and put a stop to their self-injurious behavior.

Gibbons in fact didn’t have access to anything the original authors of the Prozac papers hadn’t already had access to. The original authors were all Lilly personnel with even fuller access than Gibbons was later given. Gibbons also managed to steer clear of other data in the public domain that could be readily accessed that show incontrovertibly that Prozac and Effexor can not only trigger suicide but that the clinical trial data shows that they have led to a net loss of lives – more lives have been lost than saved.

By ignoring relevant accessible data in favor of data no-one else can access, Gibbons has turned the standard access to data argument on its head and made access to data that comes through companies a potential liability for all of us. It’s hard to call the Gibbons version of data access anything less than a perversion.

April Fool

Keeping a straight face, last May I suggested that the give away that this was a May Fool’s Day joke was that very few articles in recent years in the psychiatric or any other literature have received such withering critique as the Gibbons’ articles to which The Scientist refers. The journal’s fact checkers would never have missed this, and by picking such an egregious piece are clearly letting us in on the joke.

The best response, I suggested, was to carry on in joke mode and invite Dr Gibbons to chair an interview panel to recruit academics to whom companies would be prepared to make data available in the manner Lilly and Wyeth had done with him.

What began as a joke took shape later in the year. In contrast to Roche’s resistance to data access, GSK facing a $3 Billion fine fought back with a press release that got them great kudos. Andrew Witty, their CEO, said they were going to make their clinical trial data available. The Gibbons experiment began to look more and more like a trial run for company efforts to manage calls for data access. And there is considerable evidence that academics and FDA explored controlled data access as a way to close down debate on treatment related adverse events (Access to data)

This has now led GSK to set up a portal for researchers to access recent trial data (GSK data access). To do so they will have to submit an analytic protocol to an independent panel (Independent Panel) whose job is to decide if this application for access to the data is going to be sanctioned.

It is of course only responsible to put the proper research governance in place. We don’t want people able to get in and fish around in the data and start making irresponsible claims that will alarm the public. We, in GSK, are completely committed to proper analytic procedures, as advocated in Bad Pharma and we take a leaf out of Bad Science when it comes to concerns about alarming the public as Andrew W did with MMR vaccines. Ah if Andrew Wakefield hadn’t existed we’d have had to invent him.

When does Yes mean No

Here is what’s happening. In 1962, Louis Lasagna introduced the first consent form into clinical trials involving new drugs. It simply asked the patient to sign that they had been told that the drug they were being treated with either was or might be an investigational new drug. One consequence of the thalidomide crisis of 1962 was that FDA made these “informed consent” forms mandatory.

All industry trials now have informed consent forms but their purpose is just the opposite to Lasagna’s original intention. We now have slippery forms that hide the riskiness the original form conveyed. Of greater importance to companies, they tell anyone who reads them that the company will make sure that no-one gets to see their data other than those who are authorized. This may sound good but it means we only show data to the regulator – and only then when we chose to. It’s doubtful if any patients spot that they are consenting to their data being hidden for ever when they sign.

GSK decides America doesn’t exist 

Just as the meaning of consent forms was turned inside out, Andrew Witty’s new proposal turns the meaning of data access inside out. If “fishing” is not allowed, and the only way to look at the data is on the basis of a predetermined protocol, then essentially outside investigators can probably only come up with the same answers that GSK came up with in the first place. Rather than being a well paid inside outsider like Gibbons, anyone applying for the data will be an unpaid outsider and even more valuable to companies accordingly.

There is no proposal to turn the data over to someone like me and let me fish. The data will be retained within GSK while I and others will be able to access the variables we pre-specify – with our every key stroke monitored (for quality control purposes). The data won’t move over to my computer where I might be able to look at it one way this week and another way next week or where it could be combined with other data from other drugs in its class.

Thirteen years ago, a US court mandated me to go into GSK’s healthy volunteer trials archive in Harlow. I was looking for evidence of suicidality in these studies, but what leapt off the page was that these volunteers were having withdrawal problems. Under the new arrangements, there won’t be any pages for anything to leap off.

The US Supreme Court has ruled on this issue in the Matrixx case stating that investors have a right to fish in the data and make their own minds up as to what it shows. The Supreme Court has ruled against the kind of data access on offer now from GSK and other companies.

More generally, as Jack Scannell puts it, Christopher Columbus didn’t reach his pre-specified primary endpoint.  If we take GSK’s recommended approach we would not now accept that America exists.

The plural of anecdote is data

Fishing as any good angler will tell you is an art. It is also critical to science. Just as they turned the meaning of Consent inside out, companies have made Fishing a term of abuse.

When you catch a fish you know what you’ve got. In contrast we don’t know what clinical trials can do to data. For instance, when the SSRI trials began first, it is quite possible that companies didn’t know they could hide suicides and suicidal acts by selecting the right patients.  Every time both a drug and the illness it treats produce the same problem, whether heart attack, homicide or diabetes, clinical trials risk hiding a drug induced problem rather than revealing it.

In just the same way, by repeating the mantra that the plural of anecdote is not data, as Dee Mangin spotted, companies have obscured the fact that the original phrase was “The plural of anecdote is data”.

If the plural of Anecdote were not Data, Google and the entire Big Data movement would be out of business. This is a good example of how conflicts of interest can be helpful – it was clearly critical for the currently intensely fashionable Big Data movement to demolish the idea that anecdotes are not useful by discovering the original version of the quotation.

From a Big Data point of view, Andrew Witty’s Data Access proposals look like efforts by the Chinese government to censor the Internet. But for his efforts, Andrew is portrayed by BMJ as the candidate of Hope and Ben Goldacre has apparently described his offer as a “cartwheel” moment.

End of the Line

What happened next in the Gibbons story is a cautionary tale. Several sets of academics wrote to Archives pointing out the lethal flaws in the Gibbons’ papers that ranged from inappropriate data inclusion to incompetent mathematical calculations to biased interpretations. Letters like this would once have been published in a good journal – science depends on robust debate about what the data shows. Without debate, error is propagated and the self-correcting function of science is disabled, as Barney Carroll has said. But Archives refused to print.

Archives were forced to back down – perhaps only because it didn’t seem to be a good idea to draw too much attention to what was going on.  In future a journal editor would have a perfect defense against publishing letters from any critics of industry. Why publish when on the face of it, Gibbons or Witty or Wakefield are claiming access to data that their critics don’t have?   Of course, science depends on debate about data – but it does not depend on publishing the rants of people who are unhappy about something but have not analyzed the data. From an editor’s point of view allowing letters that were not based on the data seems to be tackling the man (Gibbons, Witty or Wakefield)  rather than the ball.

Tackling the man is not cricket.  But provided you play the ball all is fair, even eleven lined up against one as in cricket – the English epitome of fair-play.

Illustration: Glaxo Buys Open Science (PDF), copyleft 2013 Billiam James


RxISK: Research and report prescription drug side effects on RxISK.org.

Search. Report. Contribute.


You and your meds. Give the real story. Get the real story.

Pharmageddon

Pharmaceutical companies have hijacked healthcare in America, and the results are life-threatening.

 

Dr. David Healy documents a riveting and terrifying story that affects us all.

 

University of California Press (2012)

 

Available on Amazon.com

 

Comments

  1. Johanna says:

    Billiam James’ poster is a thing of beauty and should be plastered to the walls of every medical school – not to mention the counters of every drugstore where you and I line up to buy GSK’s latest products! It reminded me of something the great Clarence Darrow wrote in the introduction to his pamphlet Crime and Criminals, about using information “responsibly.” (Darrow, a famous criminal lawyer, believed that massive social inequality, not poor folks’ wickedness, was at the root of most crime):

    “This address is a stenographic report of a talk made to the prisoners in the Chicago jail. Some of my good friends have insisted that while my theories are true, I should not have given them to the inmates of a jail.

    “Realizing the force of the suggestion that the truth should not be spoken to all people, I have caused these remarks to be printed on rather good paper and in a somewhat expensive form. In this way the truth does not become cheap and vulgar, and is only placed before those whose intelligence and affluence will prevent their being influenced by it.”

    • Thanks Johanna! Your reference to Clarence Darrow is spot on! Because like the prison masters of his day, GSK and the Regulators agree with the theory of open science, but in practice argue that it is inappropriate to share the truth (all clinical trial data) with the prisoners (the patients!).

  2. GSK’s position on data sharing reminds me of President Nixon’s position on sharing the White House secret tape recordings (which were made without informed consent). Congress demanded the tapes. Nixon refused for as long as possible, then handed over a thoroughly bowdlerized set of *transcripts* of the tapes instead of the actual tapes. Nixon won the battle but lost the war.

    GSK will never willingly allow free access to trial data. Nor will any other company of that size. Legislation is needed, and even then, we have to hope large amounts of data don’t inexplicably go “missing.”

  3. David_Healy says:

    Quite extra-ordinarily (close to spooky) at much the same time as this post went up, the following comment was posted on Not So Bad Pharma. This is a cross-posting. There is a follow-up comment on Not So Bad Pharma.

    From Clifford Miller:

    Hello David,

    This is one of those instances where a correction is likely to be most welcomed by the corrected.

    Enjoy:

    The correction original quotation is “the plural of anecdote is data”.

    I will not bore you with the details but that is also evidentially correct and it is based on the identical mechanism we use for proof in science [and in rechallenge and in dechallenge – which you know all about].

    So where is the correct quote recorded?

    ‘Raymond Wolfinger’s brilliant aphorism “the plural of anecdote is data” never inspired a better or more skilled researcher’ Nelson W. Polsby PS, Vol. 17, No. 4. (Autumn, 1984), pp. 778-781. Pg. > 779.

    This is confirmed in an exchange of emails between Wolfinger and Fred Shapiro, the Editor of The Yale Book of Quotations, The Oxford Dictionary of American Legal Quotations, and others in a linguistlist.org listserv post by Shapiro – you can click on the listserv link and look it up yourself:-

    http://listserv.linguistlist.org/cgi-bin/wa?A2=ind0407a&L=ads-l&P=8874

    Subject: Re: “Plural of anecdote is data” (Ray Wolfinger)
    From: Fred Shapiro
    Reply-To: American Dialect Society
    Date: Tue, 6 Jul 2004 23:21:27 -0400
    Content-Type: TEXT/PLAIN
    Parts/Attachments:
    Parts/Attachments TEXT/PLAIN (34 lines)

    On Tue, 6 Jul 2004 [log in to unmask] wrote:

    > Nelson W. Polsby PS, Vol. 17, No. 4. (Autumn, 1984), pp. 778-781. Pg.
    > 779: Raymond Wolfinger’s brilliant aphorism “the plural of anecdote is
    > data” never inspired a better or more skilled researcher.

    I e-mailed Wolfinger last year and got the following response from him:

    “I said ‘The plural of anecdote is data’ some time in the 1969-70 academic
    year while teaching a graduate seminar at Stanford. The occasion was a
    student’s dismissal of a simple factual statement–by another student or
    me–as a mere anecdote. The quotation was my rejoinder.
    Since then I have missed few opportunities to quote myself. The only
    appearance in print that I can remember is Nelson Polsby’s accurate
    quotation and attribution in an article in PS: Political Science and
    Politics in 1993; I believe it was in the first issue of the year.”

    I also e-mailed Polsby, who didn’t know of any early printed occurrences.

    What is interesting about this saying is that it seems to have morphed
    into its opposite — “Data is not the plural of anecdote” — in some
    people’s minds. Mark Mandel used it in this opposite sense in a private
    e-mail to me, for example.

    Fred Shapiro

    ————————————————————————–
    Fred R. Shapiro Editor
    Associate Librarian for Collections and YALE DICTIONARY OF QUOTATIONS
    Access and Lecturer in Legal Research Yale University Press,
    Yale Law School forthcoming
    e-mail: [log in to unmask] http://quotationdictionary.com

  4. David,

    Oops – posted this over on “Not So Bad Pharma” in relation to “anecdotal” but guess it should go here – [as your post above indicates]:

    “If you have not already seen it, you might enjoy this paper by a clinical epidemiologist of good standing:

    “In Defense of Case Reports and Case Series” – Jan P. Vandenbroucke, MD, PhD

    http://www.epidemiology.ch/history/PDF%20bg/Vandenbroucke%20JP%202001%20in%20defense%20of%20case%20reports.pdf

    And why should we need to call any kind of epidemiologist a “clinical” one? Surely epidemiology is dependent upon evidence from individual clinical cases to make the assessment necessary to establish causation – in other words – from case series. All else is merely statistics. Medicine seems to have lost sight of how causation is established in complex biological systems – and in epidemiology.

    Rechallenge is the most extreme example of case series – a case series of one. So if someone tells you about a rechallenge, EBMers say we should place no store on it as “anecdotal” – but if it is reported in writing and well documented – it is not. So where is the logic of that? The only difference is of credibility for third parties – about the reliability of the report – and if for an anecdotal report one can establish credibility/reliability then the anecdotal report is just as much proof as the written one.

    So when a multiplicity of reports arise all telling much the same story independently – a case series – we cannot ignore it – especially because regularity is being demonstrated – the most essential element of any form of scientific evidence.

    So all those reports of individuals who claimed they could not come off Seroxat/Paxil are more than mere data – that is case series evidence of addictiveness – and MHRA just ignored that “inconvenient” evidence for years.

    “Anecdotal” evidence covers all human observations, scientific and otherwise. If a “scientist” tells you what he has observed – that we are told by EBMers is “anecdotal” and to be dismissed – but if it is written down in his lab book it is not. So the only difference is one of belief in the reliability of the account, so if the reliability of the account can be established then there is no reason not to consider “anecdotal” evidence in medicine.

    If all science were to be based upon “I cannot be bothered to check, so I am going to ignore it”, as EBM appears to be with its evidentially peculiar and sadly misconceived hierarchy of evidence – there would be no science.

    If C.T.R. Wilson was not bothered to follow up on and ignored those odd unexpected tracks in the saturated vapour in his weather experiment – one wholly unrelated to particle physics – we may not have had the invention of the Cloud Chamber – a bit of a breakthrough in particle physics.

    So how much are we missing in medicine as a result of EBMer blindness? A great deal.”

  5. Vandenbroucke’s article ought to be mandatory reading for all practicing physicians…

  6. Typical “smoke and mirrors” crap from GlaxoSmithKline…

  7. Jennifer says:

    Have discoveries made in quantum physics generated any dialogue or insights within the practice of evidence-based medicine? I perhaps unnaturally assumed that the discovery of the impact of the observer on the observed might introduce caution with regard to interpreting any scientific data, including medicine trials where, as you note, the illness and the drug may produce the same effects.

    We tend to find what we are looking for. All of us. Which reveals another collusion. Not only do drug companies and unwitting doctors find what they are looking for, but so do those of us, individuals and families, who are desperately seeking quick fixes for our pain and evidence that promises these fixes exist.

    Unfortunately, it’s hard to gather and share anecdotal evidence in 7 minutes of patient contact time…

Leave a Comment

*