Editorial Note: This post offers our response to the Keller letter. It was published in BMJ in early February. Again all material is available on Study329.org. The response does not make much of the fact that not all of the Als in Keller et al have lined up to sign their letter. One of the intriguing points in all this is tracking the line taken by GSK and by Keller and colleagues.
Jon N Jureidini, David Healy, Mickey Nardo, Melissa Raven, Elia Abi Jaoude, Catalin Tufanaru, Joanna Le Noury
Re: Restoring Study 329: Response to Keller
The response by Keller and selected colleagues [1] to our Restoring Study 329 article alleges three overarching faults[2]: bias and lack of blind ratings in relation to harms; lack of detailed methodology; and failure to consider the available methodological knowledge regarding paediatric depression from twenty-four years ago. Regarding the second issue, there is in fact a detailed explanation of all the methods in our paper and its RIAT Audit Record (appendix 1). We tackle the first and third issues below.
Efficacy
While there was uncertainty twenty-four years ago about the appropriate rating scale to use in pediatric depression trials, there were serious methodological problems in the conduct and reporting of Study 329 that have nothing to do with that uncertainty. Instead, in their reporting of efficacy in Study 329[3], and their defence of it, Keller and colleagues have asked that the field suspend many widely held tenets about clinical trial analysis, by asking us to do the following:
- accept that the a priori protocol is not binding, and that changes can be made to the outcome variables while the study is ongoing, without amending the protocol with the IRB or documenting the rationale for the change
- ignore the requirement to correct the threshold of significance for the analysis of multiple variables
- ignore the requirement that when there are more than two groups, preliminary omnibus statistical analysis needs to be done prior to making any pairwise comparisons between groups – an integral part of the ANOVA analysis declared in the Study 329 protocol
- allow the parametric analysis of rank-order, ordinal rating scales [CGI, HAM-D and K-SADS-L Depressed Mood Items] rather than the expected non-parametric methods specifically derived for this kind of data
- allow 19 outcome measures to be added to the original eight at various times up to and after the breaking of the blind, purportedly according to an analytical plan ‘developed prior to opening of the blind’ (In spite of multiple requests, neither GSK nor Keller and colleagues have ever produced this analytic plan, suggesting that either it does not exist, or that it contains information unsympathetic to their claims.)
- accept the dismissal of protocol-specified secondary outcomes and the introduction of rogue variables on the grounds that ‘the Hamilton Depression Rating Scale (our primary outcome measure) had significant limitations in assessing mood disturbance in younger patients’, when none of the protocol-specified secondary outcome measures that they discarded were based on the HAM-D, and two of the rogue measures that they introduced were HAM-D measures
- accept the clinically dubious improvements in four of these rogue variables as evidence of efficacy. (Although these measures achieved statistical significance in the pre-defined eighth (final) week of the acute phase of the study, they did not do so in the weekly assessments over the previous seven weeks, a pattern unseen in any known antidepressant; we are working on another manuscript analysing Keller et al’s rogue variables.)
There was no ambiguity about the appropriateness of these methodological manoeuvres when Study 329 was conducted and reported. However, although some of these problems were obvious when the paper was first published, others were not apparent until we had access to the raw clinical data. This lack of transparency erodes confidence that RCTs will be conducted, analysed and reported free from covert manipulation.
Furthermore, Keller and colleagues also failed to report on the continuation phase of Study 329, even though that was a protocol-specified outcome. A report of this phase is almost ready for submission by us.
Harms
With regard to harms, Keller and colleagues are simply incorrect in many of their claims about our purported bias and lack of blind ratings.
First, our paper makes it clear that both coders in the re-analysis were blind to randomisation status.
Second, there was no ‘re-scoring’. This odd choice of words raises doubts that Keller et al have much expertise in analysing harms. We used a dictionary that adhered much more closely to the verbatim terms used by the face-to-face interviewers. The fact that Keller and colleagues say that we have labelled emotional lability as suicidality makes us wonder if they have seen the individual patient level data; it was the SKBs coders who came up with the term ‘emotional lability’, not the face-to-face interviewers, whose verbatim terms were of suicidal thoughts and behaviour. Simply using the verbatim terms that the named authors or their colleagues had used when faced with these adolescents reveals a striking rate of suicidal events. To argue that our return to these verbatim terms was arbitrary is bizarre.
Third, we made it clear there is unavoidable uncertainty in coding, and we invited others to download the data we have made available and juggle it to see if they can improve on our categorisation of the data. In our correspondence with BMJ, we made it clear that there are items that GSK could argue are more appropriately coded differently. We would be receptive to a rationale for alternate coding of certain items that is cogently argued rather than simply asserted, but our hunch is that a disinterested observer reviewing the coding as presented by GSK across all 1500 adverse effects in this study (or 2000+ if we include the continuation phase) would conclude that our efforts are a better representation of the data.
Fourth, reading our paper makes it clear why we reviewed the clinical records of 93 subjects; these were the subjects who dropped out or became suicidal. Our claims about underreporting of adverse events stand independently of that non-random sub-sample.
With regard to suicidal ideation and attempts, Keller et al. refer to a reanalysis by Bridge and colleagues, which found that there was no significant difference in suicidality between paroxetine and placebo. But Bridge et al. relied on Keller et al.’s misleading 2001 report.
With regard to bias, our point was that the best protection against bias is rigorous adherence to predetermined protocols and making data freely available. We, like everyone, are subject to the unwitting influence of our bias. The question is whether the Keller et al publication of 2001 manifests unconscious bias or deliberate misrepresentation.
The original and restored studies, the study data, reviews and responses are all available at Study329.org, offering a broad range of options when it comes to consideration of authorship, research misconduct and the newly described species, ‘research parasite’[4].
References
1 Keller MB, Birmaher B, Carlson GA, Clarke GN, Emslie GJ, Koplewicz H, Kutcher S, Ryan N, Sack WH, Strober M. Restoring Study 329: efficacy and harms of paroxetine and imipramine in treatment of major depression in adolescence. Response from the authors of the original Study 329. BMJ 2015;351:h4320
2 Le Noury J, Nardo JM, Healy D, Jureidini J, Raven M, Tufanaru C, Abi-Jaoude E. Restoring Study 329: efficacy and harms of paroxetine and imipramine in treatment of major depression in adolescence. BMJ. 2015 Sep 16;351:h4320.
3 Keller MB, Ryan ND, Strober M, et al. Efficacy of paroxetine in the treatment of adolescent major depression: a randomized, controlled trial. J Am Acad Child Adolesc Psychiatry 2001;40:762-72.
4 Longo DL, Drazen JM. Data sharing. N Engl J Med. 2016;374:276-7.
Competing interests: As noted in our RIAT paper
annie says
The response does not make much of the fact that not all of the Als in Keller et al have lined up to sign their letter.
tracking the line
All the Als…No Diving, No Fishing, No Looking :/
R O S E T A X
GREEN LOB CAD
“GSK doing a grate job”
DAVID HEALY
https://pbs.twimg.com/media/CPEHhDgVAAAKSGg.jpg
This was A1.ways the Best Quote of the Day…..
ben goldacre @bengoldacre Apr 1
GSK doing a great job of ditching the anti-social practices that never made pharma enough to justify reputation hit http://flnx.co/a5CLS
Trying not to Overload Intellectual Property, here are some more:
Different Take on GlaxoSmithKline by Trevor Butterworth, and, Study329.org
http://www.stats.org/to-err-is-human/
‘massaging’ being the operative word..
With the where withal of a molten tub of margarine, Ben Goldacre of lur.pak fame whose dedication to Allthings GlaxoSmithKline is breath taking…
Know which side of the bread is buttered….
ben goldacre @bengoldacre 14h14 hours ago
I have little respect for academics who think it’s better to put narrative reviews, essays, and analysis into journals rather than elsewhere
SenseAboutScienceUSA @SenseScienceUSA
To err is human. Why not not give scientists an “A” for admitting their errors? http://bit.ly/1VYl9Iw #AllTrials #trials #science #stats
Why not give scientists an “A”…
Why not give GlaxoSmithKline a “K”…
Why not give Alltrials a “G”…
Gift..
GSK Retweeted
The Crowd @thecrowd Apr 1
Imagine if all companies went open on IP that benefits society? @GSK just did something important. http://bbc.in/1ZPBori
just ‘did’ something important
Adam Jacobs @statsguyuk
It’s hard to see how any journal with < 100% acceptance rate could reasonably claim to subscribe to #AllTrials https://twitter.com/jocalynclark/status/717003315566010368 …
Back to Top
via Held Day rD
The original and restored studies, the study data, reviews and responses are all available at Study329.org, offering a broad range of options when it comes to consideration of authorship, research misconduct and the newly described species, ‘research parasite’[4].
Research Pals..
Johanna says
I think David Healy hit the nail on the head in his interview with the Guardian last fall when the study was published:
“This is a very high rate of kids going on to become suicidal. It doesn’t take expertise to find this. It takes extraordinary expertise to avoid finding it.”
For Keller & Co. to pretend they couldn’t have reported these simple facts – that they were passive victims of a side-effect dictionary imposed on their profession by disinterested experts sitting on a mountaintop somewhere – is ludicrous. And disgusting.
annie says
Kellers’ Legacy Lives On…’soul wincing’ for those of us involved with Seroxat
“I like to employ the following analogy courtesy of Dr. David Healy
Healy has worked tirelessly to expose data implicating antidepressants in risk of suicide and violence, maintaining a database for reporting, writing and lecturing about cases of medication-induced death that could make your soul wince.
By Kelly Brogan
April 5, 2016 | 2:17pm
http://nypost.com/2016/04/05/are-antidepressants-making-women-sicker/
annie says
“… a wounded Glaxo.
http://www.thisismoney.co.uk/money/markets/article-3526384/Shire-AstraZeneca-GlaxoSmithKline-Pfizer-s-hit-list.html
and Trev butterworthy
Trevor Butterworth @Butterworthy
The #AllTrials campaign in the US welcomes @CHDIFoundation as a supporter of clinical trial #transparency!
annie says
“unquestionable value for patient safety
http://www.madinamerica.com/2016/04/study-329-response-to-keller-colleagues/
One thought on “Study 329: Response to Keller & Colleagues”
Drtim on April 6, 2016 at 5:20 pm said:
Thank you Professor Healy and all colleagues for your intellectual rigour, fastidious attention to detail and determined adherence to defined scientific methodology. This crucially important re-introduction of academic integrity into clinical trial data analysis (via such objective and extensive re-analysis) is so greatly needed, and it is of unquestionable value for patient safety in psycho-pharmacology. Those compelled to dismiss and denigrate your vital work may perhaps be less focused on issues of safety? Deeply interested doctors outside your field who, in increasing numbers record their profound professional discomfort and concern regarding the apparently cavalier use of prescription drugs with life threatening and very serious ADR’s, recognise and applaud your courage and integrity.
BOB FIDDAMAN says
It’s difficult to know just how many teenagers were affected by the “oversight” of Keller et al.
The original study was a golden opportunity for Glaxo spokespersons to claim safety and efficacy in this particular population.
Once news broke (even though they had known it all along) that Paxil was causing these suicidal feelings and attempts in teens, Glaxo spokesperson, Alistair Benbow, played it down by claiming on national TV that, “…if you imagine a school of more than a thousand children all of whom are deeply troubled by depression, less than a small class size would have these suicidal thoughts or attempts…”
My grasp of maths isn’t that great but I’ve tried to breakdown Benbow’s actual class size here
– http://fiddaman.blogspot.co.uk/2015/09/alastair-benbow-devil-is-in-details.html
annie says
Selected highlights from three GSK selected KOLs in three short movies: MHRA, European Patients Forum and Med Tech Europe
“I think what GSK is doing is very good
“GSK has been a trail blazer
“Stepping out practises
GSK @GSK 1h1 hour ago
Is there a need to reform industry relationship with healthcare professionals? Key opinion leaders share their views
http://www.gsk.com/en-gb/behind-the-science/how-we-do-business/does-the-relationship-between-the-pharmaceutical-industry-and-healthcare-professionals-need-to-change/
These changes are significant but we are committed to transforming our business model so that patients are at the heart of every decision we take
GSK Chief Medical Officer, Murray Stewart
Laurie Oakley says
“It is always the respectable classes, the polished…who are the most susceptible to evil. To be intelligent, as many are at least in a narrow, analytical way, is morally neutral. These respectable citizens are inculcated in their elitist enclaves with “values” and “norms,” including pious acts of charity used to justify their privilege, and a belief in the innate goodness of [American] power. They are trained to pay deference to systems of authority. They are taught to believe in their own goodness, unable to see or comprehend—and are perhaps indifferent to—the cruelty inflicted on others by the exclusive systems they serve. And as norms mutate and change, as the world is steadily transformed by corporate forces into one of a small cabal of predators and a vast herd of human prey, these elites seamlessly replace one set of “values” with another. These elites obey the rules. They make the system work. And they are rewarded for this. In return, they do not question.”
http://www.truthdig.com/report/page2/finding_freedom_in_handcuffs_20111107
Walter K says
If Drazen has come up with the phrase ‘research parasite’, then accept it as a badge of honour. There are no depths they will not plumb to attempt to prevent truth from prevailing over corruption and dishonesty.
Walter
kiwi says
“With regard to suicidal ideation and attempts, Keller et al. refer to a reanalysis by Bridge and colleagues, which found that there was no significant difference in suicidality between paroxetine and placebo. But Bridge et al. relied on Keller et al.’s misleading 2001 report.”
What a house of cards!!
annie says
Meanwhile, at HQ, Kate of GKK talks to…..fresh air..
“We need more ambitious longer term TARGETS
GSK Verified account @GSK 9h9 hours ago
“We need to challenge ourselves to bring patients closer to the science we do” – Kate at #GPC2016
https://pbs.twimg.com/media/Cfv5MVFWsAAuxfK.jpg
https://pbs.twimg.com/media/Cfv8g0CWcAAP1WL.jpg
GSK @GSK 9h9 hours ago
Kate Knobil explains what needs to be done to involve patients more in R&D at #GPC2016
GSK @GSK 10h10 hours ago
Kate Knobil will be speaking on patient involvement in R&D very shortly at #GPC2016. Stay tuned!
Noble..