Data Based Medicine and Cochrane Inc

September, 20, 2018 | 17 Comments

Comments

  1. Imagine if architects had for three decades insisted that the best foundations were laid in sand – and everybody believed them!

  2. I was wondering how the Cochrane Evidence Base for antidepressants worked out based on the recent Cipriani Study and Associated Hoo-ha ..

    I took this one at random which told ‘Results from this review are consistent with this interpretation and might contribute to developing and keeping up to date an evidence‐based hierarchy of antidepressants to be used by clinicians (both specialists and general practitioners) (Barbui 2011).’

    Citalopram appears to be a suitable option to be used for moderate‐to‐severe acute major depression because it showed to be more effective than other antidepressants (namely, paroxetine and reboxetine) and it was overall well tolerated.

    https://www.cochranelibrary.com/cdsr/doi/10.1002/14651858.CD006534.pub2/full

    Why it is important to do this review

    To shed light on the field of antidepressant trials and the treatment of major depression, a group of researchers agreed to join forces under the rubric of the Meta‐Analyses of New Generation Antidepressants Study Group (MANGA Study Group) to systematically review all available evidence for each specific newer antidepressant. We have up to now completed some individual reviews about fluoxetine (Cipriani 2005a), sertraline (Cipriani 2009b), escitalopram (Cipriani 2009c), milnacipran (Nakagawa 2009), fluvoxamine (Omori 2010), and a number of other reviews are now underway. Thus, the aim of the present review is to assess the evidence for the efficacy and tolerability of citalopram in comparison with TCAs, heterocyclics, MAOIs, SSRIs, SNRIs and other antidepressants in the acute‐phase treatment of major depression.
    Agreements and disagreements with other studies or reviews

    Even though it is matter of ongoing discussion in the scientific literature (Gartlehner 2010; Gartlehner 2011), there is now robust evidence that there are statistically and clinically significant differences among antidepressants (Cipriani 2009a). Results from this review are consistent with this interpretation and might contribute to developing and keeping up to date an evidence‐based hierarchy of antidepressants to be used by clinicians (both specialists and general practitioners) (Barbui 2011). Even though citalopram was not among the best treatments in terms of efficacy, it scored well in terms of acceptability and remains an important option for physicians when an AD is to be prescribed for moderate‐to‐severe major depression.

    https://www.psych.ox.ac.uk/team/andrea-cipriani

    I am currently Editor in Chief of Evidence-Based Mental Health. I am also on the Editorial Board of the Lancet Psychiatry and one of the Editors of the Cochrane Collaboration for Depression, Anxiety and Neurosis.

  3. David,

    You provide in this article a most interesting and refreshing perspective.

    As Cochrane’s raison d’être is founded upon a false “gold standard” of evidence it follows that the evidence it generates is also founded on a premise of falsely assumed validity.

    Once we get to that position, the evidential value of Cochrane reviews can be seen in a new light.

    It is certain that there is no sound basis for an hierarchy of evidence nor is there a theoretical basis for positing one exists.

    We can demolish the concept of the hierarchy with reductio ad absurdum. It is relatively simple to achieve [more on this below]. The hierarchy is a construct made-up out of thin air like a conjuror’s trick or the rules of a childrens’ playground game.

    So what is the hierarchy’s true purpose? We do not need to ask that. We just need to see what its effect is.

    The hierarchy puts evidence created by the pharmaceutical industry [in the form of RCT’s and their analyses] at the top. The hierarchy acts as a commercial tool serving the interests of a drug industry out of control for profits. It functions just like a cuckoo in the nest. It pushes out all competition to put the form of evidence the drug industry can control and dictate at the top. RCTs are expensive financially to create. A wealthy drug industry can afford them. You can and I cannot.

    Case reports and clinical and case histories are substantially much less expensive and you and I and anyone else can document and report them with relative ease.

    By concentrating its efforts on meta-analyses and RCTs to the exclusion of all other relevant and significant evidence, Cochrane can be seen to be servicing the drug industry to the detriment of medicine and patients.

    Why is the construct of an hierarchy of evidence a false idol?

    Evidence just does not work like that. My co-author and I published a number of peer reviewed papers addressing some of the issues. [References to two appear at the end.]

    What we do know is any specific item of evidence is weighted according to the inherent reliability of that particular item which must be assessed on a case-by-case basis.

    And we also know that different items of evidence come together to form a factual matrix having greater value than just the sum of the parts.

    It follows that there can be no hierarchy of evidence nor is it valid to put meta-analyses at the top with RCTs just beneath.

    How can we know that?

    With your expertise in psycho-pharmacology you know that for proof a treatment effect exists it is the RCT or the study which is trumped by a single well-documented rechallenge in a single patient or three well-documented cases of dechallenge. So if there is a conflict with an RCT or an epidemiological study, these well-documented positive rechallenge and dechallenges are the more convincing evidence.

    So where do we put dechallenge and rechallenge in the hierarchy? Above meta-analyses?

    That would seem to be the case. So why do they not appear there?

    And where do we put in the hierarchy a poor quality RCT or one that is based on falsified data or reported in a ghost-written publication?

    Do we put that above or below a case series of well-documented cases or even an individual case report?

    Once we get to that position we can apply the same approach to meta-analyses. Where in the hierarchy do we put those which omit relevant evidence, such as relevant trials and their outcomes?

    And in this context it is worth noting the history compiled by my co-author Professor Donald Miller regarding the public scandal of the Cochrane meta-analysis regarding albumin administration in critically ill patients: Cochrane Injuries Group Albumin Reviewers (CIGAR). BMJ 1998;317:235-240.

    Albumin was more expensive for the UK’s NHS than alternatives. The study was funded by UK’s NHS, which could cut costs by replacing albumin with crystalloid. There were 30 RCTs involving 1419 patients.

    Conclusion: Risk of death 6% higher in patients given albumin.

    “… up to 30,000 patients in Britain alone have died because they were treated human albumin solution.” reported The Times July 24, 1998.

    “300 die as health chiefs dither.” reported The Observer (London) on July 26, 1998

    The response of the principal author to this furor: “We were amazed but totally confident we are accurate…having studied all the evidence I am sure we are right.”

    However:
    ● none of the (seven) authors cared for critically ill patients in the icu
    ● deaths 30 days included
    ● omitted relevant trials
    ● included trials that gave albumin on a daily basis for hypoalbuminemia, not hypovolemia
    ● combined heterogeneous trials (adults and high-risk neonates)

    Later, Wilkes and Navickis publish a systematic review in 2001 involving 55 RCTs and 3504 patients in Ann Intern Med. Conclusion: no effect of albumin on mortality, albumin is safe.

    The Cochrane Collaboration [as it was known then] quietly removed the Albumin review from its library of meta-analyses.

    REFERENCES

    Miller DW, Miller CG. On Evidence, Medical and Legal. Journal of American Physicians and Surgeons 2005;10:70-73.
    http://www.jpands.org/vol10no3/miller.pdf

    Miller CG, Miller DW. The Real World Failure of Evidence-Based Medicine: The International Journal of Person Centered Medicine Volume 1 Issue 2 pp 295-300.
    http://www.donaldmiller.com/Real_World_Failure_of_Evidence-Based_Medicine.pdf

  4. A couple of corrections to the prior post:

    “A wealthy drug industry can afford them. You and I cannot.”

    Re Albumin meta-analysis:

    “- deaths less than 24 hours excluded, greater than 30 days included”

  5. In a recent monograph on preventing and reversing Alzheimer’s Disease (AD) (https://smartech.gatech.edu/handle/1853/59311), I addressed the issue of why we retained so-called ‘failed’ treatments in our analysis, as follows.

    Our reading of thousands of abstracts on laboratory experiments and clinical trials of potential AD treatments has shown
    1) in vitro experiments typically performed on neural cells tend to have reasonably positive outcomes, at least for those papers that surface in the peer-reviewed published literature;
    2) in vivo experiments typically performed on rodents (but other small animals as well) tend to also have reasonably positive outcomes, albeit somewhat less than in vitro experiments;
    3) when these potential treatments reach the human clinical trial stage, especially the later phases, the success rates plummet!

    The explanation for this discrepancy given most often is the species difference. Humans are different from rodents et al, and their physiological responses to stimuli are different as well. However, the toxic experiential and exposure background differences between humans who live in the sea of toxic exposures in the real world and animals who live in the very controlled environment of the laboratory are rarely, if ever, discussed.

    Our monograph showed there are many hundreds of potential causes for AD (ranging from Lifestyle to Occupational/Environmental exposures). For a given individual, some causes have happened in the past, and are no longer happening, but their damage trail remains. Other causes are ongoing, have caused damage, and continue to cause damage.

    Why would anyone expect a human being with such a toxic history to respond to a potential treatment the same way that a laboratory animal raised in a controlled environment would respond to that treatment? Furthermore, why would anyone expect a human being with such a toxic history to respond to a potential treatment the same way that another human being without such a toxic burden would respond to that treatment?

    I give the example of Dr. Terri Wahls, an M.D. who was able to reverse her own case of Multiple Sclerosis (MS). She used two main types of treatments: lifestyle changes (mainly dietary) to reverse the MS and neuromuscular electrical stimulation (NMES) to reverse the damage resulting from MS. It was only when her diet achieved near-pristine status that the NMES produced positive effects.

    While Dr. Wahls’ experience represents only one data point, it is a very powerful data point. Consider its implications. Suppose a clinical trial were conducted to evaluate the potential for NMES to reverse the damage from MS. Suppose further that Dr. Wahls’ dietary-dominant contributing factor to MS and her reaction to NMES were typical of the participants in such a clinical trial. If the participants did not address their diet during the clinical trial, they would not respond positively to the NMES (as was the case for Dr. Wahls initially). The trial would be interpreted as a failure of NMES. However, in this hypothetical example, the NMES is not the reason for the clinical trial’s lack of success. Failure to remove the cause of the disease and subsequent damage is the problem!
    We cannot rule out failure to remove cause as a reason for the massive failure of myriad AD treatments in the clinical trials of the past three decades. That is why we have retained even so-called ‘failed’ treatments in the present full-spectrum study of existing AD treatments. We don’t know which treatments failed because 1) they were intrinsically ineffective or 2) their beneficial effects were overwhelmed by the strong negative effects of the ongoing causes remaining operable. In fact, we don’t know whether comprehensive, timely, and thorough removal of the relevant AD causes by themselves would have obviated the need for many of these AD treatments!

    There is also the issue of ‘manufactured’ data, but that remains for another day.

  6. Excellent article, David, I see a ton of history of science and statistics here. Under Kuhnian models of science, value neutrality hands everything over to the most powerful player in the arena; under the Popperian model of science, it does not. Popper knew that by allowing the subjective into any equation into science it would be corroded. But then he was discussing Science.

    RCTs CAN work. When they are Science. But when for-profit research is done to end of maximizing profit, those activities fall outside the realm of Science, and should be called “Science-Like Activities” (SLAs).

    As you know, twisted values led corporate medicine to twist their practices toward profits as well, a problem that causes accountants to calculate the ROI of specific medical care choices in terms while Quality of Life Years and other objective measures are applied to the same end strategically to maximize returns.

    I recall when all of medicine was going to go bankrupt due to the excess numbers of infants that were being born prematurely; they stood up NICUs and since the practice of vaccinating with HepB for anyone over 1 kg (a adult rat weight 1 kg!) with 250 mcg aluminum hydroxide, well, medicine has not exactly gone bankrupt, has it? The crash teams come when it’s time to vaccinate in the NICU. They know exactly what they are doing.

    Not surprising then that no one answered the IPAK NICU Challenge to produce a single study that shows that injecting aluminum hydroxide in the NICU is safe. No one knows how many infant lives have been sacrificed at the alter of almighty medicine because NICU deaths are never attributed to vaccination. But they know.

    More background info here:

    http://ipaknowledge.org/nicu.php

    All of clinical research must be reformed to where a firewall exists between the sources of funding to test drugs, biologics, and devices and those who conduct the science. The two should never meet. An independent NGO should handle the review processes of the grants and award the funds. Regulatory capture is nearly 100% in three-letter Federal alphabet. And data analysis for significance must be replaced with analyses that estimate reproducibility and generalizability. The next successful epoch in Science, if it comes, will be prediction science if we reach the Fifth Stage.

    See more here

    https://www.unbreakingscience.com/The-Five-Stages-in-the-Ontology-of-Science.php

    and tune into episodes of Unbreaking Science where we will outline a path to return objectivity to Science and help the public learn how to reliably identify Science from Science-Like Activities.

    • A word to the wise about unwise use of a word.

      Be wary when you see people writing using the word “science” as if is represents and identifies something that can be discussed using a single word.

      A good rule of thumb is you are being mislead whenever you read something where the author writes about “science” using that one word. It is meaningless to do that.

      People write reverentially about a unitary “science” as if that single word describes a single kind of activity and a single kind of information all of which can be trusted.

      The reality is far far removed.

      Once you start to think about all the different and often conflicting meanings that one word has you will realise that those who claim to engage in “science” do not understand that one word has so many meanings that it has become meaningless.

      It is mistaken and a misleading way of thinking. We are also mislead into thinking that “science” is a fount of all reliable knowledge.

      “Ice-cream” is a word which describes something we can experience and we identify as having common features. Even that can be misleading – is it ice-cream if made without cream?

      With “science” we cannot say the same – the difficulties are manifold and greater. There in no such thing as “science”. There are sciences and they are all very different. The information they generate is probabilistic, of widely varying qualities and far less reliable than people realise.

      Even the distinction some draw between exact and inexact sciences is not clear. Physics is considered by many to be an exact science because it is associated with successful reductionist experiments. But such experiments are impossible in numerous branches of the sciences we group together under the banner of the single word “physics”. Take astronomy, space or atmospheric physics as examples where reductionist experiments are generally not possible.

      How sound are Cochrane’s foundations? Is an RCT “science”?

      RCTs provide no knowledge to predict the outcome of treatment or its risks in an individual patient.

      RCTs do not tell us:

      – when factor X can be predicted to cause outcome Y in a particular patient;

      – that X is the only cause or factor in the cause of Y;

      – when X does not cause Y;

      – why or when Y will appear when X is not present.

      RCTs therefore cannot verify hypotheses about when or why X does or does not cause Y.

      RCTs also provide no information enabling us to predict new outcomes in addition to Y.

      RCTs provide no mechanism for prediction beyond a probability of how often we might expect X to cause Y.

      RCTs also provide no knowledge to predict the outcome of a treatment involving two or more drugs of unknown interactivity.

      Meta-analyses (systematic reviews) that combine RCTs addressing the same question have their own set of problems and biases

  7. Thank you, Dr. Healy, for this insight into what is wrong, and why, with Cochrane and the world of Pharma science.

    In my community of parents–most of whom are working to recover their children from environmental and/or iatrogenic injuries that have been normalized by mainstream medicine (ASD, allergies, asthma, ADHD, anxiety, etc.)–two terms are loathed: “standard of care” and “evidence-based.”

    Real world experience has shown us that doctors use these terms to justify prescriptions and to ignore–even intentionally refuse to even examine–any data or argument to the contrary.

    As an example, when my dad was recently in the hospital recovering from a stroke, he was experiencing “sundowning” episodes that would become violent. The doctor prescribed Seroquel. I researched the drug, printed a stack of information on the dangers of it from respected journals, along with lawsuits the maker had lost or were still fighting, for injuries, deaths, and marketing the drug off-label. I also printed articles an older and much safer drug. The hospital psychologist refused to look at them, saying he didn’t respect any science from the sort of journals I had pulled the studies from. Again–he did not look at them–I was holding articles from respected peer-reviewed journals. I told him to leave the room and not return. The doctor arrived soon after. He did leaf through the information, saying it was highly respectable and he was impressed with my research–but he refused to prescribe the older, safer drug, saying the Seroquel was “standard of care.” I said it’s an off-label use and the maker had been sued over it and been found guilty. The doctor said he knew, but it was still “standard of care” and since we refused to allow it, and he refused to prescribe the older drug, we were asked to leave the hospital and my dad was discharged early.

    Many doctors won’t even pay patients or their families the respect of leafing through information. A parent will refuse a drug for their child, attempt to discuss the diet-based approaches that have been working–improving a child’s symptoms–and the doctor will dismiss the evidence right before his eyes, call it coincidental, say there is no “scientific evidence” to support the diet approach. This is now, at last, changing, but it has been a long road and most doctors are still denying it.

    And we all know what happens when a parent tells a doctor that after a round of vaccines, their child changed. Lost language, lost motor skills, began walking on tip-toes, hand-flapping, having meltdowns. They are told that the “evidence” all says this cannot happen, the vaccines are not to blame. They disregard all evidence before them and, as you say, they blindly follow the message drilled in them, “trust us not them.” Trust Pharma science not your own patients.

    As parents who research deeply, we see everyday that the glowing titles, summaries, abstracts, and conclusions of studies, as well as blanket claims by public health agencies, do not match the data. We present this hard evidence to public health officials in attempts to get them to understand why policies must change, and they stare at us without comprehension, or they roll their eyes, or they gleefully continue on promoting their policies as if to justify their past actions and their loyal resistance to anything not coming from on high, from the almighty CDC or AAP or AMA.

    This under-oath testimony by Dr. Kelley of the Kennedy Krieger’s laboratory is quoted in JB Handley’s new book–already a bestseller–HOW TO END THE AUTISM EPIDEMIC.

    (begin quote)
    Lawyer: Do you agree with the statement that vaccines do not cause autism?

    Dr. Kelley: No.

    Lawyer: Is it generally accepted in the medical community that vaccines do not cause autism?

    Dr. Kelley: It is a common opinion.

    Lawyer: It is generally accepted in the medical field that vaccines do not cause autism?

    Dr. Kelley: I have no basis to judge that. It is most often when physicians are commenting on that they say there is no proven association.

    Lawyer: Do you know the position of the American Academy of Pediatrics about any link between vaccines and autism?

    Dr. Kelley: Yes. They also say there is no proven association.

    Lawyer: Do you agree with the position of the American Academy of Pediatrics?

    Dr. Kelley: I agree with their position as a public health measure. I don’t agree with it scientifically.

    Lawyer: You are actually arguing for a link between vaccines and autism in this case, aren’t you?

    Dr. Kelley: I am.

    Lawyer: And that is contrary to the medical literature, isn’t it?

    Dr. Kelley: It’s not contrary to the medical literature that I read. It is contrary to certain published articles by very authoritative groups who say there is no proven association in large cohort studies.

    Lawyer: Your opinion is contrary to, say, the opinion of the CDC, correct?

    Dr. Kelley: It is contrary to their conclusion. It is not contrary to their data.
    (end quote)

    I–and my community of exhausted parents–are so very grateful to doctors such as you standing up and speaking out. Thank you.

    • Dear Bernadette.

      I have my “war” with Seroxat/Paxil, and dr. David Healy and others are beacons of light there aswell.

      I have just grazed the surface of vaccines and what they cause, but I do not doubt for a minute they cause great harm. My opinion is that they cause autism.
      My opinion is also that there are older vaccines, that both work and cause much less harm. But I would never compare them to the ‘new era’ of vaccines.

      When Americans nowadays talk of autism found in about 1 in every ~80 (or so) you have to be rather dim not to understand we are doing alot of things wrong.
      Anyone defending ‘possible’ causes, are those who are afraid they are adding to this epidemic.

      Of course, we as humans, has the ability to find the root cause to things like this.
      But when IT IS caused by human intervention, it is also hidden by the same.

      I cannot see us getting anywhere closer to the truth, when supposedly unbiased entities like Cochrane act more and more like the manufacturers themselves.
      Ove, Sweden

      • Ove,

        I am not too sentimental about the older generation of vaccines but I note the paper by Gayle DeLong:

        “Is “Delitigation” Associated with a Change in Product Safety? The Case of Vaccines”

        https://link.springer.com/article/10.1007/s11151-017-9579-7

        “Abstract
        “This study investigates whether the threat of litigation induces firms to provide safer products in a regulated industry. I analyze whether removing litigation risk or “delitigation” of product liability is associated with a change in the safety of vaccines. Using U.S. nationwide and state-level data, I find that vaccines that were licensed after legislation that preempted most product liability lawsuits are associated with a significantly higher incidence of adverse events than were vaccines that were licensed under a previous regime that permitted consumers to sue. Oaxaca decomposition suggests that the difference is due to the policy change. The results suggest that product safety deteriorates when consumers are no longer able to sue manufacturers.”

        Of course, the critical issue here was the 1986 US Vaccine Injury Compensation Bill which took vaccine litigation out of the civil courts, whereas for instance in the UK for example manufacturers have more or less de facto immunity on all medicines and biologicals because the polticised Legal Aid Agency alias the Legal Services Commission alias the Legal Aid Board blocks litigation. Since of course the licensing bodies are dependent on pharma sponsorship we are in downward spiral.

        • I agree, from what little I know about the matter.
          I was surprised that not even americans can sue, amazing how they got that through? But I guess lobbying is also strong in America.

          Sweden is, of course, more similar to the U.K. than the U.S.
          Here it has been taken a step further, with the installation of a ‘nationwide insurance for damages done from pharmaceuticals’.
          https://lff.se/a-unique-type-of-insurance/for-patients/

          On the surface, it looks good to be able to get compensation without going to court. But it is, in reality, an insurance for the pharmaceutal companies to stay out of in-court litigation.

          I don’t know if the english version (linked above) is as comprehensive as the swedish version. But they aren’t even trying to hide the benefits to Big Pharma. In fact they invite them to join, because the insurance itself is a company run to make profit.

          And the ‘handouts’ that victims recieve is pathetique: as a first payout, for victims with established diagnose of ‘narcolepsy’ (and related issues) due to Pandemrix: £4’500

          A, from what I understand, lifelong condition, £4’500 is just another blow to their faces.

  8. http://www.ageofautism.com/2018/09/british-medical-journal-finally-goes-over-the-top-on-hpv-vaccine-safety.html

    https://www.bmj.com/content/362/bmj.k3694/rr

    Re: Challenges of independent assessment of potential harms of HPV vaccines Lars Jørgensen, Peter Doshi, Peter Gøtzsche, Tom Jefferson. 362:doi 10.1136/bmj.k3694

    In the light of this information [1] what are the ethical grounds for the continued prescription of these products which seem to be neither properly tested or monitored? How can we look to some long term nebulous benefit when public agencies are apparently so negligent of safety? And what do we do about those agencies?

    I have also been reading with dismay the shortly to be published ‘The HPV Vaccine On Trial: Seeking Justice for a Generation Betrayed’ [2] which parallels and augments much of what has been said here.

    [1] Lars Jørgensen, Peter Doshi, Peter Gøtzsche, Tom Jefferson, ‘Challenges of independent assessment of potential harms of HPV vaccines’, BMJ 2018; 362 doi: https://doi.org/10.1136/bmj.k3694 (Published 24 September 2018)

    [2] Mary Holland, Kim Mack Rosenberg and Eileen Iorio, ‘The HPV Vaccine On Trial: Seeking Justice for a Generation Betrayed’ with an introduction by Luc Montagnier, Skyhorse, 2 October 2018

  9. Dear DH,
    I’m so glad I got to read this thought provoking write up; and all the comments which followed. Being a faculty in a medical college, I too have long considered the Cochrane Reviews as the go-to place for the final verdict. I now realise they could very much be flawed. That does leave me quite flummoxed though. Where do we go, to find reliable opinion then?

Leave a Reply to David Healy