Saturday, August 17, 2013

Limitations (and failure) of Medical Research

Background

The other day, I had a conversation with an academic on Twitter who retweeted a link to the Morris & Krieger study Does Male Circumcision Affect Sexual Function, Sensitivity, or Satisfaction?-A Systematic Review. This study has been fully discredited.

A number of individuals, including myself, called this researcher out. We pointed to numerous other studies as counter-examples. This researcher couldn't be convinced. After all, the study was a "systematic review" which, in the academic (and Wikipedia) communities, means infallible.

I pointed out that the "study" was performed by a pro-circumcision fanatic who was also a circumfetishist. Academics, apparently, put no stock in these details: they assume that each researcher is ethical, unbiased, and would never skew results.

I pointed out that the "study" only looked at 36 of 2675 studies on the subject. Apparently, looking at less than 2% of the studies on a subject qualifies as "systematic" in the research community and is fairly standard. Who knew? But of course the selection criteria couldn't be skewed! How could a researcher possibly creatively select studies to show a specific result? This doesn't happen, apparently.

This researcher stubbornly refused to believe that there were any flaws in this study and that it represented scientific fact. She pointed out another "systematic review" that, even in the summary, admitted that it reviewed "low quality studies" and "more analysis was needed".

The Failure

My interaction with this academic researcher highlighted the complete and abject failure of medical research to properly vet research and to present the truth. The reason for this failure appears to be naïveté and special interest. The article Lies, Damned Lies, and Medical Science highlights a number of failings of medical research. The failings that I have identified are:

  • The belief that each researcher is honest, ethical, and credible
  • The belief that there is no reason to vet a researcher's pedigree or motivations
  • The belief that only the "type of study" rather than the study methods and/or inclusion criteria is important (this is a failure of Wikipedia as well)
  • Dubious methods for determining scientific consensus

The Reality

The reality is that lots of research is done for dishonest reasons. Lots of researchers are out to promote a certain point of view. If there was research commissioned by BP showing that there was no significant harm to wildlife after the Gulf oil spill, would you believe it? Of course not—only a fool would believe this research to be credible. But in the medical community (and Wikipedia), credibility has no value. Any attempt to point out conflict-of-interest, financial, personal, or political motivations of researchers is considered a valueless ad-hominem attack (personal attack) and is rejected out-of-hand. In the "real" (non-academic) world, there are dishonest people. Let's not be foolish.

The idea that the "type of study" makes a particular study infallible is nonsense. Researchers can choose their own criteria for systematic reviews. Just as you wouldn't trust BP's results on Gulf oil spill effects, you can't judge that Morris's selection criteria for his studies is unbiased either. If his criteria is only studies that show effects on glans sensitivity (and nothing else), then he will get the desired result. If you choose to study the part of the penis that is the least sensitive and present in both intact and circumcised men, you'll likely get the outcome you want. Since both academics and Wikipedians fail to vet study criteria, they fail to produce the truth.

More than one Wikipedian has mentioned that scientific consensus is determined by (literally) counting up the number of studies showing a certain conclusion. Not only does Wikipedia's own policy on this subject fail to confirm the validity of this method, but it makes very little practical sense. Clearly, researchers with a common point of view (such as oil companies, genital mutilation advocates, etc.) will commission research to try to "prove" that their viewpoint is fact. Those with a financial interest in study outcome will fund lots of researchers at lots of universities. And what if that money comes with required study criteria? I asked multiple times for the Wikipedians to provide me a source that confirms that counting up studies is a way to prove scientific consensus. Not surprisingly, none was given.

My Take-Away

I honestly believe that the academic I encountered on Twitter was quite foolish and naïve. Wikipedians and other academics feel the same way and have the same mindset. And that mindset has many shortcomings. We do not live in a perfect world and we cannot assume that every person or researcher is honest. To believe a "systematic review" without vetting its methods, its authors, or by validating its results by checking other studies is dangerous and foolish.

Science works to prove theories by aggregating facts and evidence. Medical research may or may not be scientific. To blindly assume that all research is science is lunacy.

3 comments:

  1. In some way, it's almost a circular argument. Morris himself removed three studies from inclusion because they were "involving self-selected participants recruited
    via author-managed anti-circumcision facilities
    and that investigated psychological phenomena
    [28,41,42] were removed".

    So, to point that an author is a circumcision advocate is seen as an ad hominem, but for Morris to indicate that 3 studies were performed by "anticircumcision" groups is supposed to be elimination of bias?

    The whole "anticircumcision" label is quite mistaken, as intactivist organizations do not oppose circumcision per se, either as a medical procedure when needed or as an elective procedure when undertaken by a consenting adult, but only the forced genital cutting of children of any gender who are too young to understand what's going on and who cannot legally consent.

    And when it comes to bias, one would have to wonder what do intactivist organizations that work on a volunteer basis have to profit from reducing the circumcision rate? And meanwhile, what do circumcision advocates loose when the circumcision rate changes?

    Nobody makes any financial profit from stopping an unnecessary procedure. Our motivation to study and share awareness of this topic is basically one of morality, ethics, human rights. We can't really say the same of Morris, Gray, Tobian, Halperin, Bailey, Krieger and the other usual circumcision advocates, who make a living out of this.

    ReplyDelete
  2. Now that I'm reading Morris' publication in more detail, I also note some critical points.

    He seems to imply that there is a problem with using "self-selected participants recruited via author-managed anti-circumcision facilities", but yet when considering the RCTs, no word is said that the participants were all volunteers and recruited by institutions that promote circumcision, with the understanding that all participants would be circumcised, some early and some at the end of the study.

    I also find it interesting that Morris would debate almost two pages of discussion to criticize those studies that reached conclusions that differ from his desired conclusion, i.e. Sorrells, Bronsealer, Pang, Frisch...

    I was also wondering why he excluded histological studies such as the writings of Taylor on the ridged band and anatomy of the foreskin. Apparently though, he is going to review histological studies on a separate paper.

    Can Morris and Krieger in good faith indicate that they have no conflict of interest? This question needs no answer.

    ReplyDelete
  3. You bring up a very good point about motivations. There are no "honest" reasons to promote genital cutting, but some very honest reasons to try to end it. The money is all on one side of the table.

    As long as the research/academic community ignores conflict of interest and credibility, it will fail to produce the truth. It is scary that we make important decisions about our health (and children's health) based on non-scientific methods.

    I very much appreciate circleaks.org. It gave me great visibility into some of the true motivations behind the folks at Wikipedia, Twitter, and elsewhere. Truly eye-opening.

    ReplyDelete