Meta-analysis studies do not work

Three recent examples provide all the ammunition one needs to discount meta-analysis studies:

1. A new PLOS Med study on the diabetes drug Metformin has the medical community up in arms. Researchers could not rule out anything from a 25% decrease to a 31% increase in the risk of death for any cause, or a 33% decrease to a 64% increase in the risk of cardiovascular mortality from taking the drug. The finding is surprising because the drug has been considered to be the first-line treatment for diabetes since 1998.
   
   The meta-analysis was sharply criticized on methodological grounds. In all, researchers found 25 studies, of which 13 had enough data on the outcomes to form the basis of a meta-analysis. Opponents argued that they were often comparing apples to oranges, in that a 1998 UK study was over a 12-year period, while many of the other studies in the analysis lasted only six months or a year. Critics claim that you cannot answer this question on studies of less than eight, nine, 10 years' duration.
   
2. The reported benefits of selective serotonin reuptake inhibitors (SSRIs) for treating the repetitive behaviors of autism spectrum disorders (ASDs) may be exaggerated by the selective publication of positive results.
   
   In a recent meta-analysis, the use of SSRIs was associated with a modest but significant improvement in repetitive behaviors. After accounting for publication bias, however, the improvement lost statistical significance, according to the May issue of Pediatrics.
   
   "Without timely, transparent, and complete disclosure of trial results, it remains difficult to determine the efficacy of available medications. It is especially ironic that problems in making research data available about the use of this class of drugs in children created a firestorm almost 10 years ago," the author of the editorial noted. "Unfortunately, this problem has not been resolved."
   
   Additional commentary in Pediatrics underscored that the problem of pediatric publication bias is not confined to studies of SSRIs. Researchers evaluated 160 randomly selected, NIH-funded studies and 758 randomly selected completed studies involving children. They found that only 53% of the former, and 29% of the latter, were published, which they cited as evidence of "substantial publication bias."
   
3. A recent Archive of Internal Medicine study discounting the efficacy of fish oil was another meta-analysis gone wrong. See our response here.

What's the worst part of the meta-analysis problem? The studies on Metformin and antidepressants received no media attention. This would have been a great opportunity to bring the issue to the public's attention. Alternatively, the fish oil study was headline news at every major media outlet. Very few mentioned that the study was littered with bias.