Meta-Analysis Mess

  • by: |
  • 06/06/2007
Nissen's entire approach is a case study of how to use statistical tools to manufacture biased results. In “Why Most Scientific Research Findings Are False” John Ioaniddis, a professor nstitute for Clinical Research and Health Policy Studies, Department of Medicine, Tufts-New England Medical Centera notes that a study relying on meta-analytic finding from inconclusive studies where pooling is used to “correct” the low power of single studies, is probably false and biased.

Research findings from underpowered, early-phase clinical trials would be true about one in four times, or even less frequently if bias is present. Epidemiological studies of an exploratory nature perform even worse, especially when underpowered, but even well-powered epidemiological studies may have only a one in five chance being true. Nissen’s research which combines small clinical trials to conduct a epidemiological study of an exploratory nature that deliberately excludes patients without heart attacks, that does not independent confirm if one took place and does not have access to patient level data, to find a risk he believes is there may or may not be true but it is certainly biased and likely to be false.

Indeed, stating that there is a increase of 40 percent risk in heart attacks relative in Avandia users compared to others raises two other troubling questions. First, it is generally true that, as Ioannidis claims that independent of molecular or genetic confirmation of a cause and effect, “too large and too highly significant effects may actually be more likely to be signs of large bias in most fields of modern research.”

They should lead investigators to careful critical thinking about what might have gone wrong with their data, analyses, and results.

For instance, in the 1980s a Swedish epidemiological study found that people with hip and knee replacements had a 30 percent greater increase of kidney cance than those who had no surgery. That study did not cause the authors to run to Congress and the media with results at a politically sensitive time. Instead, it suggested further epidemiological analysis and observational analysis which lead to the conclusion that the ‘relationship’ between orthopedic implants and kidney cancer was “noise” as opposed toa signal of something going wrong.

The proper response to the Avandia exercise would be to conduct further research and to put the general risk of heart events in context, Nissen, the authors of the editorials supporting his claim and the NEJM have not done either. Instead Nissen ran to the media and Congress with a highly speculative report. The NEJM gave the article prominence and failed to run a cautionary editorial.

Center for Medicine in the Public Interest is a nonprofit, non-partisan organization promoting innovative solutions that advance medical progress, reduce health disparities, extend life and make health care more affordable, preventive and patient-centered. CMPI also provides the public, policymakers and the media a reliable source of independent scientific analysis on issues ranging from personalized medicine, food and drug safety, health care reform and comparative effectiveness.

Blog Roll

Alliance for Patient Access Alternative Health Practice
Better Health
Biotech Blog
CA Medicine man
Cafe Pharma
Campaign for Modern Medicines
Carlat Psychiatry Blog
Clinical Psychology and Psychiatry: A Closer Look
Conservative's Forum
Club For Growth
Diabetes Mine
Disruptive Women
Doctors For Patient Care
Dr. Gov
Drug Channels
DTC Perspectives
Envisioning 2.0
FDA Law Blog
Fierce Pharma
Fresh Air Fund
Furious Seasons
Gel Health News
Hands Off My Health
Health Business Blog
Health Care BS
Health Care for All
Healthy Skepticism
Hooked: Ethics, Medicine, and Pharma
Hugh Hewitt
In the Pipeline
In Vivo
Internet Drug News
Jaz'd Healthcare
Jaz'd Pharmaceutical Industry
Jim Edwards' NRx
Kaus Files
Laffer Health Care Report
Little Green Footballs
Med Buzz
Media Research Center
More than Medicine
National Review
Neuroethics & Law
Nurses For Reform
Nurses For Reform Blog
Opinion Journal
Orange Book
Peter Rost
Pharm Aid
Pharma Blog Review
Pharma Blogsphere
Pharma Marketing Blog
Pharmacology Corner
Pharmaceutical Business Review
Piper Report
Prescription for a Cure
Public Plan Facts
Real Clear Politics
Shark Report
Shearlings Got Plowed
Taking Back America
Terra Sigillata
The Cycle
The Catalyst
The Lonely Conservative
Town Hall
Washington Monthly
World of DTC Marketing
WSJ Health Blog