PCORI's Information Imbalance

  • by: |
  • 01/25/2012
I received a bunch of emails from people regarding my previous post on PCORI and I was also quoted in a recent Politico article on PCORI punting on looking at diseases and instead spending more money and hiring consultants to yet again decide what it should hire more consultants and hold more meetings about what it should study.   I received a form letter from PCORI's executive director Joe Selby that reads in part:

National Priorities for Research and Research Agenda. The priorities and agenda provide a framework and identify the broad questions that must be addressed so that patients can make better and more personalized decisions in partnership with their clinicians across all areas of health.

Our initial research agenda is not prescriptive about which conditions or treatments to study. It is a starting point. We hope that patients and other stakeholders will join with us in the coming months as we apply this framework to identify the specific questions that are most important for PCORI to address.

An initial research agenda with the goal of deciding what broad questions a neither research nor starting point, it's group therapy.

In my previous post I wondered by the National Pharmaceutical Council would hold a conference on asymmetries in information coming out of PCORI and made a somewhat snarky comment about not knowing what asymmetry is with respect to PCORI unless it mean dividing up it's grant money to stakeholders.

It was explained to me that 'asymmetry' has to do with an extremely important issue:  Namely while  that PCORI, AHRQ, etc will not only determine what information is sent out to doctors, patients, health plans  but will also control the means of production and distribution.  Unlike NIH, what PCORI will put out can and will be used in determining what health plans should and will pay for.  But given PCORI and AHRQs financial focuse on dissemination it begs the question of whether the mostly outdated information that will be distributed via government web sites, conferences and academic detailers will crowd out other forms of information and studies.  

The NPC event on asymmetry will be important if it raises these questions  http://www.npcnow.org/Public/Education___Events/Events/2012_events/Asymmetry_2012/asym12.aspx

Further, PCORI is loaded with what are called a heuristic biases.   I encourage anyone who doesn't think that PCORI is designed to establish asymmetries to read Judgment under Uncertainty: Heuristics and Biases  Amos Tversky; Daniel Kahneman Science, New Series, Vol. 185, No. 4157. (Sep. 27, 1974), pp. 1124-1131.  http://links.jstor.org/sici?sici=0036-8075%2819740927%293%3A185%3A4157%3C1124%3AJUUHAB%3E2.0.CO%3B2-M

Tversky and Kahneman describe how subjective assessment of probabliities leads to systematic and severe errors in judgment.   Even more so if the subjective assessement is also systematic. 

Why?  As T and K explain: judgments are all based on data of limited validity, which are processed according to heuristic rules.
What are these heuristics?  The first one of representativeness is used to assess a person, product, or event based on how similar it seems to be of stereotypes or preconceptions we already have in mind.   PCORI uses representativeness by assessing, funding and disseminatiing CER in terms of how well it fits the subjective belief that all CER will lead to lower costs and better health.   That is, if a particular approach to research and dissemination fits the pre-existing stereotype of CER then PCORI assumes it will improve health delivery.  Moreover, any objectiion raised to CER by introducing information basedon  the prior probability, or base-rate frequency, of the outcomes of CER is ignored.   Simply stated, PCORI never looks at the actual relevance and impact of CER to determine if it effective.  Rather, the problem is one of poor dissemination, not design or scholarship.  This insensitivity to prior outcomes is a hallmark of representative bias and explains the total lack of PCORI's self criticism.

Second, PCORI and AHRQ institutionalize insensitivity to sample size.   If a study compares two products and is representative of the desired CER outcome then it is accepted.  It adheres to the law of small numbers," according to which even small samples are highly representative of the populations from
which they are drawn. PCORI's is framed by the expectation that a valid hypothesis about a population will be represented by a statistically significant result in a sample- with little regard for its size.  So for instance Steve Pearson and Sean Tunis, two people who's organizations stand to be enriched by PCORI cash cite clinical trials that 'concluded' injection of medical cement into compression fractures of the spine produced no better pain relief than "sham" injections and the FDA's withdrawing approval of  Avastin for advanced breast cancer (claiming there were no benefits given the risks) as examples of good CER.  

But both ignore the extremely small sample sizes of studies for both issues.  Rather, the small studies are not only highly representative of the population as a whole but representative of what good CER is.    And this exposes another, more profound bias institutionalizd by PCORI, name that of availability.    We remember and regard as true based on what information happens most often or is more available.   Hence, because PCORI and AHRQ will produce and deliver most of the information on medical treatments and will control the outlets for such information it uses the heuristic of availability to create the perception that the CER it produces is the "right" path.  Or as Kahneman quips: "What we see is what it is."

Finally, PCORI reinforces the view that only CER and evidence based medicine it produces should be talked about, shared and acted upon.  So what if consumers with their own genetic information, companies that produce novel results, or physicians want to learn more about individualize care want to share and use such knowledge.   It turns out that they can't:  Under the health care law all the actionable information for benefit coverage and reimbursement is channelled through AHRQ and PCORI.   It's ok for AHRQ funded academic detailers to wine and dine docs about the blessings of CER but it is illegal for innovators to talk about and share scientific knowledge about the novel uses of their products. In otherwords some medical knowledge (PCORI/AHRQ produced CER generated by the people who have a subjective bias that CER is wonderful) is more equal than others.

There is more to discuss regarding asymmetry and there will be more to come.   But as you can see, I believe I was wrong to dismiss asymmetry as a Beltway slogan.   Asymmetry is a real problem.   As Tversy and Kahneman note: "The rational judge will nevertheless strive for compatibility,
even though internal consistency is more easily achieved and assessed. In particular, he will attempt to make his probability judgments compatible with his knowledge about the subject matter, the laws of probability, and his own judgmental heuristics and biases."

The way PCORI Is structured and the way in which AHRQ generates and produces CER, incentivizes the use of these biases.  By seeking to generate acceptance of CER and embedding results into clinical decision tools, PCORI and AHRQ makes "internal consistency" of the research it's funds the foundation of medical decsionmaking.    It thereby institutionalizes several errors in judgment that will leave patients less healthy.


Center for Medicine in the Public Interest is a nonprofit, non-partisan organization promoting innovative solutions that advance medical progress, reduce health disparities, extend life and make health care more affordable, preventive and patient-centered. CMPI also provides the public, policymakers and the media a reliable source of independent scientific analysis on issues ranging from personalized medicine, food and drug safety, health care reform and comparative effectiveness.

Blog Roll

Alliance for Patient Access Alternative Health Practice
Better Health
Biotech Blog
CA Medicine man
Cafe Pharma
Campaign for Modern Medicines
Carlat Psychiatry Blog
Clinical Psychology and Psychiatry: A Closer Look
Conservative's Forum
Club For Growth
Diabetes Mine
Disruptive Women
Doctors For Patient Care
Dr. Gov
Drug Channels
DTC Perspectives
Envisioning 2.0
FDA Law Blog
Fierce Pharma
Fresh Air Fund
Furious Seasons
Gel Health News
Hands Off My Health
Health Business Blog
Health Care BS
Health Care for All
Healthy Skepticism
Hooked: Ethics, Medicine, and Pharma
Hugh Hewitt
In the Pipeline
In Vivo
Internet Drug News
Jaz'd Healthcare
Jaz'd Pharmaceutical Industry
Jim Edwards' NRx
Kaus Files
Laffer Health Care Report
Little Green Footballs
Med Buzz
Media Research Center
More than Medicine
National Review
Neuroethics & Law
Nurses For Reform
Nurses For Reform Blog
Opinion Journal
Orange Book
Peter Rost
Pharm Aid
Pharma Blog Review
Pharma Blogsphere
Pharma Marketing Blog
Pharmacology Corner
Pharmaceutical Business Review
Piper Report
Prescription for a Cure
Public Plan Facts
Real Clear Politics
Shark Report
Shearlings Got Plowed
Taking Back America
Terra Sigillata
The Cycle
The Catalyst
The Lonely Conservative
Town Hall
Washington Monthly
World of DTC Marketing
WSJ Health Blog