Avoiding "Waste" in Research: Why Methods Matter

Limitations in the design, implementation, and analysis of clinical research can produce biased study results that could have serious consequences for patients. For example, common methodological problems may overestimate benefits of treatments and underestimate risks.

Getting the methods right is essential for any comparative clinical effectiveness research (CER) or patient-centered outcomes research (PCOR) project. That’s why we at PCORI emphasize that “methods matter” and are working hard to ensure methodological rigor in the projects we support and encouraging other funders to do likewise.  

The Cost of Methodological Problems

It is wasteful to invest resources in research that will not produce valid and reliable results. A recent study in the BMJ underscores this point. Researchers assessed more than 1,200 clinical trials for “avoidable waste” attributable to methodological problems. The team found that incomplete reporting prevented them from fully evaluating 41 percent of the trials, and inadequate methods resulted in another 43 percent being at high risk of generating biased results in at least one area. Common problems included study staff being aware of which patients received which treatments and exclusion of patient data from the analysis without adequate explanation.

Interestingly, the authors say “simple and inexpensive methodological adjustments” could reduce the risk of bias in half of the trials that they found to have at least one methodological shortfall.

So, if many weaknesses in study design and analysis are correctable—often with limited extra effort—then why do we have this problem in the first place? The BMJ study identified two possible explanations:

• Poor reporting of study methods or results in the published paper and failure to present all relevant elements needed to assess the validity of the methods and results.
• A lack of communication between methodologists, statisticians, and health researchers in the planning, conduct, and analysis of research studies

PCORI’s Efforts to Strengthen Research Methods

We at PCORI are keenly aware of these issues. We’re committed to funding studies that will make high-quality, useful information available to help patients, caregivers, clinicians, employers, insurers, and policy makers make informed health decisions. If we can’t assure our stakeholders that the studies we support are methodologically sound, then we can’t expect them to trust the results.

What are we doing to meet this challenge? First, PCORI’s CER Methods and Infrastructure program and Methodology Committee are working together to fulfill their legislated mission to “develop and improve the science and methods of comparative clinical effectiveness research.”

The Methodology Committee provides guidance about the appropriate use of methods in patient-centered outcomes research and establishes priorities to address gaps in these research methods or their applications. Our Methods program then funds projects that address these gaps by developing and improving methods for research that are responsive to the needs of patients and other stakeholders. Furthermore, all PCORI-funded research must adhere to the rigorous methodological standards recommended by the committee, as outlined in the PCORI Methodology Report. Another group, PCORI’s Advisory Panel on Clinical Trials, provides advice and oversight to PCORI on clinical trial selection, design, and implementation.

To ensure collaboration between health researchers and methodologists, PCORI’s Merit Review process assesses studies on their technical merit and reviews research teams to ensure that all needed expertise is present from the outset. Once a project is funded, we require that it carefully document the research methods used, fully report findings, and make the results publicly available.

We are also focused on efforts to promote data sharing, asking awardees to submit a plan as part of their application outlining how they would make the data generated by their studies widely available so other researchers can assess the validity of results. Data sharing is one of the elements of the open science policy that PCORI is now developing.

Finally, we’ve been piloting a system to further improve the methodological rigor of funded research through a methods consultation process. In this process, methodological experts review applications and provide feedback and recommendations. We have implemented a methods consultation process for applications submitted to our Pragmatic Clinical Studies funding announcements, and we may expand it to other aspects of PCORI’s work.

Getting the methods—and ultimately the research— right is not just a core commitment of PCORI but the very foundation upon which CER and PCOR are built. The PCORI website contains more information on our work to promote strong methodology standards and our funded projects to accelerate PCOR and methodological research. 

We welcome your input on the methods work we’re doing. Please send comments to info@pcori.org.

* We would like to thank Steve Goodman, Vice-Chair of the PCORI Methodology Committee, for his contributions to and review of this post.