Blog

Open Science: PCORI's Efforts to Make Study Results and Data More Widely Available

Published: Jan. 21, 2015

This has been an important time in the ongoing discussion about open science: the effort to provide broader and more meaningful public and professional access to the results and the data from clinical research—particularly from research funded with public dollars.

The latest development in this conversation was the January 14 release of the final report by the Institute of Medicine's (IOM) Committee on Strategies for Responsible Sharing of Clinical Trial Data. The report outlines data-sharing principles, activities, and strategies for a variety of stakeholders, including investigators, clinical trial participants, funders, and sponsors. Previously, the White House Office of Science and Technology Policy had directed federal agencies that make large investments in scientific research to develop plans to support increased public access to publications and digital data.

Also, the Yale Open Data Access Project this week announced that it now has authority to begin release of Johnson and Johnson medical device clinical trial data from 2014 forward, augmenting the existing drug-data release requirements and representing the first-ever commitment to the broad release of device data.

We are very pleased to see these advances. The IOM report contains a number of suggestions that PCORI has already been considering. Through affiliations that two of us had with the IOM Committee (SG as a committee member and author, HK as a reviewer), as well as our work with PCORI, we are keenly aware of the opportunities and imperatives the report notes.

The IOM report describes not just the importance and value of data sharing but also the complexity. Indeed, we and other PCORI staff have been working in recent months to develop a framework for PCORI to address these key “open science” questions and have been grappling with those challenges that the IOM report has laid out. Recent actions of the National Institutes of Health, Gates Foundation, industry, and others will continue to inform PCORI’s policy on data transparency, sharing, and reproducibility. Our goal is to be a leader in this area.

As a major funder of patient-centered comparative effectiveness research, PCORI is well positioned to contribute to the public good by increasing the availability of clinical research findings and the data that generated the findings. Open science facilitates a broad range of activities, notably early release of findings in formats that are accessible to the public; increased transparency, including the publishing of study protocols and final reports; and data sharing, which allows researchers and funders to verify other teams’ findings and conduct new analyses using data already collected.

These activities increase the integrity of results and thus the likelihood of their influencing clinical practice. They also will allow PCORI and others to leverage research investments by providing opportunities for scientists to address different questions with the same dataset and to learn from the detailed information about how the study was conducted.

PCORI’s approach to open science has four primary goals:

  • Promote timely reporting of results of PCORI-funded studies
  • Prevent publication bias for PCORI-funded studies by releasing results of studies with negative as well as positive findings
  • Facilitate reproduction of original analyses to increase the integrity of PCORI-funded research findings
  • Promote data sharing to enable conduct of additional analyses using data from PCORI-funded studies, thereby augmenting the knowledge generated from the original study

Alignment with Plans for Peer Review and Early Release of Findings

PCORI’s first two open science goals, related to release of research findings, have been addressed in PCORI’s proposal to meet its legal mandate to peer-review and make primary research findings publicly available. This plan, made available for public comment from mid-September through early November 2014, calls for timely posting of the research table of results generated by ClinicalTrials.gov, which will also be shared on PCORI’s website; for posting of peer-reviewed abstracts, both scientific and lay versions, on PCORI’s website; and for posting of full, final reports on PCORI’s website within 12 months of the end of data analyses.

The requirement for posting of full final reports within 12 months of the completion of data analyses, which is unique to PCORI at this time, is intended to reduce risks of publication bias. Final reports will be available on the PCORI and PubMed websites, regardless of whether the study’s findings were positive, negative, or inconclusive.

We’re now reviewing the input we received during the public comment period. We plan to recommend revisions to our draft policy based on those comments when our Board of Governors meets in late February 2015.

Drafting a Plan for Study Reproduction and Sharing of Data

PCORI’s Open Science Work Group, convened last year, is composed of Board and Methodology Committee members as well as PCORI staff and outside experts. Two of us (SG and HK), as co-leaders of this group, developed a draft framework highlighting opportunities for PCORI to encourage open science. On August 26, the group presented key components of its plan to the Board.

Under the framework, which still must work its way through PCORI’s policy-development process before it becomes final, PCORI would require the following, in addition to what is included in PCORI’s proposal to meet its legal mandate to peer review and make primary research findings publicly available.

Activities to promote study reproduction and data sharing:

  • All awardees’ proposals include explicit sharing of data and metadata* (and reproducibility plan and associated budget)
  • Initial and final study protocols be posted, at the conclusion of the study or contract, on both PCORI.org and another stable URL
  • Awardees prepare their data for requests for sharing and/or reproduction
  • PCORI arranges for storage of data and relevant metadata for selected projects in a data repository with a stable URL

*Study data and metadata would include the protocol, case report forms, statistical code, data dictionary, and manual of operations, if there is one.

Some of the components of the proposed open science plan are already in place. Currently, PCORI asks applicants for research funding to propose a plan for data sharing and encourages awardees to pursue avenues to openness in research and freedom of access to underlying data (see page 14 of our application guidelines and pages 8–9 in our sample contract).

Forming PCORI Policy

We know PCORI can do a great deal to promote data sharing to magnify the benefits of our funded research. In the coming months, the Board will consider a draft policy, based on the framework proposed by the Open Science Work Group and informed by the IOM Report and what we have learned from our partners in the field. We have also benefitted from iterative feedback from our guiding Board strategy committee, the Research Transformation Committee.

We are beginning to have data from our funded primary research projects to share, and we want to ensure that PCORI awardees have a clear understanding of the ways in which this policy will affect their applications and project plans. To this end, we are working to develop guidelines to answer applicants’ operational questions, for example, indicating how applicants can prepare for data sharing. Additionally, we are considering how access to PCORI study data will be facilitated and managed.

Over the coming months, we plan to share our progress with you and the broader community as we further develop our open science initiatives. We look forward to receiving your comments on our plans.

Goodman was a member of the IOM Committee on Strategies for Responsible Sharing of Clinical Trial Data and is also the Vice Chair of the PCORI Methodology Committee and Associate Dean for Clinical and Translational Research at the Stanford University School of Medicine

Krumholz was a reviewer for the final IOM report and is a member of the PCORI Board of Governors, a Professor of Medicine and Epidemiology and Public Health at Yale University School of Medicine, the Director of the Yale Open Data Access Project, and a practicing cardiologist