233.413.1644. No, It's Not a Phone Number

Welcome to "On Contracts," a new feature of the PCORI Blog. I hope you'll find this feature a useful source of information about how our applications are evaluated, how we operate, and how we're committed to "research done differently." I welcome your questions. Email me at contracts@pcori.org; include “On Contracts” in your subject line.

In early April, a dedicated group of 233 reviewers met in Baltimore to evaluate 413 applications we received during the PCORI Funding Announcements cycle that closed on December 17, 2012. In eight review panel sessions, scientists, patients and other stakeholders completed a total of 1,644 written critiques and discussed the applications’ strengths and weakness as measured against PCORI’s merit review criteria. All of us at PCORI truly admire and celebrate their dedication, passion and professionalism.

These reviewers came from nearly all 50 states (see map). Nearly three-quarters took paid or unpaid leave from their jobs to attend. More than one-third had to make alternate caregiving plans for family members.  And yet they came.  One patient reviewer told us, “it does take commitment because you’re not there just as window dressing. You’re there to make a difference. So the timing commitment to me was worthwhile.”  All told, reviewers volunteered some 10,000 hours of their valuable time to help ensure that the research applications submitted to us underwent a rigorous, fair, and patient-centered review. We’re thankful for their efforts to help us fund “research done differently.

First-time observers remarked on the professionalism and commitment of the reviewers and the high quality of the discussion. Reviewers themselves remarked on the sincerity and insightfulness of their peers and the high quality of the applications. Veteran PCORI reviewers—those who participated in our inaugural funding cycle—noted how far our merit review has come —in process, training, and logistics—and how quickly.  For this review cycle, for example, there was comprehensive online training, reviewer town halls and in-person mock reviews.  Patient and other stakeholder reviewers received additional support via Patient/Stakeholder Review Officers (PSROs) and a cohort of reviewer mentors—as well as 24/7 access to the digital PCORI Patient-Stakeholder Reviewer Community.  “When I ran into confusion or felt overwhelmed,” one patient reviewer said, “there were mentors there to support me every step of the way.”  Reviewers, mentors, and PSROs used the community to micro-blog throughout the process; they sought feedback, shared tips and tricks for effective review, and made transportation arrangements.

Reviewer commitment did not end with the last in-person session on April 4th. They also responded to a survey and provide detailed written feedback.  Their consensus: review panel chairs ensured that different viewpoints were heard and both scientific and patient and other stakeholder reviewers made valuable contributions. We earned high marks for pre-panel training, town halls, the mentor program, and in-person “mock” review training. Here is some specific feedback:

• Two-thirds of reviewers attended the town halls; the majority who attended found them useful
• The vast majority of reviewers attended pre-panel training and found it useful;
• Most patient and other stakeholder reviewers tried our online community application and two-thirds found it helpful;
• More than three-quarters of patient and other stakeholder reviewers found their mentors helpful; and
• More than three-quarters found the in-person training useful.

Was it perfect?  No. Reviewers encouraged us to be more transparent about the administrative triage process, which is when staff exclude applications that do not meet our basic funding requirements, such as lacking a comparator or including cost-effectiveness as a primary aim. Participants also called for more detailed explanations of scoring and voiced their concerns about the number of applications they reviewed.  We’re working to improve this process and try to reduce the number of applications received in the future.

We welcomed these and other helpful criticisms. Over the next few months, we will continue to work to improve our merit review process, and we’ll implement some of your suggestions for the next round.

Until then, I encourage those of you who attended this latest review session to keep providing feedback. If you’ve not served as a reviewer but would like to apply, I urge you to do so. If you want to know more about how we assess proposals for research, you can find additional information here. You can also get more information about our training resources for reviewers. And if you have any additional comments or questions, please email us here.

At the point that this blog was published, author Michael R. Dueñas was PCORI's Director of Contracts Management.