Enhancing the PCORI Reviewer Experience: Turning Feedback into Action

PCORI's unique process for research application review brings together people of diverse backgrounds and experiences. When we established this approach more than a year ago, we knew all of the scientists, patients, and other healthcare stakeholders who would volunteer as reviewers would have important perspectives to contribute. What we didn’t know was whether they’d work well together or whether the process would be effective.

We needn’t have worried.

Our reviewers respect each others’ perspectives and demonstrate a consistently collaborative spirit while completing the admittedly difficult work we’ve asked of them. We see a great deal of promise in this type of review process for research proposals designed to answer questions important to patients and those who care for them.

“What PCORI is doing with its research is empowering patients to be a part of the research questions that are important for their quality of life and care over the long term,” says Vernal Branch, a patient advocate from Virginia who has served as a reviewer.

Scientists, patients, or other stakeholders share what inspired them to become a reviewer on our Meet the Reviewers page.

Our reviewers have impressed us in another way—by offering thoughtful, honest feedback on our review process, which is admittedly a work in progress. Their suggestions for improvement are clear, helpful, and sometimes challenging. Greg Sawchyn, an Ohio physician and former reviewer, perhaps put it best when he said the challenge we face in refining the process is to continue “building the plane while we’re flying it.”

During a public webinar on October 30, we were pleased to share more of that feedback. We presented results of a reviewer survey, and three reviewers—Sawchyn, Branch, and Julie Panepinto, a pediatric hematologist and researcher from Wisconsin—gave first-hand accounts of the strengths of the process and how it can be improved. We also outlined our plan for implementing some of those suggested improvements and other refinements.

Survey Says…

We began by describing the results of a survey we’ve conducted after each round of reviews. The data provide an excellent gauge of reviewer perceptions; more than 90 percent of our reviewers responded.

Reviewers had many positive impressions about their experiences, especially during the in-person meetings where scientists, patients, and other stakeholders come together to discuss their preliminary evaluations of research proposals. Eighty-five percent of patient reviewers “strongly” or “somewhat” agreed that different points of view were appropriately aired during the meetings, a number that jumped to 90 percent for scientists and 100 percent for other stakeholders.

“What I was most struck by is that having other stakeholders involved really brought perspectives to research review that have been not present in other situations where I’ve reviewed,” said Panepinto.

The survey also pointed out an asymmetry that can be a challenge in bringing together patients and scientists. More than 90 percent of patient reviewers “somewhat” or “strongly” agreed that scientific reviewers brought valuable input to the discussion. But only 75 percent of the scientists said that the patient reviewers added value.

Our task is to help the scientific community better understand the importance of including patient perspectives in the review process. Similarly, we must be vigilant to ensure that patients have the tools to contribute the expertise that comes from first-hand experience with a particular disease or condition.

Improving the Process

Through the survey and interviews with former reviewers, we identified themes for improvement. No reviewer objected to working hard—but they offered ways to make the process less burdensome.

Foremost, they noted that the process demands a substantial time commitment: training, conducting an initial online review of proposals, then attending an in-person session. To be more efficient, we’ve started creating standing panels in which reviewers will use their training and experience for multiple cycles. We believe this is an effective way to leverage reviewers’ developed understanding of the review process.

Another adjustment: simplifying the application-evaluation criteria, which many reviewers said they sometimes found redundant. We condensed our original eight criteria to five, which include our unique requirements for proposals. Finally, we continue to evaluate and refine our training for new reviewers to ensure that all are adequately prepared to share their valuable perspectives.

A Rewarding Experience

Although we recognize the need to enhance the reviewer experience, we are also encouraged by the positive feedback we got. “This was the most positive, collaborative review process that I've participated [in],” one reviewer told us. Another said, “It was very rewarding to be part of the process, and I truly believe it resulted in a more substantial and ultimately meritorious consideration of the applications.”

Overall, more than 95 percent of reviewers said they’d be willing to participate again.

We hope you’ll join them. We continue to need interested, committed scientists, patients, and other healthcare stakeholders to serve as reviewers for our broad funding announcements. We’ll also need reviewers for the one-time panels that review applications resulting from our increasing number of topic-specific funding announcements.

Branch had a piece of advice for anyone who might consider applying. “Relax,” she said. “Don’t be afraid to ask questions, especially from a patient’s point of view. I learn something every time I come to a PCORI merit review meeting.”

If you are interested in becoming a reviewer and have not previously applied, please visit our merit review page to find out how. If you’d like to hear what some of our reviewers have had to say about their experience, you can meet several of them here. And if you’ve already served as a reviewer, and especially if you took the time to provide feedback on your experience, thanks for your support and suggestions for how we can improve our process. We share with you the goal of funding high-quality comparative effectiveness research designed to help patients and those who care for them make better-informed healthcare decisions.