Observations from PCORI Application Reviewers
We posed a series of questions to the three reviewers who participated in our public webinar on October 30 to get a sense of their experiences evaluating PCORI research proposals. Here is what Vernal Branch; Gregory Sawchyn, MD, MBA, and Julie A. Panepinto, MD, MSPH, had to say. The views expressed in this blog are those of the reviewers and not necessarily those of PCORI.
What impact do you hope to have as a reviewer?
Vernal Branch: My impact as a reviewer is to help bring the voice of the patient to the research that is being funded and to make sure the voices of those with diseases have improved quality of life and better health outcomes.
Gregory Sawchyn, MD, MBA: I hope that my creativity and analytical skill will help ensure that the best research is funded and that projects with potential, but not ready for prime time, get useful feedback to make future applications great.
Julie A. Panepinto, MD, MSPH: As an applicant, I hope to get a reviewer who has the background and expertise to review my work; so as a reviewer, I hope I have the qualifications and bring the knowledge to be critical of an application and ultimately help the applicant to be successful.
What have you found to be unique about the experience with PCORI, compared with your other work in health, advocacy, and research?
Branch: My experience with PCORI has been truly collaborative, being able to offer input from the beginning about the research questions; whereas with other work in this arena, there has not always been that type of transparency.
Sawchyn: The patient involvement is the biggest difference. Clinicians and researchers have always had a close relationship. Both groups are comfortable with that relationship. The addition of patients makes some researchers and clinicians a bit squeamish, but that is a good thing. New ideas and creativity are a result, and we need that to advance care for patients.
Panepinto: The main difference is the use of other stakeholders as reviewers, whether they represent policy or patient-advocacy groups. Stakeholder reviewers bring necessary insight because the work proposed often directly involves those stakeholders.
What do you like about PCORI’s review process?
Branch: The PCORI review process is organized with online training and webinars to help reviewers understand what is being asked of them. Just before the actual review, there is also an orientation that I think helps to clarify things for the reviewers by letting them know what will happen in their individual panels.
Sawchyn: I like the dialogue that develops in the in-person merit review where people from different backgrounds and expertise have the opportunity to share their thoughts. I also like the exchange of ideas that is possible with written feedback to applicants.
Panepinto: I like that PCORI is following logistics similar to what I had previously experienced and I think works well. Although I’m not used to it, I like that PCORI has brought the patient, stakeholder, and scientist researchers around the same table to exchange information.
What improvements have you seen in the process?
Branch: Improvements have been made in the criteria of the written critiques. In the beginning there was some duplication, but with comments from reviewers, PCORI made changes and streamlined the criteria.
Sawchyn: PCORI, as a fairly new organization, has done what many start-ups do. It employs an iterative process to learn as it goes. I have been impressed with its willingness to embrace change that will lead to better results and prevent the process from getting stale and bogged down.
Panepinto: PCORI is continuously listening to the reviewers and the applicants and critically evaluating itself. Between the first and second time I reviewed for PCORI, the process improved logistically for both the reviewers and the applicants.
What would you like to see done differently?
Branch: Because some of the reviewers come from communities around the country from which PCORI would like to get applications, I would like there to be training on how to use reviewers to present calls for applications.
Sawchyn: I would like to see PCORI hold in-person review meetings in different settings, rather than just East Coast cities. I think PCORI and researchers could learn a lot from holding the meetings in new locations and meeting new people.
Panepinto: I would like to see a standing committee with rotating members to add stability and normalize the process. PCORI is already doing this, and I think it will be a great change. It will let reviewers get used to each other and also allow applicants to have the same reviewers consider resubmitted applications.
What do you want others to know about PCORI and how it evaluates research proposals?
Branch: PCORI is unique in offering a chance for patients and other stakeholders to come together to help them with advice and also in the transparency of their funding for research.
Sawchyn: PCORI does a great job with packaging the submitted proposals for review. Patient and stakeholder reviewers may not be experts in the area the research proposal covers, but the application structure ensures that reviewers receive sufficient background to do the reviews.
Panepinto: Usually when I’m asked about PCORI, the most important message is that PCORI applications and proposal review are not the same as those of other agencies. It’s important to pay attention to what PCORI wants to see in applications, particularly requirements related to patient engagement.
What advice would you give to a new reviewer or individuals interested in becoming a reviewer?
Branch: This is a chance, as a reviewer, to have an impact on research that will enhance people’s lives and help with their decision making, so they can choose treatment options that will add to their quality of life.
Sawchyn: Jump in head first and make your opinions known. Don’t be intimidated by others’ titles or the number of letters they have after their names. Fresh eyes can see things from a perspective that someone with all of the degrees in the world may miss.
Panepinto: As a new reviewer, it’s extremely important to recognize the privilege of serving and take it very seriously. Know you may have to put more time into the reviews as you start. Don’t be afraid to seek out advice from more seasoned reviewers. At the in-person review, make your voice heard, even if your view is different from that of other reviewers.
In your experience, how did the mix of scientist, stakeholder, and patient reviewers work out?
Branch: Everyone who was at the table during this process brought, along with expertise and experience, a unique perspective to the application reviews. I was really impressed by the panels on which I have participated, especially the collaborative way in which the panel members interacted.
Sawchyn: I liked the mix, and I liked that reviewers did not have titles or roles defined on name cards at the in-person review. Everyone was on equal footing.
Panepinto: It was important to have patients and other stakeholders with a voice equal to that of the scientific reviewers. Also, as a scientific reviewer, it was helpful to have another scientific reviewer present on each application. Scientific reviewers aren’t necessarily experts on everything contained within the application: we have strengths in some areas and not others.
Do you feel that your contributions were valued by fellow reviewers and staff?
Branch: Yes, I think my opinions were of value; once our discussions were over, I could see that I had changed opinions.
Sawchyn: I think my views were valued, and I think I had a positive influence on other reviewers who may have had more expertise in the area being reviewed than I did.
Panepinto: Yes, everyone allowed voices to be heard and views to be stated and discussed. That was helpful to all involved and allowed our group to come to consensus when we were finalizing the reviews.
If given the opportunity, would you participate in the review process again?
Branch: Most definitely!
Sawchyn: Absolutely.
Panepinto: Absolutely!
Branch, a patient reviewer, is currently the Public Policy Advisor for the Virginia Breast Cancer Foundation, has served on the National Cancer Institute’s Director’s Consumer Liaison Group, and has been a member of the board of directors or an advisor to a variety of patient advocacy groups and research projects
Sawchyn, a stakeholder reviewer, is president of Caduceus Capital, LLC, and senior director of the Clinical Guidance Councils for OhioHealth
Panepinto, a scientist reviewer, is a professor of pediatric hematology; and vice chair of value; and director of the Center for Clinical Effectiveness Research at the Medical College of Wisconsin/Children’s Hospital of Wisconsin
The views expressed in this blog are those of the reviewers and not necessarily those of PCORI.