Volkswagen Foundation experiments with new peer review method
Coloures-Pic - stock.adobe.com
In the current call of the funding initiative "Open Up – New Research Spaces for the Humanities and Cultural Studies", the foundation is testing a new review process for the first time: "Distributed Peer Review". In this interview, Dr. Hanna Denecke, team leader in the "Exploration" profile area, explains the experiment.
In a Distributed Peer Review, or DPR for short, the applicants are supposed to review each other. How exactly does this work?
Hanna Denecke: It's not that complicated: When submitting their own proposal, applicants agree to review other applications submitted for the same funding program. At the same time, of course, they also agree to have their own proposals reviewed by fellow applicants.
The core of the exercise is to test how to make research funding more efficient, effective, and equitable.
Is this a new procedure or do other science funding organizations already have experience with it?
The method is not entirely new, but so far it has only been used to a very limited extent. The European Southern Observatory (ESO), for example, allocates observation time at its telescopes via DPR. At the Netherlands Research Council (NWO), DPR is used in smaller funding programs such as the so-called XS programs in the natural sciences and the humanities and social sciences.
At the Foundation, we are not afraid to try out such a new and unusual approach. We want to use the process for larger projects and where there is a high degree of disciplinary heterogeneity, so we definitely see ourselves as pioneers in this respect. We are particularly interested in learning more about the review process: What are the advantages? What are the challenges? And how do applicants deal with them?
What specific benefits does the Foundation hope to gain from the new format?
We believe that DPR has great potential: there is a larger pool of reviewers available than in a conventional panel review. This means that each application can be reviewed multiple times, which improves quality assurance. And there is the added benefit that applicants receive more feedback on their research proposals. The procedure could also contribute to making administrative processes faster and more efficient. We anticipate that this will also increase applicant satisfaction.
Can the Foundation guarantee that the results are comparable with those of conventional review procedures?
Admittedly, there are still many unanswered questions. Therefore, in consultation with the Board of Trustees, we have decided to experiment with DPR in parallel with the existing panel review process. We are providing additional funding for this purpose. This means that funding recommendations will ultimately result from both the panel review process and DPR.
Hanna Denecke heads the "Exploration" profile area, in which the foundation supports creative, daring and curious scientists - and is now testing the new "Distributed Peer Review" review process for the first time.
That sounds pretty demanding…
Yes, it means a lot of extra work. But by using parallel procedures we can formally compare the two review processes in terms of quality, efficiency, and consistency. We are also supported in this by members of the Research on Research Institute (RoRI): Tom Stafford and Stephen Pinfield from the University of Sheffield.
Our intention is not to simply replicate the results of the panel review process – and I certainly don't expect the results of the two processes to be completely identical. That wouldn't happen if two separate panel procedures were run in parallel. And if it turns out that the two processes select different applications, this should not be seen as a shortcoming. At the very least, it would shed light on the relevance of the discussions and consensus-building in the jury meetings that accompany conventional panel review. In the case of DPR, the results are not evaluated purely quantitatively, but are supplemented by the qualitative evaluations contributed by the participants.
And what will you do with the results?
We will, of course, be open to discussion and make all the results publicly available. The core of the exercise is to test how to make research funding more efficient, effective, and equitable. We look forward to contributing to the wider debate on the review system.
This unusual call also means more work for the applicants – how will they benefit?
We are aware that we are asking a bit more of those who participate in this particular "Open Up" call. After all, they commit themselves to reviewing up to 5 different applications. This sounds like a lot, but these are short proposals, and the review will be based on standardized evaluation forms. It should also be mentioned that for this call we only accept applications in English.
Notwithstanding, one clear advantage is that by taking part in this experiment, participants enjoy the unique possibility to have two chances to be funded – one via the traditional panel review and another through the DPR mechanism. In addition, they may find the opportunity to review other proposals an interesting and informative experience, especially for early career researchers. And the opportunity to take part in what we consider to be a highly relevant experiment for innovation in the peer review system can be a stimulating experience.
Do you expect reservations from the scientific community?
As with any review process, yes. Typical concerns are that applicants acting as reviewers will not have the necessary expertise. However, even when recruiting reviewers for the panel procedure, we are not usually able to cover the full range of specific expertise required. There may also be concerns about theft of ideas. Here, we expect all those involved to treat the applications confidentially. Another concern could be that applicants may systematically submit lower ratings in order to increase the chances of their own proposal being selected. We take this into account wherever possible, for example when allocating the applications to be reviewed among the participants. The applications are also reviewed anonymously – double blind – so that only the exploratory nature and quality of the research idea are assessed.
As both review procedures run in parallel, we believe that neither the applicants nor the disciplines will be disadvantaged by the new process. Nevertheless, throughout the whole procedure, such concerns will be addressed and taken into account on an ongoing basis.
Will the Foundation use DRP in other initiatives going forward?
The scientific monitoring and evaluation of our experiment will help us to identify possible future applications, challenges, and opportunities. The potential is certainly there.
