Single post
How to read, structure and apply proposal assessors/referee comments
1. Read the feedback, but take some time to cool down and to digest it.
At least for me, when I read a strong criticism on a project and it was not funded, I get upset and even offended. This is especially true if I believe it was good. So much work and thinking that went into it, and the assessors just didn't get it….
My first instinct is to rip out a page and write down what they got wrong if I can write a rebuttal. It's not a good idea...
2. Own the feedback
Assessors have a reason for what they critic and everyone we all have a logic and are right inside our own little skull kingdom. When I let feedback sit for a day and reread it, I often see and agree with the referee's point.
Most significantly, if they get it wrong, it is usually our fault. We just did not explain it well enough, supplied too much or too little information, or arranged the proposals incorrectly. I now always read the feedback a second time with the mindset that what is said there is correct, and I am incorrect.
What can I learn and how will it help me better my proposal?
3. Structure the feedback: “must be addressed”, “should be addressed”, “does not need to be addressed”
After taking a step back and owing it, I return into defence mode and structure the feedback. This is useful for seeing the main arguments and for writing a rebid or response.
If I find a point being criticised more than once, I copy the comments and put it in the “must be addressed” category since it informs me that something is obviously incorrect/missing in the proposal. Also, if there is a comment that entirely undermines the grant, I will place it in this category.
It is usually difficult to respond to these "sinkers." A common issue I notice is a misunderstanding of the core fund philosophy, such as examining the grant like a journal manuscript or criticizing a Marsden application for not having or a specific plan for achieving “impact”. Comments claiming that the research has already been done or that anything critical is lacking are also difficult to respond. It depends on the circumstances, but I usually try to reformulate the perceived problem and use as much evidence as possible to refute it, such as new research, publications, or a reminder of the proposal guidelines. It is important not to criticise or attack the reviewer rather to provide facts, such as stating that “there was a misunderstanding concerning xyz due to”.
The trivial stuff, side notes, etc. I put it the “does not need to be addressed” category, anything in between in the “should be addressed” category.
Be aware that “should” and not need category can be dangerous in a Marsden rebuttal since the quality or missing rebuttal is included in the grades see page 8 in the link below.
However, often there is not enough room to address everything. This system also gives you an indication how much room you should spend on each point and where you can cluster answers.
I recommend and if time allows, I will go over this list with the Associate Investigators and other trusted individuals (see also 6) and create a final list.
Link to Marsden Council Guidelines: Link
4. Be clear about your audience
As always, being aware of your audience and writing for them is essential.
In a Marsden reply, for example, your rebuttal will get to the panel members rather than the referees. Your panel's "champion(s)," means the people who will present and/or best-case fight for your grant, are especially critical. Provide them with enough ammunition to defend your proposition if they still liked your full proposal.
As said in the Marsden Panel and Council Guidelines:
"Where referees disagree, the Council and panel members must use their own judgment in determining which referee reports to emphasise and what score to assign. These deliberations should be guided by considerations such as:
• the member's own level of expertise on the subject
• the comments made by referees to explain their grades
• the relative competencies of the referees
• the responses by applicants to the referees’ comments
• possible conflicts of interest."
There is no rebuttal in an MBIE rebid, but you may get some of the same assessors as last year. They might remember what they didn't like the last time, so address their concerns to the letter in a rebid. This is usually easier because you can overhaul, align, and restructure your rebid.
5. Use positive words and if the feedback is stellar remind them of it
If you address a criticism, phrase it positively e.g. rather than writing "referee 2 criticised the lack of abc," say "referee 2 proposed that in experiment yxz we could add abc." It is beneficial to repeat the favourable comments from the referees, but only if there is adequate space and it organically flows into the text. Again, you must provide enough ammunition for panel members who support your proposal to defend it. There will be a trade-off between good referee comments that reinforce the opinion of your champion(s) and more important a forceful defence of the perceived weak parts.
6. Get feedback on your feedback/rebuttal
Sounds silly, but it is valuable especially for a rebuttal to get more feedback. At least include your Associate Investigators but also check that with your research office and maybe trusted colleagues’ who have won a Marden or know the system very well. They will be busy, so ask for their help and timeline early. I always wonder how much a rebuttal can accomplish but you should give it your best shot. If the proposal is rejected at least, you will have a solid starting point for next year's rebid.
For a more in-depth look, check out this article on panel dynamics, rebuttals I came across recently: Article