Whole proposal reviewer synopsis - fund or no fund?

When a reviewer completes the review of a proposal, it might be useful to encourage them to synthesize their most cogent feedback over the whole thing as a comment - as if it were a blog post with a comment section. When you get to the bottom, the system could ask for those more general thoughts, perhaps with a specific prompt of: "would you fund this as it stands, why or why not?" That would help encourage the reviewer to crystalize their most important advice.

Great feedback Ben! This clearly seems to be a missing feature.

I think what we'll do is add a 3rd step in the review process. For now this will just have a text entry field, but in the future this could have various ratings fields depending on the situation. I think it's best to have this review summary posted as a new discussion thread. This will allow people to reply to it and proceed with a discussion if they want.

The alternative would be to post all the summaries in the same thread, or just have them as a separate thing that cannot be directly replied to.

@b_good @timputman @andrewsu if you guys would like to provide a review summary for @tongli's proposal you can now do so.

For now we've added this as a last step in the review process. Let me know what you think. My apologies for the slowness on getting this done.

@jspauld
I think some sort of final conclusion panel would also be really helpful. Ideally the ratings assigned by each reviewer for the various categories would be presented in a table format, so that the reviewers' responses can be compared to one another. Some more general comments about the changes which need to be made would also be helpful.

Thanks great feedback. We will try to add ratings that replicate the criteria that the actual reviewers will be considering.

I'd consider removing the current prompt and replacing it with either:

• provide your general feedback on the proposal here
• a prompt which is created by the proposal authors

In particular, I would remove "would you fund this proposal as it stands?" Since the reviewer is not in a position to actually fund the project, it's a moot point. I think the feedback is the valuable aspect of a review, and it's best to avoid forcing reviewers into making a subjective judgement that could have unintended or negative consequences. I believe the current standard for journal peer review is to omit your recommendation to the editor from the review body. I think it's fine to have reviewers score proposals based on metrics, but I see less benefit to the fund/no-fund prompt.

I think I agree with you. The intention was to get the reviewer in a frame of mind that the actual reviewers will be in — get them thinking about any major flaws that would prevent it being funded.