Has our scientific system been optimized by an evolutionary process?

@caseygreene made an inline comment on this proposal text:

Given a wide variety of problems in our current scientific system, isn't it worth experimenting with ways we might improve it?

He says:

Think of evolution: sure, existing living systems are bizarre and often seem sub-optimal but if you just start making changes to the genetic code willy-nilly, it's probably not going to make the system more efficient. How are you going to measure your impact? How will you know if this is improving or harming the process of scientific research? How you will you adjust if it turns out that items need to be changed?

There's two points I want to make:

1. I see no evidence or reason to think that our scientific system has been optimized by an evolutionary process. Living systems don't have this many flaws. What is the mechanism by which evolutionary forces are working? I will grant you that things are slowly moving in the right direction. But damn. Why has it taken science funders like 20 years to figure out that requiring scientists to publish the results of their research open access would be a good thing?

2. We are not making changes willy-nilly. We explained why having grant proposals reviewed in the open would be a good thing. Then we proposed funders create grants program that require openly posted proposals, and we proposed funders create an incentive for scientists everywhere to share feedback and insights on those proposals. It's all very logical. Science funders should create incentivizes for behavior that is good for science and society as a whole.

Let me know if you have further thoughts.

My main point was less around the process that generated our current system, and more towards an interest in metrics that you will use to assess your changes. How are you going to measure the impact of thinklab? How will you determine whether its influence is positive or negative for the practice of science? I think these are key elements to address in the proposal.

It's a good question. Unfortunately, I'm not sure there is an obvious answer. I think it will just be a general sense amongst participants and observers of how much impact the research is having.

I do think scientific publishing, like all systems, evolves. However, I don't think this discussion will add to the proposal or @caseygreene's comment.

I read @caseygreene's comment as the proposal doesn't provide convincing evidence that the changes will be beneficial and doesn't offer a framework for assessing the benefits. This comment falls in line with my general feedback that the proposal needs to be more evidence based and quantitative.

@jspauld, regarding "we are not making changes willy-nilly" — you need to establish this point in the proposal not via discussion, since the proposal alone will determine the winner of the open science prize. Reviewer feedback and counterpoints only matter if they affect the proposal. In other words, @caseygreene's comment has no relevance beyond how it instructs you to improve the proposal.

Regarding measuring impact:

I don't believe there is any good evidence based or quantitative metric for determining the impact that Thinklab may have. Measuring scientific impact is just really difficult. The idea that we measure a scientists impact by the number of papers they publish is at the very core of the problem we are trying to solve. So we're certainly not going to use that metric!

On Thinklab we measure impact via direct peer assessment of the value added by each contribution. The best way I can imagine that we could measure the impact of Thinklab as a whole would be to simply ask the people that have used and observed the platform. But we are not going to be able to ask them until we actually try it! That's why I highlighted the importance of doing experiments at the end of the proposal.