Thinklab: A platform for open review of research grant proposals

Funding opportunity: The Open Science Prize
Awaiting Funder

Thinklab: A platform for open review of research grant proposals

What is Thinklab?

Thinklab is a platform that facilitates two things:

  1. Open review of research grant proposals
  2. A highly collaborative version of open notebook science

We're creating Thinklab because we believe the interests of science and society are best served if the entire scientific process is open — not just the software, data, and papers produced at the end. And while we're convinced this kind of openness is in the best interests of science as a whole, we understand many scientists will not feel it's in their personal interests. That's why our goal is not just to facilitate open research, it's to actually create incentives for it.

Thinklab is intended for broad use across all of science, including biomedical research. An early version can be seen at thinklab.com.

Why grant proposals should be reviewed in the open

  1. Researchers can get feedback when it's most valuable: before a project begins
    Scientists often work on projects for years at a time. Given this, it's critically important to work on the right thing and have the right approach. Feedback at an early stage — especially when it comes from a cognitively diverse pool of people — has the potential to save incredible amounts of time. It may reveal that an entire project was flawed from conception, or it may light the path towards a much more impactful approach.

  1. Best practices can spread across science faster
    Best practices in science are changing fast. Having proposals reviewed in the open, gives the community a chance to share those best practices — even across disciplines. This could include sharing ideas on making data reusable, making research reproducible, or just recommending tools and methods.

  1. Scientists can learn from and build upon each other's ideas
    Research proposals are valuable scientific outputs in and of themselves. They will often contain some of the most cutting edge ideas in science. Having proposals (and the review discussion related to them) published openly on the internet would create a valuable resource that the entire scientific community can learn from and build upon.

  1. Openness produces more collaboration and less redundancy
    When people know what each other are working on, it creates more possibility for forming collaborations. At the same time, it reduces the redundancy of having multiple research groups working on the same thing in isolation.

In conclusion, open review can improve research plans, reduce wasted resources, spread best practices, accelerate the exchange of ideas, and lead to more collaboration with less redundancy. In short, it should significantly accelerate scientific progress. Many of these ideas have been talked about by Daniel Mitchken: The Transformative Nature of Transparency in Research Funding [1]

But will scientists participate?

While it may be clear that having grant proposals reviewed in the open would be a very good thing for science, the question is: will scientists participate? Won't they be concerned about people stealing their ideas?

The answer is yes, many will be concerned. Scientists currently operate under a system that values the publication of papers in journals above all else. Under this system, keeping ideas secret is seen as a good strategy in the publish-or-perish game — a game that they're forced to play, but never asked to play. In our view, the solution to this predicament is clear: we need to change the rules of the game! We need to align the incentives of scientists with what's in the best interests of science and society as a whole.

Thinklab as a service for science funders

If our goal is to open up grant proposal review, we believe science funders are in the best position to do it. Funders can use the power of the purse to create new incentives and compel adoption of new behaviors. In fact, they're already doing it. Many funders have started to require that the results of the research they fund be made publicly available. That changes the game!

Thinklab wants to help funders take the next step. We want to help funders create grant programs that require openly posted proposals. In addition, we want to help them manage an open peer review process where anyone from the scientific community can share ideas and feedback to help improve research plans.

With that said, we recognize that persuading funders to try a different model of funding will be challenging. Fortunately, there's something we can do in the meantime: we can do everything we can to drive adoption by making Thinklab a valuable service for grant writers directly.

Thinklab as a service for grant writers

Benjamin Good suggested [2] that grant writers could use Thinklab to have their proposals openly reviewed prior to submission for funding. There are a number of benefits to this. In fact, we've used Thinklab to get feedback on the proposal you're reading right now! X reviewers provided feedback through a total of X annotations and X comments. You can take a look at the feedback we received here: thinklab.com/p/thinklabOSP.

Benefits for grant writers

  • Open review can help grant writers improve their proposals and increase their odds of funding. Getting an outside perspective can help make sure the value of a proposed project is being communicated effectively.
  • Open review can help validate the research plan before work begins. With this validation, the research team can be confident that when results come in, they'll be trusted and valued by the community.
  • Engaging the community at such an early stage serves to increase the research team's visibility and connection with the community, leading to more collaboration opportunities.
  • Grant writers can feel better about asking for feedback as there's more in it for the reviewer. Reviewer comments are public, and reviewers are publicly acknowledged based on peer assessment of the value of their feedback.

How Thinklab works

Thinklab is a platform that facilitates open review of grant proposals, and real-time open science. All content and discussion posted to Thinklab is licensed CC-BY. Note that at this point the code that runs Thinklab itself is not open source.

The proposal review system

Design goals:

  1. To have reviewers share independently formed opinions initially.
  2. To enable a real-time discussion between reviewers and proposal authors aimed at improving the proposal.
  3. To reward reviewers by recognizing them based on the value of their contributions.

The review process:

  1. Part 1: Annotate — In this step reviewers are asked to read the proposal and make annotations wherever they have feedback (see Figure 1). To ensure an unbiased review we hide annotations and comments from other users in part 1 and 2.

  1. Part 2: Summarize — Next, reviewers are asked to summarize their most important thoughts and advice. They may also be asked to rate the proposal on various metrics.

  1. Part 3: Discuss — In this step we reveal the discussion that is already taking place. Reviewers use the inline notes they made in part one to join the discussion. Discussion can take place in inline comments, or on separate pages that link back to the proposal.

  1. Part 4: Finalize — After discussing the proposal with the authors and other users, reviewers have the option to update their initial review summary and/or review scores.

When the review process is complete, the proposal authors (or designated reviewers if the review is on behalf of a funder) will rate the value of each reviewer's contributions. These ratings contribute to an impact points system that allows us to highlight the most impactful reviewers.

Figure 1. Proposal annotation options

This menu appears when proposal reviewers select text to make an annotation.

Real-time open science

When a proposal on Thinklab gets funded, the research team has the option to continue their work as an open research project. Researchers can engage the community by sharing ideas, project plans, updates, and questions in real-time as the project progresses. Reviewers that made valuable suggestions during the proposal stage can continue to share ideas and give feedback throughout the project.

The big payoff from open research comes not just from being open, it comes from the real-time collaboration that openness enables. It comes from bringing a cognitively diverse set of people together to tackle challenging problems. In support of this Thinklab is building a system that intelligently direct researcher attention to the exact discussions and problems that match their interests and areas of expertise. These ideas are described by Michael Nielsen in his book: Reinventing Discovery: The New Era of Networked Science [3]

There's one big problem with this vision: there needs to be incentives for scientists to participate. There needs to be an incentive for researchers to share their work, and there needs to be an incentive for outside scientists to share feedback and insights when they have something valuable to add.

Benefits for reviewers/contributors

  • Everything is public
    To highlight the benefits of openness let's consider what will happen to this proposal when it's submitted to The Open Science Prize. We expect it to be discussed in private by three reviewers. Here's a question for those reviewers: wouldn't it be nice if your comments were public and would become part of the scientific record? Wouldn't it be nice if we (and others) could learn from what you said? Thinklab offers reviewers the chance for more impact and more recognition.

  • Users are recognized based on the value of their contributions.
    Thinklab has an impact points system that allows us to highlight the most impactful reviewers or project contributors. Users gain impact points primarily by conducting review and making comments that the community rates as valuable. Contributions are rated by the research team, by users selected by the sponsoring funder, or by the community at large.

Monetary rewards

While we believe our impact points system will help create an incentive, we'd like to point out that it's extremely difficult to get scientists to take time away from their research, and share feedback and ideas on the work of their peers, over the internet [3]. At the same time, we can all recognize how incredibly valuable it would be if they were to do so.

For these reasons, Thinklab proposes that science funders create an additional incentive. We propose that a portion of project grant money is set aside to reward feedback and ideas from scientists in the community. These monetary rewards could be applied at the proposal stage and/or the research stage. The system would piggy-back on the impact points system we've already discussed.

Possible concerns

  • Won't people game the system?
    It's very likely that some people will attempt to game the system. However, we believe it will be relatively easy to detect and prevent strategies such as voting rings between friends. The only way to game the system should be to actually take the time to write comments that peers find valuable.

  • The overjustification effect
    This is where external incentives decrease a person's intrinsic motivation to do something. It's a fascinating concept. However, this effect is only relevant if people have enough intrinsic motivation to do something in the first place. In the case of getting scientists to share feedback and ideas on the work of their peers over the Internet — they're simply not doing it at all. And this is despite many platforms built for this purpose [3].

  • Money in science is always bad
    Many scientists seem to have a gut reaction that tells them introducing money into science is always a bad idea. To those people we would say this: money is already in science. Much of what scientists do is driven by the need to secure money to continue doing their work. We are simply proposing that some money is distributed in a way that creates a different set of incentives.

Finally, we want to emphasize that we consider monetary rewards to be an experiment, and that we see large value in the Thinklab even without them.

What will the prize money be used for?

  • Continued development of our open review system. We will be working in close consultation with both grant writers and reviewers to create the most useful and user friendly service possible.
  • Development of features to enable a pilot program with a funder. This pilot program will likely involve having proposals openly submitted and reviewed though Thinklab.
  • Creation of APIs so our CC-BY content can be more easily accessed.

Alternatives to Thinklab

Google Docs for open review of proposals

If a researcher wants to get feedback on a grant proposal, a common solution is to put it in Google Docs, and send it to colleagues. Here's why Thinklab is better:

  • Thinklab is designed to have peer review and review comments become a persistent, citable part of the scientific record.
  • Thinklab facilitates more substantial discussion by allowing discussions to be posted to a separate page.
  • Thinklab is able to draw more people into the conversation by directing people's attention to the discussion when it matches their interests or areas of expertise.
  • Thinklab rewards participation through an impact points system that is based on peer assessment.

GitHub as an electronic lab notebook

The Open Source Malaria project is using GitHub to openly manage their project. There's a lot to like about GitHub. But here's what Thinklab can offer:

  • Thinklab is designed for science. Adding math is easy. Adding citations by DOI is easy. Each discussion (like a GitHub issue) has a DOI and is citable.
  • When leading an open research project the big challenge is actually getting scientists to participate. Thinklab rewards participation through a system that highlights the most impactful contributors. If needed we can add the additional incentive of monetary rewards.
  • In the domain of scientific discussion there is tremendous value in having a cognitively diverse set of people participate. Thinklab has a system that intelligently directs researcher attention to discussion that is relevant to their interests or areas of expertise. Such a system is not particularly relevant to software development so we wouldn't expect GitHub to be working on it.

Team and resources

Thinklab is founded by Jesse Spaulding. Jesse has a background in the startup world, and has founded and sold several startups. With no formal academic background, Jesse brings a fresh perspective to the challenges facing our scientific system. Jesse has been working with Gleb Pitsevich who has a broad background in mathematics, programming, and web development.

Thinklab is setup as a for-profit and we intend to use this status to attract talented people to help us pursue our mission. In the near term we're looking for a talented individual with a strong academic science or philanthropic background to join the team as a co-founder and lead business and community development.

Conclusion

Thinklab is a bold experiment in open science. We understand there's powerful incentives working against us. But we also understand the future of science is not a world where scientists continue to hoard knowledge and work in silos.

We believe Thinklab has a legitimate shot at opening up grant proposal review, and if we're able to do so, we believe the benefits will be enormous. Perhaps our most compelling argument is this: what we're doing has the potential to affect all of science. Anytime you can make changes that positively affect a system there's potential for massive impact. Given a wide variety of problems [4] in our current scientific system, isn't it worth experimenting with ways we might improve it?

References

0
1.
0
2.
ThinkLab as a vetting system for traditional grants
Benjamin Good, Jesse Spaulding, Jonathan Eisen, Jack Park, Daniel Mietchen, Jonathan Wren, Casey Greene (2015) Thinklab. doi:10.15363/thinklab.d58
3.
0
4.
10 Consequences of a broken scientific reward system
Jesse Spaulding (2015) Thinklab. https://thinklab.org/d/36
0
5.
ThinkLab as a vetting system for traditional grants
Benjamin Good, Jesse Spaulding, Jonathan Eisen, Jack Park, Daniel Mietchen, Jonathan Wren, Casey Greene (2015) Thinklab. doi:10.15363/thinklab.d58
Casey Greene

Just a heads up - the discussion phase UI is not nearly as slick as the single-person UI. I'm seeing a really crowded document where it's hard to connect the notes on the right to the content on the left.

Jesse Spaulding

Yes, I agree — it looks a little cluttered and crazy. With regard to connecting notes and content — you did click them right? Did you mean it's hard to see which text a comment refers to? We could try removing all other highlights when a comment is selected. That should look nice.

Casey Greene

I did not realize that I could right click them. This is helpful to know. Removing other highlights would help to clarify more, but knowing about the right click bit is a huge help already.

Casey Greene

Can you start with: What would the funding process be like with thinklab? Paint a picture of the world for me. What's better in it?

Casey Greene

Would it be helpful to specify these as things that thinklab lets you create: a Thinklab Notebook and a Thinklab Proposal. The notebook provides the tools to perform open and collaborative science, while the proposal provides you with the tools to get broad feedback on a grant.
I always find "facilitates" less concrete than the outputs.

Casey Greene

This is not discussed in the proposal below. If you're not going to talk about it, I'd suggest removing it.

Casey Greene

who is convinced? thinklab or the open science community as a whole?

Jesse Spaulding

Thinklab is convinced :) I take it this wasn't clear?

Casey Greene

How valid is this concern? How valid is it if they are already doing rigorous and reproducible research?

Jesse Spaulding

Hm, I'm not really sure what you're getting at here?

Casey Greene

Is it valid for scientists to feel that openness is not in their interest, or do they feel this as a reaction to the state right now? Is there some downside to openness that even the right incentive structure can't fix? Is that concerned lessened if they are currently using best practices for reproducibility and rigor?

Jesse Spaulding

argh.. I still don't get what you're saying :( Are you suggesting that many scientists feel it is not in their personal interests, but in reality it is? (Because they don't fully appreciate the benefits?) Anyway, the point here is to lead into a statement that Thinklab's goal is to create incentives for scientists so they do feel it is in their interests.

Casey Greene

Sorry - I'm not doing a good job of this. Inherently in science is there some value at some time to a system that doesn't require everything to be open. Does it help people take risks/etc that they wouldn't take in an open system?

An alternative is that most people are closed right now because things are hyper-competitive or that this is the way things have always been.

If the second case is true, then I'm going to look for solutions that maximize openness because they're good for science in the long run. If the former is true, I'm going to look for things that promote openness but don't necessarily require it in all cases.

I don't have an answer. This question just came to mind when reading this part.

Jesse Spaulding

Okay I get it! You mean scientists have reasons for not be open that are valid in the sense that forcing them to be open would be bad for science as a whole. There may be. Although I suspect those benefits are far outweighed by the benefits of being open in most cases. It's something to keep in mind

Daniel Mietchen

This could be resolved by using "by default" as a qualifier for the entire process being open. There are numerous conditions under which openness isn't an option even if the scientists would want it (think patient privacy, locations of near-extinct species or archeological sites in danger of being looted).

Casey Greene

Enable? Do tools that do everything required already exist + you're making better, or do they not exist outside of your platform?

Jesse Spaulding

They exist, but they need to be coupled with a) a points and/or monetary system that rewards participation and b) a system that directs researcher attention to discussion that is relevant to them. I assume you're raising a concern and creating tools that already exist.

Casey Greene

Well, I'm trying to get towards whether or not you're doing a bit more than facilitating. The overall structure of this grant doesn't entirely work for me. It feels like lots of detail about thinklab up front. I want to see up front how the world will be different with thinklab in regular use.

Casey Greene

I think you should lead with this section. Why first.

Casey Greene

Each of these points is somewhat long. Instead, can you flip this to describing what the workflow of open grant applications is like. You could then work in the benefits there. You don't even need to mention thinklab specifically. That could come up later. This same point applies to 1-4 here.

Daniel Himmelstein

Is there data on how long project's usually take? Ron Vale found, for example, that it now takes UCSF graduate students on average 6 years to publish their first paper [1].

0
1.
Accelerating scientific publication in biology
Ronald D. Vale (2015) Proceedings of the National Academy of Sciences. doi:10.1073/pnas.1511912112
Daniel Himmelstein

Any examples where early feedback was beneficial or where a lack of feedback was harmful?

Lars Juhl Jensen

One could easily argue that such specific input could as well be given only after funding has been obtained.

Daniel Himmelstein

Any examples? Open source software could be a relevant example for 2 and 3.

Lars Juhl Jensen

Q&A sites like SEQanswers and Biostars could be other suitable examples.

Lars Juhl Jensen

This is a strong point, but more for the general concept of open notebook science than specifically for what this proposal is about.

Jesse Spaulding

Hm, I was thinking the ideal time to form collaborations would be when ideas are still forming. I'll rephrase this a bit to be clear what it's referring to.

Lars Juhl Jensen

That would be an excellent time indeed. It is just a problem of too generic phrasing.

Daniel Mietchen

Collaboration can form at any point in principle. Laying out the research process in detail renders more of those points discoverable by potential collaborators.

Lars Juhl Jensen

I understand that you want to give credit, but this is not the place to do so. You want to end on a punchline. Now you have a punchline, followed by "oh this is actually not really our idea".

Jesse Spaulding

I take your point. Any suggestion on where I could move this? Maybe to the beginning of this section? It shouldn't come off like we are just taking his ideas. These are all ideas derived from first principles analysis. But it seems only right to link to Daniel's work because he has been out there promoting open proposals the most.

Lars Juhl Jensen

In my opinion you could drop the sentence entirely and just cite reference 1 somewhere (I lack the overview to tell where).

Daniel Mietchen

No need to mention my name. A good place to reference such related efforts might be right after "Thinklab is intended for broad use across all of science, including biomedical research." above.

Lars Juhl Jensen

His name is spelled Daniel Mietchen

Jesse Spaulding

Crap. Embarrassing. I thought that was fixed. Sorry Daniel!

Daniel Mietchen

Versioning would be useful for such cases, and a mechanism for removing comments that do not apply to later versions any more.

Jesse Spaulding

There actually can be versions — and in the new version your name will be corrected :) All of these inline review comments are tied to this particular version.

Casey Greene

Could you slightly rephrase this to: Why should/will scientists participate? With the current phrase you need to make an iron clad case that people actually will do it. I don't think you have the evidence for this yet. With the re-framing, it just needs to be clear that it will be advantageous for scientists to participate.

Casey Greene

I'd re-write this bit also as above. What does the world look like when researchers are using thinklab? Highlight the points where it's better for the scientist.

Casey Greene

See above: why is the world better for funders - paint me a vivid picture.

Lars Juhl Jensen

Strange backwards phrasing that initially makes it sound as if you doubt your own goal. "We believe science funders are in the best position to ..." would be a much stronger opening.

Casey Greene

A critical reviewer could easily see this as adding busywork. Funders can require a lot. If you hold the purse strings you have the power to decide what happens. This can be good or bad. Why is this good? Imagine that I'm a skeptical reader who thinks: imagine the paperwork now - even before I get the grant!

Daniel Mietchen

Key point here could be that many of the bureaucratic layers have been built to address issues that do not manifest themselves the same way in open science, so if tackled properly, open science (a stick for many, a carrot for some) could actually be combined with a reduction in red tape, which should be a carrot for every researcher.

Lars Juhl Jensen

Too vague. Link directly to next sentence to get to what you concrete want to attain.

Daniel Himmelstein

Consider some references on how current closed grant peer review is not very effective [1]. You want to stress that funders could desperately use help in selecting which proposals to fund.

0
1.
NIH peer review percentile scores are poorly predictive of grant productivity
Ferric C Fang, Anthony Bowen, Arturo Casadevall (2016) eLife. doi:10.7554/eLife.13323
Lars Juhl Jensen

Just get to the point: "We recognize ..."

Lars Juhl Jensen

Too long and vague. Don't tell me what you could hypothetically do. Tell me what you will do. "In the meantime, we will do everything to drive adoption by ..."

Casey Greene

I think this belongs first, and that it should get worked into the proposed "Why will scientists participate?" section.

Lars Juhl Jensen

Too much text saying too little. Again, nice of you to give credit to who got which ideas, but it waters down your proposal. In this case, where Ben came with the fundamental idea for the proposal, you could consider making him a partner.

Jesse Spaulding

Why do you say this waters down the proposal? I would imagine listening to the community would be seen as a good thing!

Lars Juhl Jensen

It is just unnecessarily wordy. From the perspective of a reviewer, I don't care who got which idea. "There are a number of benefits to this" says nothing that the heading "Benefits for grant writers" doesn't say. Last sentence says nothing that couldn't be said by just referring to the URL after the second-to-last sentence :-)

Casey Greene

Casey Greene, a first time reviewer on this proposal, said: "The UI is very slick!"

Lars Juhl Jensen

This section seems largely redundant with the bullet list at the very beginning of the proposal.

Casey Greene

any evidence?

Jesse Spaulding

No. I'll change the language to be less matter of fact.

Casey Greene

How do you get a proposal in front of the right reviewers to make sure that it's well reviewed. There's a lot of feeling that the twitterverse is essentially a mutual admiration society. How do you keep this from becoming that?

Jesse Spaulding

Yes, well there has to be a lot of reward! I imagine you're reviewing this as a favor, because you've used Thinklab before, and because you'll be working with @dhimmel soon. But generally it is really hard to get people to take time away from their own work to give feedback. That's why I think funders should pay people for it. On top of that we add an impact points system so people feel they are being recognized appropriately.

Casey Greene

Do I invite people, do the funders invite people, or do people self select? That was the bit that I was a bit unclear on.

Jesse Spaulding

Right now it's up to the authors to invite people. However, with funders involved that could change. I imagine it will always be open for self selected commenters as well.

Casey Greene

Who is the community here? Other scientists, etc?

Casey Greene

Let's say I'm a grad student or postdoc with a valid critique of a grant, but everyone who posted thus far loves this thing. What's the benefit to adding my (potentially negative) comment? Conversely - what's the benefit to reacting positively to a grant that is widely viewed as negative by earlier reviewers? How do you keep feedback independent of the first people to comment.

Daniel Mietchen

There may sometimes be benefits in both of these scenarios, but they might require a specific bit of knowledge (or occasionally even ignorance of mainstream views). I'd expect it to be more common, though, that if reviewers 1-3 agree, others would pile on comments that essentially amount to "+1".

Jesse Spaulding

I think it is risk/reward too. If 3 reviewers agree and a young grad student offers a dissenting opinion with a convincing argument that persuades the first 3 reviewers, this young grad student would gain a lot of respect (I would imagine).

Casey Greene

Will there be an option for this?

Jesse Spaulding

Hm, I don't know. Let me know if you have a suggestions.

Casey Greene

I could imagine someone wanting their grant text itself to be -ND. Would you like someone to copy a substantial amount of your grant to their own when applying for funding, even with credit?

Daniel Mietchen

I don't think anything more restrictive should be allowed, but CC0 would be nice.

Casey Greene

Why not? What if we switch to this and thinklab fails. Would we lose the ability to apply for funding? Yikes!

Jesse Spaulding

I'm not sure what you mean here. How would you lose the ability to apply for funding? It's not open source because we are trying to create a sustainable business so that we can achieve our goal of making science much more open and collaborative.

Casey Greene

Let's say that funders adopt thinklab to manage grant review. At some point, thinklab ceases to function. Now the entire platform has to be rebuilt to maintain the same approach to review. Is there a way that you could make it open source but offer a hosted/turnkey solution? This would give people the peace of mind that they could deal with a shutdown if they had to, but I bet the barrier to entry of issuing dois/etc would let you still price reasonably aggressively.

Jesse Spaulding

Hm, I don't know. The main concern of open sourcing now would be that a group of academics would decide this shouldn't be run as a for-profit company. They would be able to quickly launch an identical service and because of it's existence few people would signup on Thinklab. I believe it would also make it very hard for us to get funding from traditional investors. I will just say that we much prefer to have "impact investors" involved but the reality is we need to keep our options open at this point.

Casey Greene

I see - makes sense! It probably wouldn't hurt to keep thinking about this in case there's a way to make it work. I could see some trepidation around the current setup. Perhaps there's a way to ameliorate that without removing your ability to find traditional investors. I'll try to keep thinking too.

Michael Crusoe

I love everything about thinklab, except for this. Not being F/OSS is a dealbreaker for me. No way am I going to contribute to a platform that locks me in. F/OSS is completely separate from your ability to make a decent profit.

Jesse Spaulding

@michael_r_crusoe the content on Thinklab is CC-BY. In what sense are you locked in? And in what sense would you not be locked in if the Thinklab code was F/OSS?

Casey Greene

I think you might want to play with the ordering of this. As I read, I encountered these questions (see above) before they were addressed. Easier to read if you supply answers as the questions are raised.

Casey Greene

Do the reviewers see a revised proposal? Is there a way for the author to mark the places where a grant changed in response to specific comments? I might be willing to come back to see my notes overlaid on changes, but I probably wouldn't if I had to manually put them side by side.

Jesse Spaulding

It's a good feature idea. When editing the proposal we can set things up so authors will save a "commit", and have that commit reference a particular comment. In the meantime authors can just paste in a link to the diff. I'll try to do that with this proposal.

Casey Greene

I don't have: "Add inline comment" Looks like it might always happen when you click "Note"

Daniel Mietchen

Yeah, same here - is the figure showing an earlier UI?

Daniel Himmelstein

"Needs citation" would be nice.

Casey Greene

If you're going to put this in here - how does it work? Does the open notebook platform integrate with the open proposal platform?

Jesse Spaulding

It's the same platform. The discussion on the proposal just becomes part of a larger discussion of the whole project as it moves forward. Is that more clear?

Casey Greene

It is to me because I've interacted with @dhimmel's project. I'm not sure it'd be clear for your reviewers at this point, because they may have no idea how the notebook solution works.

Jesse Spaulding

For science/society.. will clarify.

Casey Greene

This is either an expansion or reduce. What's the purpose of including this here? If you don't have time to talk about it, I might remove it for now. You could mention it in a future directions, but is this within the scope of this grant? Feels a bit like scope creep.

Daniel Himmelstein

Missing period after citation. Also the book can be cited by DOI: [1]

0
1.
Reinventing Discovery
Michael Nielsen (2011) Walter de Gruyter GmbH. doi:10.1515/9781400839452
Casey Greene

I might move this above just under Scientists and Funders and call it: Why will reviewers participate?

Daniel Himmelstein

Rather than calling it a "big problem", perhaps rephrase that incentives will catalyze participation. I agree here with Lars's cart before the horse comment. I believe open science is incentive compatible currently, just few realize it. I think it's important you stress that Thinklab gives people the tools to take advantage of incentives, many of which already exist and that Thinklab doesn't have to pay for.

0
1.
How to address the risk of stealing ideas
Lars Juhl Jensen, Jesse Spaulding, Daniel Himmelstein (2016) Thinklab. doi:10.15363/thinklab.d161
Lars Juhl Jensen

That depends somewhat on what I say as a reviewer. Not everything is nice to say in public.

Casey Greene

The answer to this from reviewers could definitely be: "no." What if I'm busy and reviewing this proposal from an airplane where I can't focus enough to make deep comments? What if I'm without internet access? For the first: I realize we shouldn't review grants when we don't have sufficient time to think deeply, but we should acknowledge that this does happen. Would thinklab lead to fewer (but more successful) proposals reaching review panels? This could be a big selling point.

Daniel Himmelstein

can replace with "became"

Casey Greene

What if I'm an early career scientist, and I'm assigned Big Shot 5's grant to review? This seems like a situation with a lot of potential downside for an early career scientist. Is there a way to protect reviewers?

Jesse Spaulding

I don't think anyone would ever be assigned as a reviewer. People should only be reviewing if they actually have some expertise and expect their review to be of some value. We could consider adding anonymous commenting at some point in the future but in the near term it's not something we are looking at.

Lars Juhl Jensen

I very much share @caseygreene concern on this. It can be outright deadly to the career of a young scientist to publicly criticize a big shot. At the same time, the vast majority of scientists are early career scientists. If one cannot find a mechanism to protect them, one will very quickly run out of reviewers.

Daniel Mietchen

The road to openness is not all about fear. Intelligent comments on some big shot's stuff might actually get a newbie attention in a space where they currently get basically none, and if that attention becomes unpleasant, they might possibly have a better handle on it than in the current closed system. Plus, an open system would do the sorting into big shots and others in a more meritocratic way than the current system. The challenge is to find good ways to navigate these spaces. Are there examples from open-source that might be instructive in this context?

Casey Greene

How does this not degenerate into a mutual admiration/criticism society? It feels like some twitter discussions certainly do.

Jesse Spaulding

Good question. The ratings are private. If it was public, I would agree: users would rate each other highly as thanks, or rate each other low as revenge.

Casey Greene

Might be helpful to note this very briefly.

Casey Greene

Who gets the money? Does it go to me personally or to my science?

Jesse Spaulding

Personally. In theory it could go towards supporting your lab, but not everyone is in a lab. This is intended to reward participation from people outside academia as well.

Casey Greene

Might be useful detail to add. I didn't realize the outside academia component, so this might be a good place to let readers see that vision.

Jesse Spaulding

Actually I never thought about having money go towards supporting the person's lab. It's something that could definitely be done that way if that's what funders want. (At least for those affiliated with a lab.)

Casey Greene

Well - in this case my inclination would be to say that the incentive could go towards an individual or their research program, which allows academics with labs and members of the public without labs to participate.

Lars Juhl Jensen

Giving the money personally is certainly the strongest incentive. This does bring a few ethical and practical challenges, though. Is it fair that I spent work hours on reviewing for ThinkLab instead of working for my employer? Or does this turn reviewing into a side job? Do all universities allow their researchers to have side jobs?

Jesse Spaulding

Yeah, maybe cross those bridges when we get there :)

Daniel Mietchen

I can see numerous scenarios in which either the lab or the individual might not be in a position to accept the money (perhaps even both), but in some cases, it might work well. It's important to be flexible here, e.g. to allow active reviewers to have a say in how funds are spent in their broader area, or to donate their monetary rewards to some good cause (think a fund for minority scientists or so).

Casey Greene

Though I'd say "comments that are valuable" - because it might be possible to write comments that peers find valuable that aren't actually.

Jesse Spaulding

Well, unfortunately we will not be able to detect comments that peers find valuable but that are not, in fact, valuable. So yes, you can game the system by writing comments that peers think are valuable but that actually are not. LOL

Daniel Himmelstein

This paragraph would benefit from a citation or two.

Lars Juhl Jensen

Without reading up on the overjustification effect, I do not understand this part.

Casey Greene

I'm not sure that this is true. I'd say that a very small minority are already doing this.

Casey Greene

What makes thinklab different than all of these others? Speaking of which - what is the related work to thinklab? Haven't seen it yet.

Lars Juhl Jensen

This comes across as awfully patronizing!

Casey Greene

I agree. Rubbed me the wrong way too.

Casey Greene

I'm still not entirely sure that the case has been made that the proposed system is better. Is it valuable because it's different, or is there some evidence that it's inherently better?

Casey Greene

I'm honestly not entirely sure that discussing the role of money in this is helpful. Sure, it may be a way to get people to participate, but it seems like refining the specific incentive structure is an experiment that needs to be performed. With some preliminary data, you could make an argument for why this incentive structure is good. Right now, it feels like this is one of many, w/o evidence, but it feels a bit concrete.

Lars Juhl Jensen

Considering that content will be copyright of the contributors, describing it as yours may rub some people the wrong way.

Casey Greene

As you can see from my comments, I was wondering about alternatives earlier. What if you worked these parts into the envisioned future bits that I discussed earlier. You could envision what the future would be like with open grant review. Then you could discuss how much of this can be done now (and how difficult it is) with existing tools if a scientist was motivated to do this.

Jesse Spaulding

I will rephrase this. Simply making the point that not all conversations have to be crammed into the sidebar.

Casey Greene

Gotcha! I actually didn't even realize that there was a "discussion" beyond the sidebar until I got to the later categorization part.

Jesse Spaulding

What kind of evidence are you looking for? This is more a statement of logic: if we direct people's attention to discussion that matches their interests then we will draw more people into the conversation. Google Docs does not do that. If you want attention on your Google Doc you have to share it with people. I guess the proposal needs to better explain that a "Thinklab Inbox" will surface content automatically based on a users's interests. Relates to this part of the proposal.

Casey Greene

This seems outside of the domain of proposal submission, which this focuses on. Can you integrate w/ github/continuous integration providers to provide a verifiable record behind figures?

Daniel Himmelstein

Consider changing to "the analog of a GitHub issue"

Daniel Himmelstein

Not sure what you mean exactly here. This whole paragraph repeats already established concepts.

Casey Greene

There's not a lot here to judge Jesse on. The specificity level is low.

Lars Juhl Jensen

Comes across as vague. How is Gleb related to ThinkLab?

Casey Greene

Really? Which ones.

Jesse Spaulding

I was referring to the incentive to keep everything secret until publishing a paper. Will clarify.

Casey Greene

What the the incentives though? Is it inertia or are there actual incentives?

Jesse Spaulding

Well they are concerned about people taking their ideas and publishing something first. This gets even more concerning if we're asking people to share proposals. I do think these concerns are overblown but nonetheless they are there. And I do believe there are many benefits to the researcher for being open that aren't fully appreciated. And yes, inertia, that's big.

Casey Greene

What about using the term "challenges" and highlighting "inertia" and "fear"? Incentives (particularly aligning with how the term is used in this grant) feels like someone is paying scientists to be closed.

Casey Greene

I'm not sure that this is the case now - science definitely feels at least within-team collaborative. Can you make this a concrete example that is a bit more connected to the way things work right now?

Lars Juhl Jensen

Having both "enormous" and "massive" so close to each other comes across as overselling.

Lars Juhl Jensen

I would leave this out.

Casey Greene

Think of evolution: sure, existing living systems are bizarre and often seem sub-optimal but if you just start making changes to the genetic code willy-nilly, it's probably not going to make the system more efficient. How are you going to measure your impact? How will you know if this is improving or harming the process of scientific research? How you will you adjust if it turns out that items need to be changed?

Jesse Spaulding

I see this proposal has completely failed to persuade you. I posted a response here: Has our scientific system been optimized by an evolutionary process?

0
1.
Has our scientific system been optimized by an evolutionary process?
Jesse Spaulding, Casey Greene, Daniel Himmelstein (2016) Thinklab. doi:10.15363/thinklab.d170
Casey Greene

I tried to read this as a skeptical grant reviewer. I actually like what you're doing, and I view your contributions positively. I wanted to give feedback that I thought would help you slightly reframe your proposal to make it more compelling to the interested but skeptical.

Jesse Spaulding

Thank you. I really appreciate you taking the time!

Daniel Himmelstein

Add citations to sources with a "formal academic background" as well to lend credibility to the claim.

Daniel Himmelstein

Consider citing RIO [1]
and the five deadly sins [2] either here or elsewhere.

0
1.
Publishing the research process
Daniel Mietchen, Ross Mounce, Lyubomir Penev (2015) Research Ideas and Outcomes. doi:10.3897/rio.1.e7547
0
2.