Thinklab: A platform for open review of research grant proposals

-

# What is Thinklab?

+

# Executive summary

-

Thinklab is a platform that facilitates two things:

+

Thinklab is a platform for open review of research grant proposals. We're creating Thinklab because we believe the interests of science and society are best served if the *entire* scientific process is open -- not just the software, data, and papers produced at the end. And while we're convinced this kind of openness is in the best interests of science as a whole, we understand many scientists will not feel it's in their *personal* interests. That's why our goal is not just to enable open research, it's to actually *create incentives* for it. We aim to create value for both scientists and science funders. Thinklab is intended for broad use across all of science, including biomedical research. An early version can be seen at [thinklab.com](http://thinklab.com).

-

1. Open review of research grant proposals
2. A highly collaborative version of open notebook science

We're creating Thinklab because we believe the interests of science and society are best served if the *entire* scientific process is open -- not just the software, data, and papers produced at the end. And while we're convinced this kind of openness is in the best interests of science as a whole, we understand many scientists will not feel it's in their *personal* interests. That's why our goal is not just to *facilitate* open research, it's to actually *create incentives* for it.

Thinklab is intended for broad use across all of science, including biomedical research. An early version can be seen at [thinklab.com](http://thinklab.com).

# Why grant proposals should be reviewed in the open

1. **Researchers can get feedback when it's most valuable: before a project begins**
Scientists often work on projects for years at a time. Given this, it's critically important to work on the *right thing* and have the *right approach*. Feedback at an early stage -- especially when it comes from a cognitively diverse pool of people -- has the potential to save incredible amounts of time. It may reveal that an entire project was flawed from conception, or it may light the path towards a much more impactful approach.

+

# How Thinklab works

-

- **Best practices can spread across science faster**
Best practices in science are changing fast. Having proposals reviewed in the open, gives the community a chance to share those best practices -- even across disciplines. This could include sharing ideas on making data reusable, making research reproducible, or just recommending tools and methods.

+

Thinklab targets two groups: scientists and science funders. We'll start this proposal by looking at Thinklab from the perspective of scientists. Later on we'll explore how Thinklab can help science funders open up research and accelerate their impact.

-

- **Scientists can learn from and build upon each other's ideas**
Research proposals are valuable scientific outputs in and of themselves. They will often contain some of the most cutting edge ideas in science. Having proposals (and the review discussion related to them) published openly on the internet would create a valuable resource that the entire scientific community can learn from and build upon.

+

## Posting an open proposal

-

- **Openness produces more collaboration and less redundancy**
When people know what each other are working on, it creates more possibility for forming collaborations. At the same time, it reduces the redundancy of having multiple research groups working on the same thing in isolation.

+

The first challenge Thinklab faces is persuading scientists to publicly share their research proposals. This is tricky because there's a fear that ideas will be stolen. We believe this fear is valid but overblown. We also believe there are a number of benefits to grant writers that are not fully appreciated:

-

In conclusion, open review can improve research plans, reduce wasted resources, spread best practices, accelerate the exchange of ideas, and lead to more collaboration with less redundancy. In short, it should significantly accelerate scientific progress. Many of these ideas have been talked about by Daniel Mitchken: [The Transformative Nature of Transparency in Research Funding](http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002027) [@10.1371/journal.pbio.1002027]

+

**A validated research plan** -- Before embarking on a research project it's critically important to make sure you're working on the right thing and have the right approach. Early feedback can validate the research plan and give researchers confidence that when results come in, they'll be trusted and valued by the community.

-

# But will scientists participate?

+

**Improved funding odds** -- It's easy to forget that not everyone has the same set of background knowledge that we have. Getting an outside perspective can make sure the value of a proposed project is being communicated effectively -- presumably improving the proposals funding odds.

**More visibility and citations** -- Research proposals are valuable scientific outputs in and of themselves. Thinklab gives them a DOI and makes them a citable part of the scientific record.

-

While it may be clear that having grant proposals reviewed in the open would be a very good thing for science, the question is: will scientists participate? Won't they be concerned about people stealing their ideas?

+

**More collaboration opportunities** -- Instead of encouraging others to compete with you, an open proposal could do just the opposite. If people see you're already working on something it's likely they'll consider collaborating with you, or simply pursue another avenue of research.

-

The answer is yes, many will be concerned. Scientists currently operate under a system that values the publication of papers in journals above all else. Under this system, keeping ideas secret is seen as a good strategy in the publish-or-perish game -- a game that they're forced to play, but never asked to play. In our view, the solution to this predicament is clear: we need to change the rules of the game! We need to align the incentives of scientists with what's in the best interests of science and society as a whole.

-

## Thinklab as a service for science funders

+

## The open review process

-

If our goal is to open up grant proposal review, we believe science funders are in the best position to do it. Funders can use the power of the purse to create new incentives and compel adoption of new behaviors. In fact, they're already doing it. Many funders have started to *require* that the results of the research they fund be made publicly available. That changes the game!

+

Once a proposal has been published it's up to the research team to invite peers to review it. Alternatively, scientists (or any approved user) can review a proposal they find interesting. Thinklab will strongly encourage a two-part review process where ideas are first submitted as "one-pagers", and only later developed into full proposals. Here's what the review process looks like:

-

Thinklab wants to help funders take the next step. We want to help funders create grant programs that *require* openly posted proposals. In addition, we want to help them manage an open peer review process where anyone from the scientific community can share ideas and feedback to help improve research plans.

+

**Review Part 1: Annotate** -- In this step reviewers are asked to read the proposal and make annotations wherever they have feedback (see [Figure {n}](#annotation)). To ensure reviewers form opinions independently we hide annotations and comments from others at this stage.

-

With that said, we recognize that persuading funders to try a different model of funding will be challenging. Fortunately, there's something we can do in the meantime: we can do everything we can to drive adoption by making Thinklab a valuable service for grant writers directly.

+

**Review Part 2: Summarize** -- Next, reviewers are asked to summarize their most important thoughts and advice. They may also be asked to rate the proposal on various metrics.

+

**Review Part 3: Discuss** -- In this step we reveal comments from other users. Reviewers are asked to use the notes they made in part one to join a real-time discussion with the authors and other reviewers. Discussion can take place in inline comments or in longer form discussions that exist on their own page.

-

## Thinklab as a service for grant writers

+

**Review Part 4: Finalize** -- After discussing the proposal, reviewers have the option to update their review scores and/or provide updates on how their overall assessment of the proposal has changed.

-

Benjamin Good [suggested](https://think-lab.github.io/discussion/thinklab-as-a-vetting-system-for-traditional-grants/58) [@10.15363/thinklab.d58] that grant writers could use Thinklab to have their proposals openly reviewed prior to submission for funding. There are a number of benefits to this. In fact, we've used Thinklab to get feedback on the proposal you're reading right now! X reviewers provided feedback through a total of X annotations and X comments. You can take a look at the feedback we received here: [think-lab.github.io/p/thinklab-prize](https://think-lab.github.io/p/thinklab-prize).

+

This review process has been used [to gather feedback](https://think-lab.github.io/doc/12/review) on this proposal. Four reviewers made a total of 150 comments and 139 annotations. Casey Greene, a first time reviewer, [said](https://think-lab.github.io/doc/12/review#59): "The UI is very slick!"

-

### Benefits for grant writers

-

- Open review can help grant writers improve their proposals and increase their odds of funding. Getting an outside perspective can help make sure the value of a proposed project is being communicated effectively.
- Open review can help validate the research plan before work begins. With this validation, the research team can be confident that when results come in, they'll be trusted and valued by the community.
- Engaging the community at such an early stage serves to increase the research team's visibility and connection with the community, leading to more collaboration opportunities.
- Grant writers can feel better about asking for feedback as there's more in it for the reviewer. Reviewer comments are public, and reviewers are publicly acknowledged based on peer assessment of the value of their feedback.

+

[:figure](annotation)

-

# How Thinklab works

+

## Thinklab Impact Points

-

Thinklab is a platform that facilitates open review of grant proposals, and real-time open science. All content and discussion posted to Thinklab is licensed CC-BY. Note that at this point the code that runs Thinklab itself is not open source.

+

To reward reviewers for sharing valuable feedback and ideas, Thinklab has created an impact points system. This system allows us to highlight the "most impactful reviewers" for each proposal, while also creating a [global impact leaderboard](https://think-lab.github.io/leaderboard).

-

## The proposal review system

+

Reviewers earn impact points based on peer assessment of the value of their contributions. Contributions are rated by the research team, by users selected by a sponsoring funder, or by the community at large. To avoid any temptation for "reciprocal voting" ratings are submitted privately.

-

**Design goals:**

+

To create additional incentive, Thinklab aims to partner with science funders and offer monetary rewards that are tied to impact points. (More on this later.)

-

1. To have reviewers share independently formed opinions initially.
2. To enable a real-time discussion between reviewers and proposal authors aimed at improving the proposal.
3. To reward reviewers by recognizing them based on the value of their contributions.

+

## Thinklab Inbox

-

**The review process:**

+

Users are notified of new content and discussion that is relevant to them via the Thinklab Inbox. Right now it simply shows any discussion that directly matches anything the user is following. Users can follow proposals, topics, or even funding opportunities. Over time we intend to make the system smarter such that we more efficiently direct scientific attention to the exact problems and discussions that need it.

-

1. **Part 1: Annotate** -- In this step reviewers are asked to read the proposal and make annotations wherever they have feedback (see [Figure {n}](#annotation)). To ensure an unbiased review we hide annotations and comments from other users in part 1 and 2.

+

## Real-time open science

-

2. **Part 2: Summarize** -- Next, reviewers are asked to summarize their most important thoughts and advice. They may also be asked to rate the proposal on various metrics.

+

When a proposal gets funded, the research team has the option to continue their work as an open research project. Researchers can engage the community by sharing ideas, project plans, updates, and questions in real-time as the project progresses. Reviewers that made valuable suggestions during the proposal stage can continue to share ideas and give feedback throughout the project. This all occurs using the exact same discussion system that is used for proposal review.

-

3. **Part 3: Discuss** -- In this step we reveal the discussion that is already taking place. Reviewers use the inline notes they made in part one to join the discussion. Discussion can take place in inline comments, or on separate pages that link back to the proposal.

+

Daniel Himmelstein is currently using Thinklab to lead an open research project: [Repurposing drugs on a hetnet](https://think-lab.github.io/p/rephetio) [@10.15363/thinklab.4]. His project has accumulated over 300 comments from 28 contributors. In his [review](https://think-lab.github.io/discussion/review-summary/164) of this proposal Daniel says:

-

4. **Part 4: Finalize** -- After discussing the proposal with the authors and other users, reviewers have the option to update their initial review summary and/or review scores.

+

> The best thing about Thinklab is that it not that it solves a major problem in science but that it solves most major problems in science. Having used Thinklab for over a year now, I have experienced the plethora of benefits.

-

When the review process is complete, the proposal authors (or designated reviewers if the review is on behalf of a funder) will rate the value of each reviewer's contributions. These ratings contribute to an impact points system that allows us to highlight the most impactful reviewers.

-

[:figure](annotation)

+

## Alternatives to Thinklab

-

## Real-time open science

+

If a researcher wants to get feedback on a grant proposal, a common solution is to put it in Google Docs, and send it to colleagues. Here are the important ways Thinklab is different:

-

When a proposal on Thinklab gets funded, the research team has the option to continue their work as an open research project. Researchers can engage the community by sharing ideas, project plans, updates, and questions in real-time as the project progresses. Reviewers that made valuable suggestions during the proposal stage can continue to share ideas and give feedback throughout the project.

+

- Thinklab is designed to have peer review and reviewer comments become a persistent, citable part of the scientific record. Each Thinklab discussion has a DOI and is citable.
- The Thinklab Inbox serves to direct researcher attention throughout the network by *automatically* showing users discussion that matches their interests and expertise.
- Conversation is not constrained by the need to fit in the sidebar of a Google Doc. Users can post more substantial discussion to it's own page where it's easier to read, and can attract more attention.
- Thinklab recognizes reviewer contribution through an impact points system that is based on peer assessment.
- Thinklab intends to build features that will integrate with workflows at funding organizations and enable them to manage an open peer review process.

-

The big payoff from open research comes not just from being open, it comes from the real-time collaboration that openness enables. It comes from bringing a cognitively diverse set of people together to tackle challenging problems. In support of this Thinklab is building a system that intelligently direct researcher attention to the exact discussions and problems that match their interests and areas of expertise. These ideas are described by Michael Nielsen in his book: [Reinventing Discovery: The New Era of Networked Science](http://www.amazon.com/gp/product/B005OQGZ54/ref=dp-kindle-redirect?ie=UTF8&btkr=1) [@reinventing]

+

## Openness & sustainability

-

There's one big problem with this vision: there needs to be incentives for scientists to participate. There needs to be an incentive for researchers to share their work, and their needs to be an incentive for outside scientists to share feedback and insights when they have something valuable to add.

+

Thinklab sees itself as a service that helps open up science. That's our mission and our most important work. For example, all content and discussion posted to Thinklab is licensed CC-BY.

-

## Benefits for reviewers/contributors

+

While we'd like to be in a position to open source the code for Thinklab itself, at this point we feel that might jeopardize our ability to create a sustainable business. First, it creates the possibility that a group of academics would launch a competing service and thereby hamper our ability to gain traction. Second, it would likely make it very hard for us to get funding from traditional investors. While we prefer to be funded by impact investors, the reality is we need to keep our options open at this point.

-

- **Everything is public**
To highlight the benefits of openness let's consider what will happen to this proposal when it's submitted to The Open Science Prize. We expect it to be discussed *in private* by three reviewers. Here's a question for those reviewers: wouldn't it be nice if your comments were public and would become part of the scientific record? Wouldn't it be nice if we (and others) could learn from what you said? Thinklab offers reviewers the chance for more impact and more recognition.

+

# Thinklab as a service for science funders

-

- **Users are recognized based on the value of their contributions.**
Thinklab has an impact points system that allows us to highlight the most impactful reviewers or project contributors. Users gain impact points primarily by conducting review and making comments that the community rates as valuable. Contributions are rated by the research team, by users selected by the sponsoring funder, or by the community at large.

+

Thinklab's primary goal is to accelerate a transition to a much more open, collaborative, Internet-native model of research. We believe creating a service for grant writers -- as outlined above -- will produce *some* success towards that goal. However, given the inertia of our scientific system, and given the incentives that drive scientists towards secrecy, it is likely that progress will be slow.

-

## Monetary rewards

+

We see this as a problem of incentives. To drive rapid adoption of open grant proposal review we need the incentives of scientists to be aligned with what is good for science as a whole. We believe science funders hold the power to align these incentives. In fact, they're already starting to do this in other areas. Many funders have started to *require* that the results of the research they fund be made publicly available.

-

While we believe our impact points system will help create an incentive, we'd like to point out that it's *extremely* difficult to get scientists to take time away from *their* research, and share feedback and ideas on the work of their peers, over the internet [@reinventing]. At the same time, we can all recognize how incredibly valuable it would be if they were to do so.

+

Thinklab wants to help funders take the next step -- we want to help funders create grant programs that *require* openly posted proposals. In addition, we want to help them manage an open peer review process where anyone from the scientific community can share ideas and feedback to help improve research plans.

-

For these reasons, Thinklab proposes that science funders create an additional incentive. We propose that a portion of project grant money is set aside to reward feedback and ideas from scientists in the community. These monetary rewards could be applied at the proposal stage and/or the research stage. The system would piggy-back on the impact points system we've already discussed.

+

## Monetary rewards for crowdsourced review

-

### Possible concerns

+

The big payoff from open research comes not just from being open, it comes from the real-time collaboration that openness enables. It comes from bringing a cognitively diverse set of people together to tackle challenging problems. [@10.1515/9781400839452]

-

- **Won't people game the system?**
It's very likely that some people will attempt to game the system. However, we believe it will be relatively easy to detect and prevent strategies such as voting rings between friends. The only way to game the system should be to actually take the time to write comments that peers find valuable.

+

To fully realize the potential of Thinklab, there needs to be strong incentives -- not just for grant writers -- but for reviewers too! If we want scientists to take time away from *their* research to share feedback and ideas on that of their peers, we need strong incentives.

-

- **The [overjustification effect](https://en.wikipedia.org/wiki/Overjustification_effect)**
This is where external incentives decrease a person's intrinsic motivation to do something. It's a fascinating concept. However, this effect is only relevant if people have enough intrinsic motivation to do something in the first place. In the case of getting scientists to share feedback and ideas on the work of their peers over the Internet -- they're simply not doing it at all. And this is despite many platforms built for this purpose [@reinventing].

+

Thinklab proposes that science funders set aside a portion of grant money to financially reward an open crowdsourced review process. These rewards would go directly to reviewers and be paid out in proportion to how much value they add. Distribution is tied to the impact points system we described previously. The only way to game the system should be to actually write comments that peers find valuable! At some point there will likely be attempts to game the system via "voting rings", but we're confident this issue can be managed.

-

- **Money in science is always bad**
Many scientists seem to have a gut reaction that tells them introducing money into science is always a bad idea. To those people we would say this: money is *already* in science. Much of what scientists do is driven by the need to secure money to continue doing their work. We are simply proposing that some money is distributed in a way that creates a different set of incentives.

+

There has been some concern about the [overjustification effect](https://en.wikipedia.org/wiki/Overjustification_effect). This is where external incentives (money) will decrease a person's intrinsic motivation to do something. However, it should be noted that this doesn't really apply if very few people are doing the desired activity in the first place. And this is by and large the case when it comes to getting scientists to comment on their peers work on the Internet.

-

Finally, we want to emphasize that we consider monetary rewards to be an experiment, and that we see large value in the Thinklab even without them.

+

## Benefits for funders

-

# What will the prize money be used for?

+

The following describes the benefits a Thinklab partnership can offer science funders. The theme here is accelerating scientific progress by making research more open and more collaborative.

-

- Continued development of our open review system. We will be working in close consultation with both grant writers and reviewers to create the most useful and user friendly service possible.
- Development of features to enable a pilot program with a funder. This pilot program will likely involve having proposals openly submitted and reviewed though Thinklab.
- Creation of APIs so our CC-BY content can be more easily accessed.

+

**Reviewers are less likely to reject good proposals for faulty reasons.** An open real-time discussion gives authors a chance to address reviewer criticism and possibly alleviate concerns.

+

**Fundamentally flawed research is more likely to be filtered out.** If proposals are first submitted as one-page ideas, it's reasonable that 10-20 people could read each one. Anyone that notices a potential flaw can start a discussion and bring in experts to address that particular issue.

-

# Alternatives to Thinklab

+

**Ideas that have potential can be developed into great ideas.** The first ideas people have are not always the best. Thinklab gives reviewers the opportunity to work with authors to develop an average idea into a great idea.

-

### Google Docs for open review of proposals

+

**Best practices can spread across science faster.** Best practices are changing fast. Thinklab gives people an opportunity to share those best practices, even across disciplines.

-

If a researcher wants to get feedback on a grant proposal, a common solution is to put it in Google Docs, and send it to colleagues. Here's why Thinklab is better:

+

**The exchange of ideas is accelerated.** Publishing proposals (and the review discussion related to them) creates a valuable resource of cutting edge ideas that scientists can learn from and build upon. Without Thinklab these ideas might not be shared until *years* later. Fundamentally, science progresses in proportion to the speed at which we share scientific information.

-

- Thinklab is designed to have peer review and review comments become a persistent, citable part of the scientific record.
- Thinklab facilitates more substantial discussion by allowing discussions to be posted to a separate page.
- Thinklab is able to draw more people into the conversation by directing people's attention to the discussion when it matches their interests or areas of expertise.
- Thinklab rewards participation through an impact points system that is based on peer assessment.

+

**More collaboration and less redundancy.** If scientists know what each other are working on, it creates more possibilities for working together. At the same time it will likely reduce the redundancy of having multiple research groups working on the same thing in isolation.

-

### GitHub as an electronic lab notebook

+

# Product roadmap

-

The Open Source Malaria project [is using](https://github.com/OpenSourceMalaria/OSM_To_Do_List) GitHub to openly manage their project. There's a lot to like about GitHub. But here's what Thinklab can offer:

+

- Continued development of our open review system. We will be working in close consultation with both grant writers and reviewers to create the most useful and user friendly service possible.
- Development of features to enable a pilot program with a funder.
- Creation of APIs so Thinklab content can be more easily accessed.
- Integration with more external services such as CLOCKSS, Altmetrics, and PubMed.

-

- Thinklab is designed for science. Adding math is easy. Adding citations by DOI is easy. Each discussion (like a GitHub issue) has a DOI and is citable.
- When leading an open research project the big challenge is actually getting scientists to participate. Thinklab rewards participation through a system that highlights the most impactful contributors. If needed we can add the additional incentive of monetary rewards.
- In the domain of scientific discussion there is tremendous value in having a cognitively diverse set of people participate. Thinklab has a system that intelligently directs researcher attention to discussion that is relevant to their interests or areas of expertise. Such a system is not particularly relevant to software development so we wouldn't expect GitHub to be working on it.

# Team and resources

-

Thinklab is founded by Jesse Spaulding. Jesse has a background in the startup world, and has founded and sold several startups. With no formal academic background, Jesse brings a fresh perspective to the challenges facing our scientific system. Jesse has been working with Gleb Pitsevich who has a broad background in mathematics, programming, and web development.

+

Thinklab is founded by [Jesse Spaulding](http://linkedin.com/in/jessespaulding). Jesse has created and sold several startups including [CourseTalk](https://www.coursetalk.com/), a student review site for MOOCs (Massive Open Online Courses). Previously, Jesse independently [developed a high frequency trading algorithm](http://jspauld.com/post/35126549635/how-i-made-500k-with-machine-learning-and-hft) that made 500k in profits. As someone who has spent his career in the startup world, Jesse is able to bring a fresh set of eyes to the challenges facing our scientific system.

-

Thinklab is setup as a for-profit and we intend to use this status to attract talented people to help us pursue our mission. In the near term we're looking for a talented individual with a strong academic science or philanthropic background to join the team as a co-founder and lead business and community development.

+

Jesse is working with [Gleb Pitsevich](https://www.linkedin.com/in/pitsevich) who has a broad background in mathematics, programming, and web development. Gleb has a master's degree in computer science and is the founder of [Razor Theory](http://razortheory.com/), a web and mobile development company based out of Belarus.

-

# Conclusion

Thinklab is a bold experiment in open science. We understand there's powerful incentives working against us. But we also understand the future of science is not a world where scientists continue to hoard knowledge and work in silos.

We believe Thinklab has a legitimate shot at opening up grant proposal review, and if we're able to do so, we believe the benefits will be enormous. Perhaps our most compelling argument is this: what we're doing has the potential to affect *all of science*. Anytime you can make changes that positively affect a *system* there's potential for massive impact. Given a [wide variety of problems](https://think-lab.github.io/blog/10-consequences-of-a-broken-scientific-reward-system/36) [@10.15363/thinklab.d36] in our current scientific system, isn't it worth experimenting with ways we might improve it?

+

Thinklab is looking for a talented individual with a strong academic science or philanthropic background to join the team as a co-founder and lead business and community development.

+

# Conclusion

-

[@reinventing]: http://press.princeton.edu/quotes/q9517.html "Reinventing discovery: the new era of networked science, Michael Nielsen"

+

We believe open review of grant proposals will improve research plans, reduce wasted resources, spread best practices, accelerate the exchange of ideas, and lead to more collaboration with less redundancy. In short: it will significantly accelerate scientific progress. Perhaps the most compelling argument for Thinklab is this: what we're doing has the potential to catalyze changes that will affect *all of science*. Anytime you can make changes that positively affect a *system* there's potential for massive impact. Science needs people thinking outside the box!

Thinklab: A platform for open review of research grant proposals

Proposal revision comparison

Content