10 Consequences of a broken scientific reward system

A scientist's job performance is measured almost entirely based on the number of papers they've published and the impact factor of the journals in which they published. It's a reward system that's been in place since the 17th century. Unfortunately, in today's internet connected world, the continued use of this system has a range of negative consequences for science.

1. Most research is published with substantial delays and is locked behind paywalls
   Science progresses when scientists are able to build upon the latest work in their field. To create the most value, research results should be shared openly and immediately. Additionally, taxpayers who fund research for the public good should be able to access the results of that research. Unfortunately, scientists are rewarded for publishing papers with journals that charge their institutions huge subscription fees.

2. Data, software, and detailed methods are not shared
   Having access to the full research record is critical for reviewing, reproducing, and building upon scientific research. The problem is, scientists have many reasons not to share it: Withholding information can give a competitive advantage in the race to publish the next paper. It can take significant time and effort to organize and share everything in a way that others can understand. And finally, the more details you give away, the more likely it is that others will embarrass you by finding flaws in your work.

3. Negative results are not shared
   Scientific journals seek to publish surprising or interesting results. There's no reward for doing research and getting a null result. This has terrible consequences: It creates a bias towards safe projects where interesting outcomes are guaranteed. It creates a significant bias in the published literature. And it makes careers in science unnecessarily risky. It also leads to the following two points.

4. The research process is distorted by the need to generate interesting results
   Then need to generate interesting results infiltrates the research process in various subtle and not so subtle ways. The fact is research presents many opportunities to bias the results if indeed that's what you want to do (or are strongly incentivized to do.) To truly solve this problem we need to reward scientist for doing good science rather than getting interesting results.

5. Time and money are wasted on needless repetition of experiments
   When negative results aren't shared, or research is not shared in its entirety, scientist end up repeating work and reinventing processes again and again. It's a terrible waste of resources.

6. Scientists work in silos and rarely help each other outside of formal collaborations
   Because of the hyper competitive nature of publishing papers, scientist rarely share their work or help each other. The exception, of course, is when there's an arrangement to be co-authors on a paper. But there's a huge opportunity cost here: science as a whole would be much more efficient if scientists everywhere openly shared their work and collaborated with each other over the Internet. It's almost always the case that somebody somewhere has knowledge that could save you a lot of time.

7. Critical research design flaws are often not caught
   It's not uncommon that a scientist will look at a paper and completely discount the results because of the flaw in the design of the research. This should never happen. It means the research was a complete waste of resources. Scientists need to get extensive critical constructive feedback on their proposed research before work begins.

8. Statistical mistakes are widespread
   Statistics are hard. Data analysis is hard. But yet, both are critically important for designing research as well as extracting meaning from the results of research. Most of science comes down to what's statistically significant and what's not. It's not acceptable to get it wrong. While it may not be reasonable to expect everyone to be an expert on statistics, we should at least make sure every project has involvement from people who are.

9. Peer review is slow, inefficient, and ineffective
   Peer review is not the infallible system that we think it is. Studies show there's many mistakes it it simply doesn't catch. If we consider that peer reviewers are unpaid and unaccountable for their reviews, this is perhaps not surprising. As a system of providing feedback to help scientists improve their work, it's extremely poor. Peer review should really occur in real time throughout a project. There should be incentives for making valuable comments and feedback should be in the open so that other scientists can weigh in.

10. Much of research is not reproducible
    The sad truth is much of research is not reproducible. The primary reasons are likely: 1) Research methods, data, and software are not published in full. 2) The incentive to get an interesting result distorts the research process. 3) Math and statistical errors are commonly made. If research is not reproducible how can we have any confidence in what it says?

The conclusion? Our scientific reward system is no longer producing behavior that's aligned with the best interests of science and society. ThinkLab wants to solve these problems by working with science funders to create new rewards and compel adoption of a better model of research.