I appreciate the attempt to flesh out the details of how a prediction market for scientific research might be organized. I have a few brief comments, especially as the proposed framework relates to my prior work posted here https://www.researchers.one/article/2018-08-16 The author references Robin Hanson's "idea futures" framework as well as my more recent suggestion based on what I call the Fundamental Principle of Probability (FPP). As far as I can tell, there is a key difference between mine and Hanson's suggestion, in that I do not propose a prediction market for addressing this problem. I instead believe it is essential that there is a direct link between the bettors and the scientists who make the initial scientific assertions. In my framework, when author A publishes a claim, s/he would assess the probability of replication p associated to that claim, and would layout an explicit protocol for how the replication study would be carried out and what would count as a replication under stated protocol. With this, the authors post collateral and agree to offer (1-p)/p odds for a bet against replication. In this way, the results published in the paper, namely an explicit probability statement p regarding a specific replication protocol and criterion, are being directly tested during replication. And the authors of the results are on the hook for the claims they make in the paper. The prediction market framework seems very different and, to me, misses at least 2 key parts: That the authors of the work must have 'skin in the game', i.e., the authors must act as the 'house' in the betting that ensues. The replication is adjudicated based on the published odds in the original study. This is the key distinction between academic and real probabilities in the paper I link above. By contrast, prediction markets give a sense of what the general public thinks of the replicability of a given study. But this seems quite a bit different than testing a specific probability asserted by the authors of the paper. My belief is that my proposal based on the FPP is a more direct way to resolve current replication issues. If possible, I would, however, be interested to see a discussion of how the prediction market framework is and is not related to my FPP proposal.
Prediction markets are currently used for three fields: 1. For economic, political and sporting event outcomes. (IEW, PredictIt, PredictWise) 2. For risk evaluation, product development and marketing. (Cultivate Labs/Consensus Point) 3. Research replication. (Replication Prediction Project, Experimental Economics Prediction Project, and Brian Nosek’s latest replicability study) The latter application of prediction markets has remained closed and/or proprietary despite the promising results in the methods. In this paper, I construct an open research prediction market framework to incentivize replicate study research and align the motivations of research stakeholders.
➤ Version 1 (2018-09-09)
Maxx Goad (2018). A Research Prediction Market Framework . Researchers.One, https://researchers.one/articles/a-research-prediction-market-framework/5f52699b36a3e45f17ae7d70/v1.