(first posted at https://twitter.com/krzhang/status/1245288938664644608; which is why it has Twitter vibes and links to tweets. Slightly edited to flow better with medium, then some items are appended) tl;dr: Sharp writing. Callous style doesn't cover the fact the paper is written with real heart. I give << 10% that this becomes mainstream in 5 years. As homage to paper, I offer up to $18000 at 9:1 odds that such a system is created with >100 papers. (if not clear, this is a hedge where I *really want to lose* $18000). Other thoughts/criticisms: 1. certain studies take a lot of work to replicate; so those studies withstanding time is less of a proof of their quality. Idea: scale cost/reward with difficulty of work. 2. Chesterton's fence: the existing system actually protects against this regime (hard-to-replicate) where setting up the replication, counterfactuals, etc. is hard. When people spaz over methods instead of results, you actually protect against a partially-correlated set of problems: bad methods almost never create good results. 3. If we think of academia as a roof-of-work blockchain (I am somehow unironically using https://twitter.com/krzhang/status/1234891204585644032…) , the analogy of wasteful "work" is gatekeeping, language, method-checking etc. that comes from peer review. This paper urges moving from proof-of-work to proof-of-stake. 4. Idea: to cover regimes where the FPP can't be efficiently tested (see (1)), it may be good to have a *hybrid* PoW/PoS design. Concretely, you may want to set up a game where people can both challenge the results *OR* attack the methods (as a crowdsourced subgame). 5. Idea (not in original tweets): hard to decide sources of truth, counterfactuals, etc. This idea may need to co-evolve with blockchain + on-chain oracles, to automatically "Settle" things. Unlike blockchain, since this is "real life" we may have to occasionally defer to a "jury" of experts to settle the validity of a challenge which reverts a bit to peer review, however I think this is unescapable as the language needed to seriously put skin in the game is costly. (this is why IRL law is so convoluted)
I make the distinction between academic probabilities, which are not rooted in reality and thus have no tangible real-world meaning, and real probabilities, which attain a real-world meaning as the odds that the subject asserting the probabilities is forced to accept for a bet against the stated outcome. With this I discuss how the replication crisis can be resolved easily by requiring that probabilities published in the scientific literature are real, instead of academic. At present, all probabilities and derivatives that appear in published work, such as P-values, Bayes factors, confidence intervals, etc., are the result of academic probabilities, which are not useful for making meaningful assertions about the real world.
➤ Version 1 (2018-08-30)
Harry Crane (2018). The Fundamental Principle of Probability. Researchers.One, https://researchers.one/articles/the-fundamental-principle-of-probability/5f52699b36a3e45f17ae7d4a/v1.