Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
129 tokens/sec
GPT-4o
28 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Optimal Pre-Analysis Plans: Statistical Decisions Subject to Implementability (2208.09638v3)

Published 20 Aug 2022 in econ.EM, econ.TH, math.ST, stat.OT, and stat.TH

Abstract: What is the purpose of pre-analysis plans, and how should they be designed? We model the interaction between an agent who analyzes data and a principal who makes a decision based on agent reports. The agent could be the manufacturer of a new drug, and the principal a regulator deciding whether the drug is approved. Or the agent could be a researcher submitting a research paper, and the principal an editor deciding whether it is published. The agent decides which statistics to report to the principal. The principal cannot verify whether the analyst reported selectively. Absent a pre-analysis message, if there are conflicts of interest, then many desirable decision rules cannot be implemented. Allowing the agent to send a message before seeing the data increases the set of decision rules that can be implemented, and allows the principal to leverage agent expertise. The optimal mechanisms that we characterize require pre-analysis plans. Applying these results to hypothesis testing, we show that optimal rejection rules pre-register a valid test, and make worst-case assumptions about unreported statistics. Optimal tests can be found as a solution to a linear-programming problem.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (43)
  1. Research registries and the credibility crisis: An empirical and theoretical investigation.
  2. Andrews, Isaiah and Maximilian Kasy (2019). Identification of and correction for publication bias. American Economic Review, 109(8):2766–94.
  3. Inference on Winners. The Quarterly Journal of Economics.
  4. Andrews, Isaiah and Jesse M Shapiro (2021). A model of scientific communication. Econometrica, 89(5):2117–2142.
  5. A theory of experimenters: Robustness, randomization, and balance. American Economic Review, 110(4):1206–1230.
  6. Selective Trials: A Principal-Agent Approach to Randomized Controlled Experiments. The American Economic Review, 102(4):1279–1309.
  7. Christensen, Garret and Edward Miguel (2018). Transparency, reproducibility, and the credibility of economics research. Journal of Economic Literature, 56(3):920–80.
  8. Coffman, Lucas C. and Muriel Niederle (2015). Pre-analysis plans have limited upside, especially where replications are feasible. Journal of Economic Perspectives, 29(3):81–98.
  9. Curello, Gregorio and Ludvig Sinander (2022). The comparative statics of persuasion. arXiv preprint arXiv:2204.07474.
  10. Persuasion bias in science: Can economics help? Economic Journal, 127(605):266–304.
  11. Strategic sample selection. Econometrica, 89(2):911–953.
  12. In praise of moderation: Suggestions for the scope and use of pre-analysis plans for RCTs in economics. NBER Working Paper, (w26993).
  13. Guidance for industry: Statistical principles for clinical trials. US Department of Health and Human Services.
  14. Frankel, Alexander (2014). Aligned delegation. American Economic Review, 104(1):66–83.
  15. Frankel, Alexander and Maximilian Kasy (2022). Which findings should be published? American Economic Journal: Microeconomics, 14(1):1–38.
  16. Gao, Ying (2022). Inference from selectively disclosed data. arXiv preprint arXiv:2204.07191.
  17. Glaeser, Edward L (2006). Researcher Incentives and Empirical Methods. Technical Report t0329, National Bureau of Economic Research.
  18. Glazer, Jacob and Ariel Rubinstein (2004). On optimal rules of persuasion. Econometrica, 72(6):1715–1736.
  19. Gneiting, Tilmann and Adrian E Raftery (2007). Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102(477):359–378.
  20. Henry, Emeric and Marco Ottaviani (2019). Research and the Approval Process: The Organization of Persuasion. American Economic Review, 109(3):911–955.
  21. Kamenica, Emir (2019). Bayesian persuasion and information design. Annual Review of Economics, 11:249–272.
  22. Kamenica, Emir and Matthew Gentzkow (2011). Bayesian persuasion. American Economic Review, 101(6):2590–2615.
  23. Kephart, Andrew and Vincent Conitzer (2016). The revelation principle for mechanism design with reporting costs. In Proceedings of the 2016 ACM Conference on Economics and Computation, pages 85–102.
  24. Leamer, Edward E (1974). False Models and Post-Data Model Construction. Journal of the American Statistical Association, 69(345):122–131.
  25. Lehmann, Erich L and Joseph P Romano (2006). Testing statistical hypotheses. Springer Science & Business Media.
  26. Libgober, Jonathan (2020). False Positives and Transparency.
  27. Mathis, Jérôme (2008). Full revelation of information in sender–receiver games of persuasion. Journal of Economic Theory, 143(1):571–584.
  28. McCloskey, Adam and Pascal Michaillat (2020). Incentive-Compatible Critical Values. Technical Report 2005.04141.
  29. Miguel, Edward (2021). Evidence on research transparency in economics. Journal of Economic Perspectives, 35(3):193–214.
  30. Myerson, Roger B (1986). Multistage games with communication. Econometrica: Journal of the Econometric Society, pages 323–358.
  31. Olken, Benjamin A. (2015). Promises and perils of pre-analysis plans. Journal of Economic Perspectives, 29(3):61–80.
  32. Savage, L J (1951). The theory of statistical decision. Journal of the American Statistical Association, 46(253):55–67.
  33. Savage, Leonard J (1971). Elicitation of personal probabilities and expectations. Journal of the American Statistical Association, 66(336):783–801.
  34. Sinander, Ludvig (2023). Topics in mechanism design. Lecture notes.
  35. Spiess, Jann (2018). Optimal estimation when researcher and social preferences are misaligned.
  36. Sterling, Theodore D (1959). Publication Decisions and Their Possible Effects on Inferences Drawn from Tests of Significance–Or Vice Versa. Journal of the American Statistical Association, 54(285):30–34.
  37. Tetenov, Aleksey (2016). An economic theory of statistical testing. Technical Report CWP50/16, cemmap working paper.
  38. Tullock, Gordon (1959). Publication Decisions and Tests of Significance—A Comment. Journal of the American Statistical Association, 54(287):593–593.
  39. Vanderbei, Robert J et al. (2020). Linear programming. Springer.
  40. (When) should you adjust inferences for multiple hypothesis testing?
  41. Wald, Abraham (1950). Statistical decision functions. Wiley New York.
  42. Williams, Cole (2021). Preregistration and Incentives.
  43. Yoder, Nathan (2020). Designing Incentives for Heterogeneous Researchers.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets