Skip to content

Pre-Analysis Plans and AI

Pre-analysis plans (PAPs) in the AI context address how AI tools interact with research pre-registration — both the risks that AI creates for specification searching and the opportunities for more thorough pre-analysis planning.

Context & Background

Pre-analysis plans commit researchers to specific hypotheses, data, and methods before seeing results. AI tools create both challenges and opportunities:

  • Risk: AI makes it easy to quickly explore many specifications, potentially undermining pre-registered commitments
  • Opportunity: AI can help draft more comprehensive PAPs by identifying potential analyses and robustness checks
  • Verification: AI can check whether analysis code matches the pre-registered plan
  • Documentation: AI can generate detailed analysis logs showing exactly what was run

Practical Implications

  • Write PAPs before using AI for analysis: Commit to your plan before AI-assisted exploration
  • Use AI to strengthen PAPs: Have AI identify potential confounds and robustness checks during the planning stage
  • Log all AI-assisted analysis: Create complete records of what the AI did, not just what you report
  • Separate exploration from testing: Use AI for exploratory analysis, but clearly mark it as such

Key Sources