Integration and Collaboration in AI Research Work (Substack companion)

Author/Source: Paul Goldsmith-Pinkham (Yale SOM, NBER) — written companion to Markus Academy Ep. 162-8, Part 8 and capstone of his Claude Code series
Original: https://paulgp.substack.com/p/integration-and-collaboration-in
Demo repository: github.com/paulgp/ipo-bump (public)
Key Ideas
The framing, in economists' language. AI collapsed the cost of doing — coding, data construction, empirical analysis. The cost of verifying moved much less. Goldsmith-Pinkham points out that costly verification is an idea economists have worked on for decades (citing Townsend, and the costly-state-verification literature), and that the fall in action costs raises the relative cost of verification. The result is verification debt: a growing pile of AI-produced results you haven't checked.
The 2×2 (borrowed from Clifford Russell's software-engineering post): cost of making on one axis, cost of verifying on the other. The danger zone is cheap to make, expensive to verify — precisely where AI has moved empirical work. When both costs are high, "there's a huge incentive to spin the slot machine many times and see if the agent just happens to nail the task."
Why "just don't use YOLO mode" isn't the answer. Brunnermeier's question, answered directly: it's a function of how much you verify and internalize, and the more autonomy you grant, the worse it gets. Expensive-to-make work forced you to go slowly, so verification happened as a byproduct of doing. Now the code arrives finished, and understanding is the thing you still have to build.
Three moves, none of them AI tooling — they're what software engineers built for reviewing other people's work, which AI happens to supercharge: (1) review the code as reported, via git and a GitHub repo you and your LLM can read; (2) give feedback that routes back — to the agent, to coauthors, to the permanent record; (3) publish results into the draft so every number traces to code.
The demo: post-IPO abnormal returns, prompted by SpaceX's IPO and Hanno Lustig's posts on index-inclusion demand. Jay Ritter's IPO database as the event sample, CRSP via WRDS for aftermarket returns and benchmark.
Deliberately vague on method, exacting on the paper trail. He left the real decision branches to the agent — CAR vs. buy-and-hold, benchmark, event windows, delisting treatment, Ritter-to-CRSP matching — "similar to if an RA made it for me." What he specified in detail: commit at every meaningful checkpoint with messages saying why, not just what; keep a DECISIONS.md (each choice, alternatives rejected, confidence level); keep a LOG.md (plain-language narrative a coauthor could follow); and print a sample-attrition table at every stage — the N, what dropped, and why.
"The request for verifiability goes in the prompt, before the work happens." This is the post's central discipline.
The headline result: U.S. common-stock IPOs, 1975–2021, mean three-year buy-and-hold abnormal return of −19.7% against the CRSP value-weighted index. In plainer terms: a dollar invested at the close of the first trading day grows to about $0.86 for every dollar the market delivered, and 70.5% of individual IPOs land below the market. It passes the sniff test — it's the documented pattern.
"Work from an AI is work from your RA." The problem is that it will often pass the sniff test — it looks finished and polished. Every senior coauthor has reviewed a junior's work; nothing about the task is new. What's new is that most researchers never adopted the structure that makes it efficient.
Commits change the unit of verification. Work arrives in chunks — scaffolding, data loading, matching, return calculation, estimation and output. You review one bounded diff at a time instead of confronting 800 lines. You can walk the sequence as a story (pull → clean → match → compute returns → estimate) and ask whether the order makes sense. A later change shows as a 12-line diff against a baseline you already reviewed. "A repo with one giant commit doesn't give you any of this; it's just a blob with a .git folder attached."
"Documentation is a map, not a verdict." DECISIONS.md claimed cumulative abnormal returns had been computed as a robustness check. There was no CAR figure and no CAR code. It also mentioned calendar-time portfolios and hadn't done those either. It claimed work it never did. Prompting harder helps a little; the reliable fix is smaller task chunks. "These systems can be as lazy as we are."
A four-item verification checklist for this task, each a place where the code can be perfectly clean and the answer still wrong: did it use Ritter's event dates or silently construct its own (traced through the data-loading commit — it builds from Ritter's offer_date field, correctly); do computed first-day returns reconcile with Ritter's published ones, or does the repo just claim they do; how are delisted firms handled and which way does dropping them bias the mean; is the benchmark defensible and does the sign survive CAR vs. buy-and-hold?
The last item, resolved live: adding CAR alongside BHAR produced a fifth commit and −17.8% at 36 months vs. −19.7% for buy-and-hold, with a different mean-median gap across measures. Review burden: the new diff only.
GitHub Issues serve all three audiences — agent, coauthors, future-you. Issue #1: "Validate return measures against Ritter's initial returns." Then, in a fresh Claude session with no memory: "Please review issue #1 on GitHub, code up an answer, then commit and push. Use the gh command-line tool." The agent pulled the issue, oriented itself in the repo, ran the analysis, pushed, and closed the issue. "You could kick this off and check the answer from your phone."
The issue taught him something real: the exact first-day return was unrecoverable, because Claude had built everything on CRSP monthly files — monthly data has a closing price for the IPO month but cannot isolate the first trading day. Issue #3 followed ("switch to daily and aggregate up"), auto-linked to #1 by typing #1. That issue is now closed too: the daily reconstruction confirms the monthly results, and along the way the repo grew a first-day-return analysis recovering offer prices from Ritter's files, Capital IQ, and SEC 424B prospectuses. "Each round of review generated the next round of work, and the record shows the whole chain."
Two practices make the loop work: be specific ("this seems off" gets a vague fix; "diagnose why we lose four IPOs at the CRSP match stage" is actionable and checkable), and anchor feedback to a specific line of a specific commit so it carries its own context.
For coauthors: an email thread about benchmark choice gets lost; an issue stays indexed against the code. You can see what's been checked, by whom, and what's still open — verification distributed across the team and made visible. His favorite trick: after a seminar, file each question as its own issue, then have the agent draft the best available answer for each and flag which require genuinely new analysis.
Pull requests are where professional code review formally happens, and worth adopting eventually — but issues alone get most of the value, so he kept the demo there.
Can another AI do the verifying? It can, people do, and a second model reviewing the first catches real things. But he returns to the 1970s IBM training-manual line: "a computer can never be held accountable." A human signs off; the workflow's job is to make that as painless and structured as possible.
No hand-typed numbers. The pipeline emits results.tex and figure.pdf; the paper ingests them via \input and \includegraphics{}. With AI in the loop this matters more, because otherwise results get written into the draft from the model's context window rather than from code.
Overleaf syncs directly with a GitHub repo — with a caveat the video skipped: this sits behind Overleaf's paid tier, though only the project owner needs the subscription; once linked, any collaborator can hit sync. Dropbox↔Overleaf is the alternative but is also premium, so there is no free route to automatic syncing. In the demo he linked the repo, pulled everything in, wrote a skeleton referencing the results figure, pushed back to GitHub, had Claude write the results section from the command line, and pulled again — draft updated, nothing copy-pasted, and the draft's edit history now lives in the same git record as the code.
The payoff is provenance. When a coauthor questions a coefficient you walk backward: which table, which script, which commit, which logged decision. "Let me dig through my files" becomes a five-minute trace.
The essential caveat, stated plainly: a correctly wired pipeline guarantees the number in the paper matches the number the code produced, but does not promise correct code. "Clean provenance can faithfully publish a wrong estimate — which is why Part A comes before Part C."
Five takeaways: (1) ask for the trail before the work happens; (2) review commits, not full codebases — the unit of verification should be a bounded diff; (3) documentation is a map, not a verdict; (4) route feedback through issues; (5) no hand-typed numbers — code emits, the draft ingests, git connects them, and review the method before you wire it in.
On adoption: researchers mostly never adopted git because the perceived setup cost exceeded the benefit. "That cost is now roughly zero... it has the spirit of a good software engineer living somewhere deep inside. You get the best practices without the years of accumulated scar tissue that software engineers paid for them."
On the future: five years is unknowable and one year ago looked completely different. His guess is the rough edges smooth out and it becomes clearer which parts are human problems and which are computer problems. "I don't think this automates research away; it's comparative advantage all the way down, and taste, judgment, and accountability stay on our side of the ledger." Thirty years ago many-fixed-effects regressions were a real computational constraint shaping what questions people asked; now we run a thousand without thinking. Verification will feel like that eventually — still mattering just as much, and no longer the constraint.
He also notes he has a paper with Tianshu Lyu on estimating causal effects in financial return data, for readers interested in the abnormal-returns methodology behind the demo.
Summary

The written companion is substantially more precise than the video and worth reading on its own. It states the problem in economists' terms — costly verification, an idea the discipline has worked on since the 1970s — and observes that AI has changed the relative price: doing got cheap, checking didn't, and the gap accumulates as verification debt. The 2×2 makes the danger zone explicit: work that is cheap to produce and expensive to confirm.

The methodological contribution is the prompt design. Goldsmith-Pinkham was deliberately vague about the empirical choices (CAR vs. buy-and-hold, benchmark, windows, delisting, matching) so the agent's judgment would be on display and reviewable, and exacting about the paper trail: checkpoint commits with messages saying why, a DECISIONS.md recording each choice with rejected alternatives and a confidence level, a LOG.md narrative for coauthors, and a sample-attrition table at every stage. The principle he wants readers to take is that the request for verifiability belongs in the prompt, before the work happens — after the fact, you have a blob.

The results are documented in a public repo. IPOs from 1975–2021 show a mean three-year buy-and-hold abnormal return of −19.7% against the CRSP value-weighted index, with 70.5% of IPOs below market and a dollar invested growing to about $0.86 per market dollar. What makes the post valuable is that the two agent failures are reported with their consequences intact. DECISIONS.md claimed a CAR robustness check that did not exist in code or output — documentation as a claim, not a record. And resolving issue #1 revealed the entire pipeline had been built on CRSP monthly files, making the first-day return unrecoverable; the daily reconstruction in issue #3 confirmed the original estimates and additionally produced a first-day-return analysis pulling offer prices from Ritter, Capital IQ, and SEC 424B filings. Both problems surfaced only because the structure made them findable.

Parts B and C describe the machinery. GitHub Issues serve the agent, coauthors, and future-you from one artifact; a fresh session pointed at "issue #1" and the gh CLI can orient itself and close the loop unattended. Overleaf↔GitHub sync enforces the rule that no number is typed by hand — with the practical caveat, absent from the video, that both Overleaf sync routes require a paid tier. The post ends on the limit of its own method: clean provenance can faithfully publish a wrong estimate, so review the method before wiring it into the draft.

Relevance to Economics Research

This is the most complete verification protocol for AI-assisted empirical work anywhere in the wiki, and it is specified concretely enough to adopt tomorrow. The four artifacts requested in the prompt — checkpoint commits explaining why, DECISIONS.md with confidence levels, LOG.md, and a stage-by-stage sample-attrition table — map directly onto what a careful empirical economist already wants from an RA. The attrition table in particular is a discipline-specific addition worth copying regardless of whether AI is involved.

The two documented failures are the most instructive content. That the agent's own decision log claimed a robustness check it never ran should recalibrate how anyone reads AI-produced documentation: it is a map of intent, not evidence of execution, and it must be checked against code and output. That the pipeline silently used monthly rather than daily CRSP data is the canonical case of clean code producing a wrong answer — no bug, a defensible-looking choice, a materially different analysis. Neither is caught by reading the final figure.

The provenance principle answers the hallucinated-number problem architecturally rather than through vigilance, and Goldsmith-Pinkham's own caveat keeps it honest: provenance guarantees fidelity, not correctness. Review the method first. The paid-tier caveat on Overleaf sync is a real cost worth knowing before promising a coauthor the setup.

Finally, the framing of the endgame is measured in a way most AI-and-research commentary is not: comparative advantage all the way down, with taste, judgment, and accountability staying human — and verification eventually becoming routine the way many-fixed-effects regressions did, still mattering just as much while ceasing to be the constraint.