ScientistOne: Towards Human-Level Autonomous Research via Chain-of-Evidence

  • Author/Source: Google Cloud AI Research

  • Original: https://scientist-one.github.io/ — project page with 21 autonomously generated papers and their solver code

  • Key Ideas

  • The core claim: ScientistOne autonomously generates research papers in which every claim traces to code, data, or literature — a property the authors call a Chain-of-Evidence (CoE) — while matching or exceeding human expert performance on frontier algorithm-discovery tasks.
  • Three-stage pipeline, designed to satisfy CoE natively rather than retrofit it: a Problem Investigator that reads up to 100 full-text PDFs per topic and emits grounded experiment briefs; a Discovery Engine running a parallel explore-exploit search tree over candidate algorithms; and a Paper Writer with Claim Verifier that checks every claim in the draft against its declared evidence source before the paper is finalized.
  • Headline integrity numbers: 0 / 337 hallucinated references (all verified against real publications); 12 / 12 on score verification (every claimed result reproduces under re-evaluation); 14 / 15 on method-code alignment (paper descriptions match the submitted code); and a 98% numerical Claim Provenance Rate — quantitative claims traceable to experiment logs.
  • CoE Audit is offered as a general instrument, separable from the system: a post-hoc audit of whether claims in a completed paper are supported by its artifacts, comprising four checks — Score Verification, Specification Violation, Reference Verification, and Method-Code Alignment — each targeting a distinct way a claim can lose its grounding.
  • The audit's finding about the field is the most striking result. Applied to 75 papers from five autonomous research systems across five frontier systems-research tasks: every baseline exhibits at least one systematic integrity failure. Hallucinated reference rates reach 21%; score verification passes in as few as 42% of papers; method-code alignment ranges from 20% to 80%.
  • Per-system integrity scorecard (score verification / spec violations / hallucinated refs / method-code alignment): Sakana ASv2 42% / 10 of 15 / 0% / 33%; AutoResearchClaw 42% / 0 / 1.5% / 20%; DeepScientist 92% / 0 / 20.9% / 33%; AI-Researcher 75% / 1 / 9.5% / 80%; ScientistOne 100% / 0 / 0% / 93% — the only system leading on all four. The authors attribute this to maintaining evidence chains throughout the pipeline rather than adding them at write-up time. (They flag a caveat: Sakana's audited code includes non-solver scaffolding by design, inflating two of its counts, so cross-system comparison on those checks should exclude it.)
  • On raw solution quality, everyone is already at or above the human expert baseline. On the ADRS benchmark's five systems-research tasks (Prism, Cloudcast, EPLB, LLM-SQL, TXN), all systems match or exceed human experts — "consistent with prior observations that LLM-based agents rapidly converge to similar solution quality." ScientistOne leads on Cloudcast and EPLB and is on par with specialized algorithm-discovery systems (AdaEvolve, EvoX) that produce no papers at all.
  • The argued contribution is therefore not capability but capability without an integrity tradeoff — ScientistOne is presented as the only system pairing competitive solver scores with full evidence-chain verifiability.
  • Generalization test, run unmodified: six tasks spanning medical imaging, fine-grained recognition, 3D perception, and parameter-constrained language modeling — five from MLE-Bench (Kaggle competitions at Medium and High difficulty) plus Parameter Golf, a live competition to train the best language model under strict size constraints. Both systems got a knowledge base of official leaderboard solutions up to an April 27, 2026 cutoff.
  • Results against DeepScientist: Gold Medal on 3D Object Detection (0.1763) where DeepScientist scores 0.0000 and fails entirely; Above Median on AI4Code (0.8356 vs. 0.6964); Silver on iMet 2020 and iNaturalist 2019; Gold on RSNA Brain Tumor; and top-1 leaderboard performance on Parameter Golf (1.0600, SOTA as of the cutoff) where DeepScientist exceeded the 16MB artifact size limit and submitted an invalid entry.
  • A claim of genuine algorithmic novelty, not just recombination: on Parameter Golf, ScientistOne introduced Hessian-diagonal-weighted SVD initialization and an alternating-least-squares refinement loop with GPTQ — techniques novel to that leaderboard — and internal ablations isolate the ALS loop as the primary driver. DeepScientist, by contrast, introduced no algorithmic changes, only environment and portability adjustments.

  • Models used: Gemini 3.1 Pro for ScientistOne and most baselines; Gemini 3.0 Pro for AdaEvolve and EvoX. Baseline scores come from independent canonical evaluator re-runs; human and specialized-system scores from original publications.

  • Summary

ScientistOne is Google Cloud AI Research's entry in the autonomous-research-system race, and its distinguishing move is to compete on verifiability rather than raw capability. The premise is that the field has quietly converged: on the ADRS benchmark's frontier systems-research tasks, every autonomous system — theirs included — already matches or beats human expert baselines, and the differences among them are small. What is not converged, they argue, is whether the resulting papers can be trusted, and the evidence they marshal for that is more interesting than their own system's scores.

The instrument is CoE Audit, a post-hoc check on whether a finished paper's claims are actually supported by its artifacts. It runs four tests, each corresponding to a distinct way a claim can detach from its evidence: does the reported score reproduce when re-run, does the solution violate the task specification, do the cited references exist, and does the described method match the submitted code. Applied to 75 papers from five systems, it finds that every baseline has at least one systematic integrity failure — reference hallucination up to 21%, score verification passing as rarely as 42%, method-code alignment as low as 20%. These are not marginal defects; a paper whose reported number does not reproduce and whose method description does not match its code is not a paper.

ScientistOne's own architecture is a three-stage pipeline — Problem Investigator (reads up to 100 full-text PDFs to produce grounded experiment briefs), Discovery Engine (parallel explore-exploit search over algorithms), and Paper Writer with an integrated Claim Verifier that checks each claim against its declared source before the draft is finalized. The authors are explicit that the integrity results follow from maintaining evidence chains throughout the pipeline rather than bolting verification onto the write-up step, and they report 0/337 hallucinated references, 12/12 score verification, 14/15 method-code alignment, and a 98% numerical claim provenance rate.

The generalization tests are the strongest capability evidence. Run unmodified on MLE-Bench Kaggle tasks and a live Parameter Golf competition, the system earned Gold on 3D Object Detection where the DeepScientist baseline scored zero, and took top-1 leaderboard position on Parameter Golf — introducing, and ablating, techniques the site describes as novel to that leaderboard. As with all such claims, the evidence available is the project's own; the 21 generated papers and solver code are posted publicly for inspection.

  • Relevance to Economics Research

Two things here matter to economists, and neither depends on caring about systems-research benchmarks.

First, CoE Audit is a template for auditing AI-assisted empirical work in economics, and its four checks translate almost directly. Score verification is "does the reported coefficient reproduce when the code is re-run." Method-code alignment is "does the paper's stated specification match the regression actually estimated" — which is precisely the failure Goldsmith-Pinkham hit when his pipeline silently used CRSP monthly data while the write-up implied daily (summaries/integration-collaboration-substack). Reference verification is the concepts/citation-hallucination problem. Specification violation is the closest analogue to sample or exclusion-restriction violations. That an audit of this kind is cheap to run and finds systematic failures in every system tested is the argument for building it into any AI-assisted research workflow.

Second, the reported failure rates are a calibration point. A 21% hallucinated-reference rate and score verification passing in 42% of papers is what fully autonomous research production currently looks like without evidence-chain design — from systems whose solution quality already exceeds human experts. That gap between capability and integrity is exactly the shape of the concern running through this wiki: the bottleneck is verification, not production. It also sharpens the automated researcher debate — the question is not whether these systems can find good solutions (they can) but whether their write-ups can be believed.

The claim of genuine algorithmic novelty on Parameter Golf, with ablations isolating the responsible component, is a data point for the creativity question that Sandomirskiy approaches from mathematics. Read together, they point the same way: novelty is appearing in verifiable, benchmarked domains first, where a claim either holds or it doesn't.

Standard caution applies. This is a project page from the team that built the system, the audit was designed by the same team that leads on it, and 12–15 papers per check is a small sample. The audit methodology is the durable contribution; the ranking is the part to treat as a claim.