Can AI agents verify good science as domain scientists do? We evaluate scientific critique, rubric generation, and scientific taste on open-ended, novel research tasks.
Research agents produce papers and artifacts. Evaluator agents judge those outcomes, with domain scientists providing the reference.
Where does the science fall short?
Identify weaknesses in an AI-generated scientific outcome and support each criticism with evidence from the paper or research artifacts.
The verification task
What did the rubric miss?
Recognize scientific problems that an existing rubric does not cover, then revise its criteria in light of the observed outcomes.
The verification task
Which outcome is better science?
Compare scientific outcomes for novelty, meaning, and quality of evidence, and assess how an agent’s preferences align with scientists’ judgments.
The verification task
Scientists’ critiques, rubric updates, and preferences are the evaluation references. Even quickly saturating research tasks can help reveal how verification itself can be verified.
Scientists propose and review tasks through OpenReview. AI researchers run agents, then return papers and artifacts for scientific feedback and another research round.
Two connected research loops
1 /Propose
Research question, data, environment, and an initial scientific rubric.
2 /Peer review
The SciVeri-Bench team assigns three relevant reviewers to assess open-endedness, novelty, and frontier relevance, then records its decision.
3 /Implement
The proposer and task manager prepare a runnable task package.
Run agents
Research in the prepared environment
Collect outcomes
Papers and supporting artifacts
Revise the next round
Updated instructions and rubric
Scientist feedback
Critiques, criteria, and preferences
Follow the next step in the research:
Scientists identify weaknesses and refine the rubric. Their feedback guides another execution round; satisfactory outcomes return to the three assigned task reviewers.
Scientists review weaknesses and meaningful findings, citing evidence in papers, figures, tables, and artifacts.
The harness runs agents, collects outcomes, and passes the revised rubric and feedback to the next round.
Each agent investigates the task and produces a paper with code, data, and supporting artifacts. Scientific feedback and revised rubrics guide subsequent rounds.
Opus 5.5
GPT-6-Astra
GPT-6-Astra
Weaknesses lead to changes in the rubric, while scientists compare the research outcomes each round. The final outcomes and rubric return to the task reviewers.
Round 1's rubric comes from the proposal. The remaining feedback, revisions, and rankings are illustrative; agents have not run on this example task.
The SciVeri-Bench team assigns three reviewers with relevant expertise to each proposal. Proposers respond to their reviews, and the team records the final decision in OpenReview.
Scientist-in-the-Loop collection
Multi-round peer review
Task proposer
Domain scientist
3 task reviewers
Assigned by SciVeri-Bench
Evaluation via agent execution
Task manager
AI researcher
Coding agents
Research execution
Where does the science fall short?
What did the rubric miss?
Which outcome is better science?
Reviewers explain the contribution a successful result could make, the evidence it would need, and what makes the question open-ended. Discussion strengthens the task before and after agent execution.
Reviewers assess publication potential, novelty, scientific significance, open-endedness, and feasibility, with written reasoning and a confidence rating. Feasibility is considered separately from scientific value.
Highlighted scores are examples.
Three perspectives on a plant-ecology proposal
The SciVeri-Bench team assigns three reviewers with relevant expertise. Each independently assesses the whole proposal and records scores and reasoning in OpenReview.
Reviewer 1
Novelty & baselines
Test what community context adds beyond a baseline that accounts for observation history.
Reviewer 2
Scientific significance
Explain how the study separates habitat effects from neighbouring species, including what a null result would teach us.
Reviewer 3
Validation & feasibility
Check robustness across sites and held-out observations, and state the data and compute needed for the comparison.
