Methodology
Not how much science an agent knows — whether it can drive discovery.
The University of Scientific Workflow (USW) does not ask how much an agent knows. It asks whether an agent can — like a working scientist — drive genuine, progressive discovery through a real experimental loop. It engages directly with the problems today’s scientists find hard and the open problems they actually need to solve. The aim is to close the gap between benchmark success and real-world impact — so that passing USW means an agent can collaborate with scientists, or autonomously conduct research, in practice.
A common workflow in science
One iteration, eight stages — then around again
Scientific work is iterative rather than one-shot. The validation and interpretation from one cycle feed the next round's literature review and revised hypothesis. Once the loop produces something promising, the candidate is tested or deployed in a real setting, where conditions are less controlled. This is a generalized pattern — it does not hold identically in every field, but it recurs broadly.
- 01
Literature review
Read into the problem and the work already standing around it.
- 02
Ideation
Turn the reading into a candidate direction worth pursuing.
- 03
Hypothesis generation
State what you expect to be true, precisely enough to be wrong.
- 04
Experiment design
Decide what would actually test the hypothesis.
- 05
Implementation
Build the pipeline, the model, the sample — whatever the design requires.
- 06
Experiment execution
Run it — often on an HPC queue, a solver, or an instrument that takes days.
- 07
Validation of results
Check the result holds up before believing it.
- 08
Interpretation of results
Say why it matters, when it fails, and how much it improves on what came before.
One lens will not do
Each problem is verified from its own perspective
Science covers too broad a range for a single verification procedure. Every problem has to be judged from the perspective that reflects its primary scientific objective.
Materials science
Does a candidate discovered within the given search space perform well?
Climate & atmospheric science
Is the forecast or projection accurate?
Molecular biology
Is the proposed causal explanation of a molecular interaction well supported?
Physics
Is the hypothesis empirically validated?
Benchmark key points
What makes USW different
Five commitments that shape how every task is collected, reviewed, and judged.
Rigorous, multi-round peer review
A scientist proposes the task; a peer they invite from their own lab or collaboration judges whether the direction is promising enough to anchor a paper. The loop runs on OpenReview, restricted to participants.
Open-ended, frontier tasks
Challenging, realistic problems with no established answer or paper yet — spanning what scientists keep optimizing and the directions they have no solution for. Each should have the potential to be published in a Nature-family journal or at a top-tier conference.
Cross-scientific coverage
15 meaningful, challenging tasks from real scientists across 10 scientific domains — enough spread to evaluate agents consistently and holistically rather than within one field.
Scientist-based verification
Domain scientists write the rubric that decides whether a discovered solution is genuinely novel — not merely high-scoring on a headline metric.
Systematic evaluation over end-to-end research autonomy
Collection captures the key information behind every major stage of the research workflow. Varying those elements systematically shows which stages an agent handles well and where it struggles.
“Novel and frontier” is hard to pin down even for AI researchers. We use it to mean tasks addressing topics worthy of publication in a Nature-family journal or acceptance at a top-tier conference.
Benchmark construction
From a scientist's open problem to a runnable task
Three steps: the scientist collects the task, a peer reviews it round after round, and an agent runs it to set a baseline.
Task collection
The scientist picks a track, declares whether the task needs a single- or multi-task workflow, writes a research plan across the eight workflow stages, and submits the task requirements in Harbor format.
Multi-round peer review
The proposer invites a peer reviewer with real expertise in the area. Proposal and review iterate until the task meets a standard suitable for a Nature Portfolio journal, with every round tracked.
Running an agent on the task
An agent runs the finished task — establishing a human baseline score and helping the proposer sharpen the rubric.
Step 1-1 · Track selection
Four in five tasks are unsolved
A scientist proposes into one of two tracks. The split is deliberate: the benchmark is weighted toward research that has no answer yet, not toward re-running what is already published.
Reproduction
Work already published, re-run end to end.
- Published in a Nature-family journal or at a top conference within the past two years
- Addresses a problem no other paper covers, or presents a genuinely unique solution
On-going & promising research
Work with no answer yet — the bulk of the benchmark.
- Follow-up research on a Nature-family or top-conference paper from the past year
- Research you are currently running in your lab
- A problem you would like an AI agent to solve
- A problem you consider difficult or novel in your field
Step 1-3 · Research plan
A short brief for every stage
When a scientist first proposes a task they describe each core stage of the workflow — the concise project proposal a researcher writes before starting a study.
Literature review
Which references inspired the task, and which are needed to solve it.
Ideation
The main idea behind the task.
Hypothesis generation
The hypothesis the research starts from.
Experiment design
The experimental design that follows from the idea and hypothesis.
Implementation
How the experiment will be implemented.
Experiment execution
How the implementation will be executed.
Validation of results
How the experimental output will be evaluated.
Interpretation of results
How the output will be analyzed to establish that the finding is a good one.
Step 1-4 · Task requirement
Submitted in Harbor format
Six fields turn a research plan into something an agent can actually be handed and judged on.
Database
The data the experiment runs against.
Simulator
The solvers, models, and instruments the workflow invokes.
Task objective
What the task asks for, and the description that frames it.
Environment spec
The terminal or VM an agent needs to execute it.
Evaluation metric
The quantitative threshold a submitted result is checked against.
Scientist rubric
The scientist's own criteria for whether a discovery counts as novel.
A scientist also declares whether the task needs a single-task workflow or a multi-task workflow — one science task made of sub-tasks, each of which is a full workflow in its own right, run in sequence.
Step 2 · Multi-round peer review
Reviewed by the people who do the science
The proposer invites a peer with real expertise in the area — for a Ph.D. student, usually their advisor; for a professor or postdoctoral researcher, a collaborator. Proposal and review iterate until the task could anchor a Nature Portfolio paper, with the manager tracking every round, every piece of feedback, and every revision. The loop runs on OpenReview, restricted to participants.
Domain scientist
Propose
A scientist — a Ph.D. student or a professor — proposes a practical, meaningful task following the task-construction guideline.
Invited peer
Peer review
A peer the proposer invites — an advisor or a collaborator from the same lab or line of research — judges whether the direction is promising enough to anchor a paper.
Task manager
Revise for agents
The lead student assigned to the task lightly revises the proposal — dataset paths, the main workspace path — so an agent can execute it in a computer environment.
Agent run
Baseline & rubric
An agent runs the task to establish a human baseline score and help the proposer develop the rubric it will be judged by.
Evaluation protocol
Four settings of increasing guidance, judged by a scientist's rubric
Evaluation runs inside an evolutionary loop. The same task is replayed under increasingly scaffolded settings to isolate which stages an agent handles well and where error compounds across the workflow.
No Workflow
Autonomous · unguided
(Problem, Tool list) → Final outcomeThe agent gets only the problem and the tool list, and decides the entire procedure itself.
Workflow-Guided
Scientist's protocol given
(Problem, Tools, Scientist's workflow) → Stage + final outcomesThe agent is handed the scientist's workflow and is observed at each stage as well as on the final outcome.
Stepwise · Self-produced
Agent's own carry-over
Stage N: (Stage N−1 agent output, Sub-problem, Tools) → Stage N outcomeEach stage consumes the agent's own previous output, so errors compound across the workflow.
Stepwise · Human Outcome
Scientist's carry-over
Stage N: (Stage N−1 scientist output, Sub-problem, Tools) → Stage N outcomeEach stage starts from the scientist's own output for the prior stage, isolating per-stage skill.
RSRubric Score
The scientist-written rubric applied to the discovered solution — is the finding genuinely novel, and is the interpretation behind it sound?
OVOutcome Verification
Threshold-based verification of the result: discrete where the science admits a pass/fail, continuous where a score is the meaningful signal.
We do not claim to score the workflow itself. In most scientific problems the space of possible verification procedures is enormous, and the harder the problem, the harder its workflow is to judge. So the workflow is assessed indirectly — through the quality of the outcome it produced, and how genuinely novel the resulting discovery is.
Task coverage
Two verifiability axes — outcome, and workflow
A site-level classification we use to keep the hub legible. Every task sits on two axes: is the outcome quantitatively verifiable, and is the workflow verifiable against per-step targets? Known tasks verify both. Frontier · verifiable tasks verify the outcome but not the workflow. Frontier · rubric tasks verify neither and are judged by a scientist's rubric.
Known
Both axes verifiable — the outcome is quantitatively checkable and every step is scored against the scientist's ground-truth targets.
Frontier · verifiable
An open problem. The outcome stays quantitatively verifiable, but the workflow is not — an expected direction may be supplied, while execution strategy and per-step targets remain open.
Real example: spin-glass D UFrontier · rubric
An open problem whose outcome cannot be quantitatively verified, so it is judged by a scientist's rubric; the workflow is not verifiable either.
How a task is scored follows its axes. Known tasks are scored on both the per-step workflow and the final outcome. Frontier · verifiable tasks have no ground-truth per-step targets — only the outcome is scored, by method-quality and consistency criteria (equilibration, scaling-collapse quality, recovery of known limits, agreement with analytic bounds); any supplied direction is guidance, not a prescribed procedure. Frontier · rubrictasks have no verifiable outcome and are judged by a scientist’s rubric — so frontier work stays legible without a ground-truth answer.
The current landscape
Many science benchmarks — few open problems
A great deal of work has been proposed recently to measure scientific capability. Almost all of it evaluates against problems that already have answers. USW is built the other way round.
The table below compares USW against the two closest in intent.
How USW compares
Against other scientific-workflow benchmarks
Closer to how science is actually done — deploy-first and scientist-driven, mixed environments, GUI-based simulation, and multi-task workflows.
Closing the gap
There is a gap between what AI researchers imagine an agent might do for a hard scientific problem and what working scientists actually think. USW is built with scientists, not just for them — and we go and ask.
Submit a task
Bring the research you are working on now — the goal, the workflow, the tools, and the rubric you would judge it by.
Scientist survey
A preliminary survey of how scientists think about their research and where an agent could help. Results are forthcoming.
Scientist study
We show scientists existing Terminal-Science tasks and ask how hard, how novel, and how publishable they really are — and whether the headline metric is enough. When it is not, they write the rubric.
Survey and study details are shared with participants directly.