Skip to content

Contribute a research task

Bring an open-ended question from your field and help judge the science that research agents produce.

Become a task contributor Task proposal guide

Start with the contributor form. Assigned participants submit proposals and scientific feedback through OpenReview.

Prepare your proposal

Start with ongoing work, a challenging open problem, or a direction inspired by a recent paper.

01A scientific question
Describe an open-ended, novel objective, its importance to your field, and the contribution a successful result would make.
02Data & environment
Provide the data and software needed to investigate the question. Custom resources must be shareable when the task is released.
03An initial rubric
Explain what good science would look like. Criteria can evolve as research outcomes reveal weaknesses.
04Domain & keywords
Describe the scientific expertise needed to review your task. The SciVeri-Bench team assigns three relevant reviewers after submission.

Open-ended

Explain why a single scalar score cannot adequately capture the quality of the science.

Novel

Propose a question with the significance and research potential of work in a Nature-family journal.

Feasible

Describe the data, tools, compute, and practical scope, from a tractable first experiment to a longer-term moonshot.

View an example task proposal

A plant-ecology example connects the six proposal sections to an open scientific question and an initial rubric for validity, insight, interpretation, and reproducibility.

The author's identifying information is redacted in the supplied example.

Full-size image PDF
Follow the proposal writing guide →

What participating scientists gain

Co-authorship on SciVeri-Bench

Scientists whose tasks are included in the benchmark—the task proposers and their reviewers—are co-authors of SciVeri-Bench.

Intellectual property stays with you

Agent-generated outcomes belong to the task proposer and reviewers. You can use them in your publications; including benchmark co-leads as co-authors is not required.

The project supports benchmark compute and engineering. Task managers help turn accepted proposals into runnable experiments.

Two roles. One scientific conversation.

  • Propose the scientific question
  • Review papers and research artifacts
  • Refine the task and scientific rubric
  • Check the task package and environment
  • Run coding agents in the sandbox
  • Return AI outcomes and rubric scores
Task & verification OpenReviewAI outcomes
Scientists provide the question and scientific judgment. The task manager handles running the AI agents.

Prepare the runnable task

After proposal review, prepare instructions, a task configuration, an initial rubric, a Docker environment, and a verification script. Add the required data and helper files, without bundled agentic skills.

A Task Manager can help with implementation while you focus on the research question and scientific review.

Packaging & upload instructions

Task package

task/
  instruction.md
  task.toml
  rubric.md
  environment/Dockerfile
  tests/test.sh

Your role in each research round

  1. 01Propose & review

    Submit the scientific question and initial rubric. The SciVeri-Bench team assigns three reviewers with relevant expertise and makes the final decision after their reviews.

  2. 02Examine the outcomes

    Read the papers and artifacts. Identify scientific weaknesses and the evidence supporting your critiques.

  3. 03Compare & revise

    Compare two or three outcomes, then revise the rubric to address what the experiments revealed.

  4. 04Continue the research

    Agents receive feedback and a revised rubric. When you are satisfied, final outcomes return to the three assigned task reviewers.

Have an early idea? Share your research interests and we will help develop a proposal. Meet the task contributors →