Building a Grader for Autonomous Coding Agents

A packaged TaskGrader allows CORAL to evaluate agent submissions using custom Python logic while keeping evaluation dependencies isolated from the agent's workspace.

This guide walks through creating a grader package, configuring it in task.yaml, validating it with coral validate, and diagnosing common evaluation failures.

What you'll build

By the end of this guide, you'll have:

• Created a packaged TaskGrader using coral init.
• Configured the grader in task.yaml.
• Added grading logic using the TaskGrader API.
• Validated the task with coral validate.
• Learned how to diagnose common grader failures.

Quick start

Create a new task using the CORAL CLI:

coral init my-task

This scaffolds a new task with a packaged grader, a starter task.yaml, and the directory structure needed to begin writing custom evaluation logic.

Throughout this tutorial, we'll use the bundled examples/erdos task as a concrete example. It demonstrates a packaged grader, task.yaml configuration, and validation workflow using a real optimization task from the CORAL repository.

Layout

my-task/
├── task.yaml
├── seed/
│   └── solution.py
└── grader/
    ├── pyproject.toml
    └── src/my_task_grader/
        ├── __init__.py
        └── grader.py

The bundled examples/erdos task follows this same structure:

Package the grader

Each grader is a standalone Python package. This allows CORAL to install its dependencies in an isolated virtual environment without affecting either the agent's workspace or the main CORAL installation.

The package metadata lives in grader/pyproject.toml, while the grading logic is implemented under grader/src/. During validation, the commands listed under grader.setup install this package before any evaluations are run.

Configure the grader

The grader configuration lives in task.yaml. It tells CORAL how to locate, install, and execute your grader during validation and task runs.

The bundled examples/erdos task provides a complete packaged grader. The configuration below is taken from its task.yaml.

grader:
  entrypoint: "erdos_grader.grader:Grader"
  setup:
    - "uv pip install -e ./grader"
  timeout: 1100
  direction: maximize
  args:
    program_file: "initial_program.py"

entrypoint

The entrypoint tells CORAL which Python class implements your grader. It uses the format:

package.module:ClassName

For example:

entrypoint: "erdos_grader.grader:Grader"

CORAL imports this class whenever it needs to evaluate an agent submission.

setup

The commands listed under setup prepare the grader environment before any evaluations run.

setup:
  - "uv pip install -e ./grader"

Installing the grader in editable mode makes your package available inside the isolated grader virtual environment while allowing you to continue developing it locally.

timeout

The timeout limits how long a single evaluation can run.

timeout: 1100

If grading exceeds this limit, CORAL stops the evaluation and records it as a failure.

direction

The direction field tells CORAL whether higher or lower scores are preferred.

direction: maximize

Use maximize when higher scores represent better solutions, or minimize when lower scores indicate better performance.

args

Use args to pass custom configuration into your grader.

args:
  program_file: "initial_program.py"

These values are available through self.args inside your TaskGrader implementation, making it easy to customize grader behavior without modifying the grader code.

Create the grader

The grader implementation is responsible for evaluating an agent's submission and returning a score. Every packaged grader inherits from TaskGrader and implements the evaluate() method.

This tutorial focuses on the complete workflow for building a grader. For a complete reference of the available TaskGrader helper methods and advanced configuration options, see Writing a Custom Grader.

The bundled examples/erdos task contains a complete packaged grader that implements this workflow. The simplified example below introduces the TaskGrader API, while the Erdős task demonstrates how those concepts are applied in a complete implementation.

A minimal grader looks like this:

from coral.grader import TaskGrader
from coral.types import ScoreBundle

class Grader(TaskGrader):
    def evaluate(self) -> float | ScoreBundle:
        result = self.run_program("solution.py")
        return self.score(1.0, "Evaluation completed successfully.")

If you'd like to see a complete implementation, explore:

• examples/erdos/task.yaml
• examples/erdos/grader/src/erdos_grader/grader.py

Together, these files show how a packaged grader is configured, executes the submitted program, validates its output, computes a task-specific score, and returns detailed feedback using ScoreBundle.

The Erdős grader also demonstrates several commonly used TaskGrader helpers, including self.run_program(), self.args, self.fail(), and self.score(), making it a useful reference alongside the API documentation.

Returning a score together with a clear explanation makes evaluation results easier to understand. Helpful feedback allows contributors to quickly identify why a submission passed or failed without inspecting the grader's implementation.

When CORAL runs an evaluation, it loads the class defined by the entrypoint in task.yaml and calls evaluate() for every submission.

Inside evaluate(), you typically:

1. Run the submitted program.
2. Verify that its output is valid.
3. Compute a score.
4. Return the result together with useful feedback.

Example evaluation flow

The bundled examples/erdos task demonstrates a complete packaged grader for an optimization problem. Its grader follows this evaluation pattern:

1. Read the target program from self.args.

2. Verify that the program exists.

3. Execute it with a timeout.

4. Validate the returned values.

   Typical correctness checks include verifying that the program returns the expected data type, shape, value ranges, and any task-specific constraints before computing a score. Failing fast with a clear explanation makes grading easier to understand and debug.

5. Compute the final score.

6. Return the score together with an explanation.

Unlike the minimal example shown earlier, the Erdős grader performs task-specific validation before scoring. It verifies that the submitted program exists, executes within the configured timeout, checks that the returned values satisfy the problem constraints, computes the final score, and returns a ScoreBundle containing both the score and a human-readable explanation.

This structure works well for most optimization and coding tasks because it separates execution, validation, and scoring into clear steps.

Validate the grader

Before launching agents, validate the task to ensure that the grader, configuration, and seed solution all work together correctly.

Since this guide uses the bundled examples/erdos task throughout, validate it with:

coral validate examples/erdos

If you're building your own task instead, replace examples/erdos with the path to your task directory.

Once validation completes successfully, your packaged grader is ready to be used with coral start.

During validation, CORAL:

1. Creates an isolated grader virtual environment.
2. Executes the commands listed under grader.setup.
3. Installs the packaged grader.
4. Runs the seed solution.
5. Calls the grader's evaluate() method.
6. Reports the resulting score or any validation errors.

Running validation before starting a long optimization run helps catch configuration and grading issues early.

Debugging common failures

Validation failures are usually caused by configuration mistakes, runtime errors, or unexpected outputs from the submitted program. Running coral validate before starting a task makes these issues much easier to identify.

Program file not found

If the grader cannot locate the configured program, validation will fail before evaluation begins.

If task.yaml contains:

Program file not found: initial_program.py

For example, if task.yaml contains:

args:
  program_file: "initial_program.py"

make sure that file exists in the task workspace and matches the filename expected by your grader.

Runtime errors

If the submitted program raises an exception or exits with a non-zero status, the grader reports the failure instead of returning a score.

For example:

Evaluation failed:
Traceback (most recent call last):
...

The traceback usually points to an exception in either the submitted program or the grader itself. Running the program independently can help isolate the cause before validating again.

For the bundled examples/erdos task, you can run the seed program directly:

python -c "import sys, os; sys.path.insert(0, 'examples/erdos/seed'); import initial_program; print(initial_program.run())"

If the error occurs while setting up or installing the grader, rerun validation with verbose logging to inspect the grader setup steps:

coral validate examples/erdos --verbose

Invalid outputs

The grader should verify that submitted results match the expected format.

Common validation checks include:

• incorrect return types;
• unexpected array or object shapes;
• values outside an expected range;
• missing required outputs.

Providing clear error messages helps contributors quickly understand what needs to be fixed.

Timeouts

For example:

Evaluation timed out after 1100s

If evaluation exceeds the configured timeout, CORAL terminates the grading process and records the attempt as failed.

Increase the timeout only when necessary. Otherwise, prefer reducing the runtime of the evaluated program.

Protecting private evaluation data

Some tasks require hidden datasets, answer keys, or reference implementations that agents should not be able to access.

Files listed under grader.private are copied into .coral/private/, where they remain available to the grader but are hidden from agent workspaces.

grader:
  private:
    - "answers"

Inside your grader, access these files through self.private_dir instead of reading them directly from the task directory. This keeps evaluation data separate from the visible workspace while allowing the grader to use it during scoring.

Best practices

When writing a custom grader:

• Keep evaluation deterministic whenever possible. The same submission should produce the same score each time it is evaluated.
• Avoid introducing uncontrolled randomness into grading logic, as inconsistent scores make results difficult to compare and debug.
• Return clear explanations with both successful scores and failures.
• Validate inputs before computing a score.
• Store hidden evaluation data under grader.private instead of inside the visible task directory.
• Test changes with coral validate before launching a full run.

Next steps

Now that you've created and validated a packaged grader, explore the bundled examples/erdos task to see a complete production-style implementation, then continue with the guides below for more advanced grading patterns:

• Writing a Custom Grader
• Task Examples