Evaluation flow

  1. Reviewer output 70 detected

    input diff + files

    No case stops here.

  2. Suggested patch 60 with patch

    input unified diff

    10 stop if no patch patch_required_but_missing

  3. Isolated workspace 50 applied

    input pinned snapshot

    10 stop if the patch does not apply patch_apply_failed

  4. Tests 20 pass

    Runs in parallel with validators.

    input test command

    stops if tests fail tests_failed

  5. Validators 20 pass

    Runs in parallel with tests.

    input validator rules

    stops if a validator fails structural_validation_failed

    pass both 20 repaired

    30 stop if either fails

  6. Evidence bundle 70 recorded

    input hashes

    No case stops here.

    50 stopped, recorded too

  7. Metrics 70 detected 20 repaired

    input recorded results

    No case stops here.

audit_v1, 7 built-in controls × 10 cases

What is measured

A run records detection, localization, patch availability, patch application, tests, structural validation, false positives, cost, and latency.

Detection vs validation

Detection measures whether the reviewer found the seeded bug. Validation measures whether its patch applied and passed the case's tests and validators.

Case design

Cases are seeded pull-request fixtures with known failure behavior, expected files, execution requirements, and validation criteria.

Reviewer-visible context

Reviewer inputs include the pull-request diff and relevant files. Ground truth, validators, and reference patches are not sent to the reviewer.

Patch application

Suggested unified diffs are applied to copied run workspaces. A patch that cannot apply cleanly fails the validation path.

Test execution

Cases can require fixture-owned regression tests after patch application. Local execution is explicit and opt-in.

Structural validators

Validators check repair properties such as tenant scoping, idempotency, event ordering, citation grounding, and API contract behavior.

Metrics

The primary full-mode ranking metric is validated_case_rate. Supporting metrics expose the review-accuracy, repair-success, and trustworthiness dimensions, plus detection, patch apply, test pass, validator pass, false positives, cost, and latency.

Repeatability and provenance

An experiment keeps every attempt made under one protocol, with the baseline run fixed. It reports the spread across attempts and which cases changed outcome. Runs can carry a runner attestation, and packs a signed publisher release.

Baselines

Deterministic controls exercise known success paths, detection-only behavior, malformed patches, missing patches, and custom command integration.

Limitations

The benchmark is curated and small. Validators are hand-authored. Tests do not prove all correctness. Valid fixes may fail if validators are narrow. A signed record proves integrity and origin, not that a run is free from benchmark contamination.

Baseline controls

Controls are deterministic scripts built into the benchmark, not models or products.

ReviewerPurpose
reference-patchProposes each case's known-good reference.patch.
control:perfect_patchSuccess control. Supplies a correct patch, so every stage should pass.
control:keyword_gamerAdds a comment with the right keywords. Changes no behavior.
control:bad_patchSupplies a patch that does not fix the bug.
control:detects_no_patchDetects the bug and proposes no patch.
control:malformed_patchReturns malformed patch output that is not a unified diff.
custom-commandRuns an external reviewer command through structured input and output.

Scope

Code Review Arena is a local audit harness, not a large-scale public ranking. Large public benchmarks evaluate different things, often at broader scale. This project focuses on local execution-backed validation of patch outputs, paired evidence, and explicit provenance rather than presenting a single resolved percentage as a universal measure of coding-agent quality.