Methodology
How Code Review Arena scores a reviewer's findings and tests its patch.
Evaluation flow
- Reviewer output 70 detected
input diff + files
No case stops here.
- Suggested patch 60 with patch
input unified diff
10 stop if no patch
patch_required_ but_ missing - Isolated workspace 50 applied
input pinned snapshot
10 stop if the patch does not apply
patch_apply_ failed - Tests 20 pass
Runs in parallel with validators.
input test command
stops if tests fail
tests_failed - Validators 20 pass
Runs in parallel with tests.
input validator rules
stops if a validator fails
structural_validation_ failed pass both 20 repaired
30 stop if either fails
- Evidence bundle 70 recorded
input hashes
No case stops here.
50 stopped, recorded too
- Metrics 70 detected 20 repaired
input recorded results
No case stops here.
What is measured
A run records detection, localization, patch availability, patch application, tests, structural validation, false positives, cost, and latency.
Detection vs validation
Detection measures whether the reviewer found the seeded bug. Validation measures whether its patch applied and passed the case's tests and validators.
Case design
Cases are seeded pull-request fixtures with known failure behavior, expected files, execution requirements, and validation criteria.
Reviewer-visible context
Reviewer inputs include the pull-request diff and relevant files. Ground truth, validators, and reference patches are not sent to the reviewer.
Patch application
Suggested unified diffs are applied to copied run workspaces. A patch that cannot apply cleanly fails the validation path.
Test execution
Cases can require fixture-owned regression tests after patch application. Local execution is explicit and opt-in.
Structural validators
Validators check repair properties such as tenant scoping, idempotency, event ordering, citation grounding, and API contract behavior.
Metrics
The primary full-mode ranking metric is validated_case_rate. Supporting metrics expose the review-accuracy, repair-success, and trustworthiness dimensions, plus detection, patch apply, test pass, validator pass, false positives, cost, and latency.
Repeatability and provenance
An experiment keeps every attempt made under one protocol, with the baseline run fixed. It reports the spread across attempts and which cases changed outcome. Runs can carry a runner attestation, and packs a signed publisher release.
Baselines
Deterministic controls exercise known success paths, detection-only behavior, malformed patches, missing patches, and custom command integration.
Limitations
The benchmark is curated and small. Validators are hand-authored. Tests do not prove all correctness. Valid fixes may fail if validators are narrow. A signed record proves integrity and origin, not that a run is free from benchmark contamination.
Baseline controls
Controls are deterministic scripts built into the benchmark, not models or products.
| Reviewer | Purpose |
|---|---|
| reference-patch | Proposes each case's known-good reference.patch. |
| control:perfect_patch | Success control. Supplies a correct patch, so every stage should pass. |
| control:keyword_gamer | Adds a comment with the right keywords. Changes no behavior. |
| control:bad_patch | Supplies a patch that does not fix the bug. |
| control:detects_no_patch | Detects the bug and proposes no patch. |
| control:malformed_patch | Returns malformed patch output that is not a unified diff. |
| custom-command | Runs an external reviewer command through structured input and output. |
Scope
Code Review Arena is a local audit harness, not a large-scale public ranking. Large public benchmarks evaluate different things, often at broader scale. This project focuses on local execution-backed validation of patch outputs, paired evidence, and explicit provenance rather than presenting a single resolved percentage as a universal measure of coding-agent quality.