Skip to main content
Code Review Arena
MethodologyDocs

Evaluate

Run one reviewer on one pinned case and see whether its patch passes.

  • Repair gate→Run one reviewer on one case from the browser.
  • Experiments→Repeated trials under one frozen measurement protocol.
  • Compare→Two runs aligned case by case, with the stages that differ marked.
  • Verify→Reproducible checks and control outcomes for this checkout.
GitHubRun a repair gate

Evaluate

  • Repair gateRun one reviewer on one case from the browser.
  • ExperimentsRepeated trials under one frozen measurement protocol.
  • CompareTwo runs aligned case by case, with the stages that differ marked.
  • VerifyReproducible checks and control outcomes for this checkout.

Results

  • LeaderboardYour local runs, ranked by validated_case_rate.
  • RunsEvery saved run, with detection, validation and trust status.
  • ReportsAudit reports for the audit_v1 and audit_v2 packs.
  • CasesThe seeded pull requests, their tests and validators.

Docs

  • MethodologyHow a case is scored, stage by stage.
  • DocsRun, inspect and interpret Code Review Arena locally.
Run a repair gate →GitHub

Documentation

Run, inspect, and interpret Code Review Arena locally.

All documentation

Start

  • Getting started
  • CLI reference
  • Troubleshooting

How scoring works

  • Metrics
  • Patch validation
  • Structural validators
  • Reference patches

Benchmark packs

  • Audit pack v1
  • Audit pack v2
  • Adding cases

Reviewers

  • Adding reviewers
  • Custom command reviewer

Reports and CI

  • Audit report
  • GitHub Action

Start

  • Getting startedInstall, validate a pack, run one repair gate and start the API.→
  • CLI referenceCommands for validation, execution, ranking, reports and serving.→
  • TroubleshootingFixes for common install, report, API and dashboard problems.→

How scoring works

  • MetricsWhat detection_f_beta and validated_case_rate each measure.→
  • Patch validationWhat Arena does with a reviewer's patch, step by step.→
  • Structural validatorsThe repair-property checks that run beside the tests.→
  • Reference patchesThe known-good fixes used to check the pipeline itself.→

Benchmark packs

  • Audit pack v1Ten patch-backed cases: security, distributed systems, RAG, concurrency.→
  • Audit pack v2A second batch of certified cases targeting logic defects.→
  • Adding casesExtend a pack with a reproducible seeded defect.→

Reviewers

  • Adding reviewersConnect a reviewer and keep its output parseable.→
  • Custom command reviewerRun an external agent as a subprocess that prints JSON.→

Reports and CI

  • Audit reportTurn saved runs into Markdown and dashboard JSON.→
  • GitHub ActionRun Arena in CI and keep the runs as an artifact.→
Start hereGetting started
Code ReviewRepair passed · Repair failed · Not measured
ARENA

An open-source local benchmark. It applies an AI code reviewer’s patch to a pull request with a seeded bug, runs the tests and reports whether the repair passed.

Checking Arena API

Evaluate

  • Repair gate
  • Experiments
  • Compare
  • Verify

Results

  • Leaderboard
  • Runs
  • Reports
  • Cases

Docs

  • Methodology
  • Docs
  • Getting started
  • Metrics
  • Adding reviewers

Project

  • GitHub
  • Changelog
  • Security
  • Contributing
  • MIT license

Code Review Arena. Open source under the MIT license.

Controls are deterministic scripts built into the benchmark, not models or products.

Back to top ↑