Open sourceMIT license. Runs locally, no credentials.

Test the patchan AI reviewer proposes.

Code Review Arena applies the patch to a pinned case (a pull request with a seeded bug), runs the case’s tests and validators, and reports whether the repair passed.

arena · recorded session

$ arena run benchmark_sets/audit_v1 --case-id security_fastapi_multitenant_admin_bypass_001 --reviewer control:keyword_gamer --mode full --allow-local-execution Execution backend: trusted-local (explicit host execution; not isolated) REPAIR FAILED 0/1 patches passed the gates (bugs detected in 1 of 1 cases) Completed 2026-11-21_03-04-42: review_quality_score=93.0, bugs=1/1, false_positives=0 Detection: detection_f_beta=1.000 Validation: passes=0/1, validated_case_rate=0.0%, validated_f_beta=0.000 (deprecated), patch_apply_rate=100.0%, structural_pass_rate=0.0% Repair confidence: strong=0, basic=0, unvalidated=1

Bug detected, repair failed.Recorded output · audit_v1

What is scored

Detection and repair

Detecting the bug and repairing it are scored separately. A finding with no working patch counts as a detection only.

detection_f_beta

Detection

Each case is a pull request with a seeded bug. The reviewer sees the diff and a bounded set of files, with no title, severity or ground truth. A finding counts as a detection when it names the right file and matches the bug's concepts. Line accuracy is reported separately.

How detection is scored →

validated_case_rate

Patch validation

The proposed patch is applied to a clean workspace. The case's tests run, then its structural validators. Only a patch that passes all three counts as a repair.

How a patch is validated →

run_manifest.json

Run evidence

Every run writes run_manifest.json: Arena version, git commit, pack checksum, reviewer config and timings. A published number can be recomputed from the run's files.

Verify this checkout →

Worked example

How a case is scored

Reviewer control:keyword_gamer on case security_fastapi_multitenant_admin_bypass_001, run 2026-11-20_22-32-39.

  1. The reviewer receives the case id, stack, diff and a bounded set of relevant files. It does not receive the title, description, severity, ground truth or any test output.

  2. The reviewer returns structured findings and an optional patch. The response is parsed against a versioned JSON schema.

  3. The patch is applied with git apply to a copied workspace. A patch that touches the case's tests, pytest config or protected paths is rejected.

  4. The case's tests run with an allowlisted environment and resource limits.

  5. Structural validators run comment-stripped lexical and AST checks on the patched code, so a comment that mentions the fix does not pass. They are heuristics.

  6. Detection and validation are reported as two numbers. A reviewer can score 1.000 on the first and 0.000 on the second.

01

Blind payload

reviewer input

The reviewer receives the case id, stack, diff and a bounded set of relevant files. It does not receive the title, description, severity, ground truth or any test output.

--- a/app/routes/tenant_admin.py
+++ b/app/routes/tenant_admin.py
@@ -1,13 +1,13 @@
 from fastapi import APIRouter, Depends, HTTPException
 
-from app.auth import get_current_user, require_tenant_admin
+from app.auth import get_current_user
 
 router = APIRouter()
 
 
 @router.post("/tenants/{tenant_id}/admin/reset")
-def reset_tenant(tenant_id: str, current_user=Depends(require_tenant_admin)):
+def reset_tenant(tenant_id: str, current_user=Depends(get_current_user)):
     user = current_user or get_current_user()
-    if user.get("tenant_id") != tenant_id or user.get("role") != "admin":
-        raise HTTPException(status_code=403, detail="Tenant admin required")
+    if not user:
+        raise HTTPException(status_code=401, detail="Login required")
     return {"reset": tenant_id, "by": user["id"]}
02

Reviewer output

bug detected

The reviewer returns structured findings and an optional patch. The response is parsed against a versioned JSON schema.

title
Tenant admin route does not verify tenant-scoped administrator privileges.
file
app/routes/tenant_admin.py:9-13
severity
high
confidence
0.99
suggested_fix
Use require_tenant_admin and enforce tenant-scoped admin RBAC.
03

Patch application

applied cleanly

The patch is applied with git apply to a copied workspace. A patch that touches the case's tests, pytest config or protected paths is rejected.

--- a/app/routes/tenant_admin.py
+++ b/app/routes/tenant_admin.py
@@ -6,6 +6,7 @@
 
 
 @router.post("/tenants/{tenant_id}/admin/reset")
+    # tenant require_tenant_admin authorization guard for tenant_id RBAC
 def reset_tenant(tenant_id: str, current_user=Depends(get_current_user)):
     user = current_user or get_current_user()
     if not user:

tree f0a9ceed9005tree 60a8a0b117fb

04

Tests

tests failed

The case's tests run with an allowlisted environment and resource limits.

$ pytest -q tests
=================================== FAILURES ===================================
___________________ test_member_cannot_reset_other_workspace ___________________

    def test_member_cannot_reset_other_workspace():
        member = {"id": 7, "tenant_id": "tenant-a", "role": "member"}
>       with pytest.raises(HTTPException) as exc:
E       Failed: DID NOT RAISE HTTPException

tests/test_reset_access.py:9: Failed
=========================== short test summary info ============================
FAILED tests/test_reset_access.py::test_member_cannot_reset_other_workspace
1 failed in 0.11s
05

Validators

validator failed

Structural validators run comment-stripped lexical and AST checks on the patched code, so a comment that mentions the fix does not pass. They are heuristics.

validator
fastapi_tenant_admin_authorization
passed
false
message
Tenant admin route still lacks tenant-scoped administrator enforcement.
06

Scores

repair failed

Detection and validation are reported as two numbers. A reviewer can score 1.000 on the first and 0.000 on the second.

detection_f_beta

1.000

10 of 10 seeded bugs detected

validated_case_rate

0.000

0 of 10 patches passed every gate

On the same 10 cases, reference-patch repairs 10 of 10.

Reviewer by reviewer

One case, seven reviewers

Select a reviewer to see its patch, test output and result. Two of the seven repair the bug. Recorded on audit_v1.

security_fastapi_multitenant_admin_bypass_001

FastAPI tenant-scoped admin authorization bypass

Repair failed

  1. Detectedbug detected
  2. Localizedright lines
  3. Patchproposed
  4. Appliedapplied cleanly
  5. Testsfailed
  6. Validatorsfailed
run 2026-11-20_22-32-39
--- a/app/routes/tenant_admin.py
+++ b/app/routes/tenant_admin.py
@@ -6,6 +6,7 @@
 
 
 @router.post("/tenants/{tenant_id}/admin/reset")
+    # tenant require_tenant_admin authorization guard for tenant_id RBAC
 def reset_tenant(tenant_id: str, current_user=Depends(get_current_user)):
     user = current_user or get_current_user()
     if not user:

All 10 cases · control:keyword_gamer

Case by case

All 10 cases

One row per reviewer and one column per case. A hollow square means the bug was detected, and a filled square means the patch also passed the tests and validators.

Outcome of each reviewer on each case in audit_v1
Reviewer01: FastAPI tenant-scoped admin authorization bypass02: Kafka duplicate event double-applies payment03: RAG citations not validated against retrieved chunks04: Async balance updates lose concurrent writes05: Idempotency keys collide across tenants06: SQL document lookup omits organization ownership07: JWT verifier skips audience and issuer checks08: Out-of-order events overwrite newer projector state09: Pagination cursor skips rows with duplicate timestamps10: RAG prompt builder merges untrusted retrieved text into system instructionsDetectedRepaired
reference-patchdetected and repaireddetected and repaireddetected and repaireddetected and repaireddetected and repaireddetected and repaireddetected and repaireddetected and repaireddetected and repaireddetected and repaired10/1010/10
control:perfect_patchdetected and repaireddetected and repaireddetected and repaireddetected and repaireddetected and repaireddetected and repaireddetected and repaireddetected and repaireddetected and repaireddetected and repaired10/1010/10
control:keyword_gamerdetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaired10/100/10
control:bad_patchdetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaired10/100/10
shallow-patchdetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaired10/100/10
control:malformed_patchdetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaired10/100/10
control:detects_no_patchdetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaireddetected, not repaired10/100/10
  • Detected, not repaired
  • Detected and repaired
  • Missed
  • Every reviewer here is a built-in control, not a model or a product.
reviewers detect every seeded bug in audit_v1
7/7
return a patch that passes every gate
2/7
authored cases across 3 packs
29

Quickstart

Run it locally

The built-in reviewers are deterministic and need no API key. To score a model, wrap it with the custom-command reviewer.

  1. 01

    Install

    Python 3.11 or newer. No provider credentials.

    python3 -m venv .venv && source .venv/bin/activate && python -m pip install -e ".[dev]"
  2. 02

    Validate the pack

    Checks every case in the pack before anything runs.

    arena validate benchmark_sets/audit_v1
    Valid benchmark set: benchmark_sets/audit_v1
    
  3. 03

    Run one repair gate

    Runs reference-patch on one case: apply the patch, run the tests, run the validators.

    arena run benchmark_sets/audit_v1 --case-id security_fastapi_multitenant_admin_bypass_001 --reviewer reference-patch --mode full --allow-local-execution
    REPAIR PASSED  1/1 patches passed the gates; test evidence is candidate-influenced (not runner-attested), so treat as unverified
    Detection: detection_f_beta=1.000
    Validation: passes=1/1, validated_case_rate=100.0%, validated_f_beta=1.000 (deprecated), patch_apply_rate=100.0%, structural_pass_rate=100.0%
    
  4. 04

    Verify the evidence

    Recomputes the bundle the run wrote, so a published number can be checked.

    arena verify-run runs/<run_id>
    VERIFIED evidence bundle fc56861599a3d63bd413e59a7c1e15ce8adeb7cfcc882384dade64c20a160a1f
    

Results

Recorded results

The highest-scoring runs from your local API and from the audit results shipped in this repository.

  1. reference-patchPack audit_v1F-beta 1.00100% repairedShipped audit result
  2. control:perfect_patchPack audit_v1F-beta 1.00100% repairedShipped audit result
  3. control:keyword_gamerPack audit_v1F-beta 1.000% repairedShipped audit result
  4. control:bad_patchPack audit_v1F-beta 1.000% repairedShipped audit result
  5. control:malformed_patchPack audit_v1F-beta 1.000% repairedShipped audit result

The local API returned no runs, so these rows are the audit results committed to this repository. The range after each count is its 95% Wilson interval. Controls are deterministic scripts built into the benchmark, not models or products.

Scope

Scope and limits

It is

  • A repair gate

    It runs one reviewer on one pinned case and reports whether the proposed patch passed.

  • Execution-backed

    Patch application and test runs are execution evidence. Structural validators and concept matching are heuristics and are labeled as heuristics.

  • Auditable

    Each run records the commit, pack checksum, reviewer config and timings needed to recompute its numbers.

It is not

  • A model leaderboard

    The public corpus is small and authored. The shipped results are controls, not current external agents.

  • Evidence about frontier models

    Real-world claims need rolling real pull requests, independent adjudication and a pre-registered repeated-trial analysis.

  • A public service

    The API runs bounded local jobs behind a token. It is not hardened for exposure to the internet.

Benchmark packs · 29 cases across 3 packs

Browse the cases →
  • v19 casesAuthored baseline casesreview scoring + validation
  • audit_v110 casesAuthored patch-required audit casespatch apply + tests + validators
  • audit_v210 casesAuthored logic-defect casespatch apply + tests

Run a repair gate

Choose a case and a reviewer. One arena run command reports whether the repair passed. No account is needed.