Code Review Arena applies the patch to a pinned case (a pull request with a seeded bug), runs the case’s tests and validators, and reports whether the repair passed.
Detecting the bug and repairing it are scored separately. A finding with no working patch counts as a detection only.
1from fastapi import APIRouter, Depends, HTTPException2from app.auth import get_current_user, require_tenant_admin3from app.auth import get_current_user45router = APIRouter()678@router.post("/tenants/{tenant_id}/admin/reset")def reset_tenant(tenant_id: str, current_user=Depends(require_tenant_admin)):9def reset_tenant(tenant_id: str, current_user=Depends(get_current_user)):10 user = current_user or get_current_user() if user.get("tenant_id") != tenant_id or user.get("role") != "admin":
app/routes/tenant_admin.py:9-13
detection_f_beta
Detection
Each case is a pull request with a seeded bug. The reviewer sees the diff and a bounded set of files, with no title, severity or ground truth. A finding counts as a detection when it names the right file and matches the bug's concepts. Line accuracy is reported separately.
The proposed patch is applied to a clean workspace. The case's tests run, then its structural validators. Only a patch that passes all three counts as a repair.
Every run writes run_manifest.json: Arena version, git commit, pack checksum, reviewer config and timings. A published number can be recomputed from the run's files.
Reviewer control:keyword_gamer on case security_fastapi_multitenant_admin_bypass_001, run 2026-11-20_22-32-39.
The reviewer receives the case id, stack, diff and a bounded set of relevant files. It does not receive the title, description, severity, ground truth or any test output.
The reviewer returns structured findings and an optional patch. The response is parsed against a versioned JSON schema.
The patch is applied with git apply to a copied workspace. A patch that touches the case's tests, pytest config or protected paths is rejected.
The case's tests run with an allowlisted environment and resource limits.
Structural validators run comment-stripped lexical and AST checks on the patched code, so a comment that mentions the fix does not pass. They are heuristics.
Detection and validation are reported as two numbers. A reviewer can score 1.000 on the first and 0.000 on the second.
01
Blind payload
reviewer input
The reviewer receives the case id, stack, diff and a bounded set of relevant files. It does not receive the title, description, severity, ground truth or any test output.
--- a/app/routes/tenant_admin.py
+++ b/app/routes/tenant_admin.py
@@ -1,13 +1,13 @@
from fastapi import APIRouter, Depends, HTTPException
-from app.auth import get_current_user, require_tenant_admin
+from app.auth import get_current_user
router = APIRouter()
@router.post("/tenants/{tenant_id}/admin/reset")
-def reset_tenant(tenant_id: str, current_user=Depends(require_tenant_admin)):
+def reset_tenant(tenant_id: str, current_user=Depends(get_current_user)):
user = current_user or get_current_user()
- if user.get("tenant_id") != tenant_id or user.get("role") != "admin":
- raise HTTPException(status_code=403, detail="Tenant admin required")
+ if not user:
+ raise HTTPException(status_code=401, detail="Login required")
return {"reset": tenant_id, "by": user["id"]}
02
Reviewer output
bug detected
The reviewer returns structured findings and an optional patch. The response is parsed against a versioned JSON schema.
title
Tenant admin route does not verify tenant-scoped administrator privileges.
file
app/routes/tenant_admin.py:9-13
severity
high
confidence
0.99
suggested_fix
Use require_tenant_admin and enforce tenant-scoped admin RBAC.
03
Patch application
applied cleanly
The patch is applied with git apply to a copied workspace. A patch that touches the case's tests, pytest config or protected paths is rejected.
--- a/app/routes/tenant_admin.py
+++ b/app/routes/tenant_admin.py
@@ -6,6 +6,7 @@
@router.post("/tenants/{tenant_id}/admin/reset")
+ # tenant require_tenant_admin authorization guard for tenant_id RBAC
def reset_tenant(tenant_id: str, current_user=Depends(get_current_user)):
user = current_user or get_current_user()
if not user:
tree f0a9ceed9005→tree 60a8a0b117fb
04
Tests
tests failed
The case's tests run with an allowlisted environment and resource limits.
$ pytest -q tests
=================================== FAILURES ===================================
___________________ test_member_cannot_reset_other_workspace ___________________
def test_member_cannot_reset_other_workspace():
member = {"id": 7, "tenant_id": "tenant-a", "role": "member"}
> with pytest.raises(HTTPException) as exc:
E Failed: DID NOT RAISE HTTPException
tests/test_reset_access.py:9: Failed
=========================== short test summary info ============================
FAILED tests/test_reset_access.py::test_member_cannot_reset_other_workspace
1 failed in 0.11s
05
Validators
validator failed
Structural validators run comment-stripped lexical and AST checks on the patched code, so a comment that mentions the fix does not pass. They are heuristics.
validator
fastapi_tenant_admin_authorization
passed
false
message
Tenant admin route still lacks tenant-scoped administrator enforcement.
06
Scores
repair failed
Detection and validation are reported as two numbers. A reviewer can score 1.000 on the first and 0.000 on the second.
detection_f_beta
1.000
10 of 10 seeded bugs detected
validated_case_rate
0.000
0 of 10 patches passed every gate
On the same 10 cases, reference-patch repairs 10 of 10.
03Reviewer by reviewer
One case, seven reviewers
Select a reviewer to see its patch, test output and result. Two of the seven repair the bug. Recorded on audit_v1.
security_fastapi_multitenant_admin_bypass_001
FastAPI tenant-scoped admin authorization bypass
Repair failed
Detectedbug detected
Localizedright lines
Patchproposed
Appliedapplied cleanly
Testsfailed
Validatorsfailed
run 2026-11-20_22-32-39
--- a/app/routes/tenant_admin.py
+++ b/app/routes/tenant_admin.py
@@ -6,6 +6,7 @@
@router.post("/tenants/{tenant_id}/admin/reset")
+ # tenant require_tenant_admin authorization guard for tenant_id RBAC
def reset_tenant(tenant_id: str, current_user=Depends(get_current_user)):
user = current_user or get_current_user()
if not user:
04Case by case
All 10 cases
One row per reviewer and one column per case. A hollow square means the bug was detected, and a filled square means the patch also passed the tests and validators.
The highest-scoring runs from your local API and from the audit results shipped in this repository.
ReviewerPackDetection F-betaRepairedSource
reference-patchPack audit_v1F-beta 1.00100% repairedShipped audit result
control:perfect_patchPack audit_v1F-beta 1.00100% repairedShipped audit result
control:keyword_gamerPack audit_v1F-beta 1.000% repairedShipped audit result
control:bad_patchPack audit_v1F-beta 1.000% repairedShipped audit result
control:malformed_patchPack audit_v1F-beta 1.000% repairedShipped audit result
The local API returned no runs, so these rows are the audit results committed to this repository. The range after each count is its 95% Wilson interval. Controls are deterministic scripts built into the benchmark, not models or products.