Audit report
A local execution-backed audit of code-review agents on 10 patch-required seeded bugs.
validated_case_rateGenerated 2026-06-18 15:38
reference-patch
trust not recorded
Detected 100%, Applied 100%, Tests 100%, Validators 100%, Repaired 100%.
100% repaired
control:perfect_patch
trust not recorded
Detected 100%, Applied 100%, Tests 100%, Validators 100%, Repaired 100%.
100% repaired
control:keyword_gamer
trust not recorded
Detected 100%, Applied 100%, Tests 0%, Validators 0%, Repaired 0%.
0% repaired
Stops at: tests failed
control:bad_patch
trust not recorded
Detected 100%, Applied 100%, Tests 0%, Validators 0%, Repaired 0%.
0% repaired
Stops at: tests failed
control:malformed_patch
trust not recorded
Detected 100%, Applied 0%, Tests –, Validators –, Repaired 0%.
0% repaired
Stops at: patch apply failed
| Reviewer | Model | Trust | Pack snapshot | Run | Detection F-beta | Validated rate | Deterministic pass rate | Patch apply rate | Test pass rate | Structural pass rate | False positives / case |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Reference Patch | - | Trust not recorded | not recorded | 2026-06-18_10-24-25 | 1.000 | 1.000 | 100.0% | 100.0% | 100.0% | 100.0% | 0.000 |
| Control: Perfect Repair | perfect_patch | Trust not recorded | not recorded | 2026-06-18_10-24-28 | 1.000 | 1.000 | 100.0% | 100.0% | 100.0% | 100.0% | 0.000 |
| Control: Keyword Gamer | keyword_gamer | Trust not recorded | not recorded | 2026-06-18_10-24-31 | 1.000 | 0.000 | 0.0% | 100.0% | 0.0% | 0.0% | 0.000 |
| Control: Weak Repair | bad_patch | Trust not recorded | not recorded | 2026-06-18_10-24-34 | 1.000 | 0.000 | 0.0% | 100.0% | 0.0% | 0.0% | 0.000 |
| Control: Malformed Patch | malformed_patch | Trust not recorded | not recorded | 2026-06-18_10-24-38 | 1.000 | 0.000 | 0.0% | 0.0% | - | - | 0.000 |
The report file holds rates only, so each stage shows its rate in tenths on a 0% to 100% axis. Repaired is the share of execution-backed cases (those with runnable tests). Detected is detection recall. Controls are deterministic scripts built into the benchmark, not models or products.
arena validate benchmark_sets/audit_v1 arena run benchmark_sets/audit_v1 --reviewer reference-patch --mode full --allow-local-execution arena run benchmark_sets/audit_v1 --reviewer control:keyword_gamer --mode full --allow-local-execution arena leaderboard runs/ --metric validated_case_rate --beta 1.0 arena audit-report runs/ --output docs/reports/audit-v1-results.md