Validate a benchmark pack

Code
arena validate benchmark_sets/v1
arena validate benchmark_sets/audit_v1
arena validate benchmark_sets/audit_v2

Run deterministic controls

Code
arena run benchmark_sets/audit_v1 --reviewer reference-patch --mode full --allow-local-execution
arena run benchmark_sets/audit_v1 --reviewer control:keyword_gamer --mode full --allow-local-execution
arena run benchmark_sets/audit_v2 --reviewer shallow-patch --mode full --allow-local-execution

Rank runs

Code
arena leaderboard runs/ --metric validated_case_rate --beta 1.0 --include-unverified

Generate the audit report

Code
arena audit-report runs/ --output docs/reports/audit-v1-results.md

Serve the API

Code
arena serve