Evaluate
Verify this checkout
A generated snapshot of reproducible checks and deterministic control outcomes. A check is not recorded when its command was not run for the snapshot.
Generated 2026-06-18 15:38Checks
9 checks · 2 passing · 7 not recorded in this snapshot
- passing
- failing
- not recorded
The generator records each check it runs.
python scripts/ generate_verification_snapshot.py --run-validation --run-quality-checks --generate-reportQuality checks0 of 4 passing
- not recorded: Backend testsBackend test status is only recorded when executed by the generator.
make test - not recorded: LintLint status is only recorded when executed by the generator.
make lint - not recorded: TypecheckTypecheck status is only recorded when executed by the generator.
make typecheck - not recorded: Dashboard buildDashboard build status is only recorded when executed by the generator.
npm run build
Pack validation0 of 1 passing
- not recorded: audit_v1 validationValidation has not been run in this snapshot.
arena validate benchmark_sets/audit_v1
Control baselines2 of 2 passing
- passing: reference-patch baseline10 of 10 cases repaired. Validated case rate 1.000.
arena run benchmark_sets/audit_v1 --reviewer reference-patch --mode full --allow-local-execution - passing: control:keyword_gamer baseline (adversarial)0 of 10 cases repaired. Validated case rate 0.000.
arena run benchmark_sets/audit_v1 --reviewer control:keyword_gamer --mode full --allow-local-execution
Capabilities0 of 2 passing
- not recorded: Audit report generationGenerate an audit report to capture this status.
arena audit-report runs/ --output docs/reports/ audit-v1-results.md - not recorded: Custom-command reviewerRun the adapter tests to capture this status.
pytest tests/test_custom_command_reviewer.py
Controls
Deterministic controls from saved local audit_v1 runs. They are not model results.
- reference-patchKnown-good patch artifacts10/10 repaireddetection F-beta 1.000 · validated rate 1.000
- control:perfect_patchHarness happy path10/10 repaireddetection F-beta 1.000 · validated rate 1.000
- control:keyword_gamerPlausible comments fail validation0/10 repaireddetection F-beta 1.000 · validated rate 0.000
- control:bad_patchDetects but supplies bad fixes0/10 repaireddetection F-beta 1.000 · validated rate 0.000
- control:malformed_patchDetects but supplies an invalid patch0/10 repaireddetection F-beta 1.000 · validated rate 0.000
Reproduce locally
python3 -m venv .venvsource .venv/bin/activatepython -m pip install -e ".[dev]"arena validate benchmark_sets/audit_v1arena run benchmark_sets/audit_v1 --reviewer reference-patch --mode full --allow-local-executionarena run benchmark_sets/audit_v1 --reviewer control:keyword_gamer --mode full --allow-local-executionarena leaderboard runs/ --metric validated_case_rate --beta 1.0 --include-unverified
Troubleshooting
Installation, stale dashboard data, missing reports and empty-run guidance are documented step by step.