Checks

9 checks · 2 passing · 7 not recorded in this snapshot

  • passing
  • failing
  • not recorded

The generator records each check it runs.

python scripts/generate_verification_snapshot.py --run-validation --run-quality-checks --generate-report

Quality checks0 of 4 passing

  • not recorded: Backend testsBackend test status is only recorded when executed by the generator.make test
  • not recorded: LintLint status is only recorded when executed by the generator.make lint
  • not recorded: TypecheckTypecheck status is only recorded when executed by the generator.make typecheck
  • not recorded: Dashboard buildDashboard build status is only recorded when executed by the generator.npm run build

Pack validation0 of 1 passing

  • not recorded: audit_v1 validationValidation has not been run in this snapshot.arena validate benchmark_sets/audit_v1

Control baselines2 of 2 passing

  • passing: reference-patch baseline10 of 10 cases repaired. Validated case rate 1.000.arena run benchmark_sets/audit_v1 --reviewer reference-patch --mode full --allow-local-execution
  • passing: control:keyword_gamer baseline (adversarial)0 of 10 cases repaired. Validated case rate 0.000.arena run benchmark_sets/audit_v1 --reviewer control:keyword_gamer --mode full --allow-local-execution

Capabilities0 of 2 passing

  • not recorded: Audit report generationGenerate an audit report to capture this status.arena audit-report runs/ --output docs/reports/audit-v1-results.md
  • not recorded: Custom-command reviewerRun the adapter tests to capture this status.pytest tests/test_custom_command_reviewer.py

Controls

Deterministic controls from saved local audit_v1 runs. They are not model results.

  • reference-patchKnown-good patch artifacts10/10 repaireddetection F-beta 1.000 · validated rate 1.000
  • control:perfect_patchHarness happy path10/10 repaireddetection F-beta 1.000 · validated rate 1.000
  • control:keyword_gamerPlausible comments fail validation0/10 repaireddetection F-beta 1.000 · validated rate 0.000
  • control:bad_patchDetects but supplies bad fixes0/10 repaireddetection F-beta 1.000 · validated rate 0.000
  • control:malformed_patchDetects but supplies an invalid patch0/10 repaireddetection F-beta 1.000 · validated rate 0.000

Reproduce locally

From the repository root
python3 -m venv .venvsource .venv/bin/activatepython -m pip install -e ".[dev]"arena validate benchmark_sets/audit_v1arena run benchmark_sets/audit_v1 --reviewer reference-patch --mode full --allow-local-executionarena run benchmark_sets/audit_v1 --reviewer control:keyword_gamer --mode full --allow-local-executionarena leaderboard runs/ --metric validated_case_rate --beta 1.0 --include-unverified

Troubleshooting

Installation, stale dashboard data, missing reports and empty-run guidance are documented step by step.

Open troubleshooting