Audit Pack v2 is a 10-case pack that targets small but high-impact logic defects: the kind that pass a casual read yet change behavior in production. Every case is authored leak-free (no ground-truth vocabulary in the diff, comments, or test names) and fully certified (the buggy baseline fails, the reference fix passes, and the tests kill every viable mutant).

Cases

  • money_discount_rounding_001: per-unit reduction loses money on multi-unit orders
  • ratelimit_window_boundary_001: fixed-window limiter admits one past the cap
  • permission_precedence_001: boolean precedence bypasses a guard
  • overdraft_min_balance_001: lowest-balance tracker compares the wrong way
  • progress_zero_division_001: completion percentage divides by zero
  • page_count_ceil_001: page count floors and drops the last page
  • truthiness_default_001: fallback discards an explicit zero
  • preview_truncation_001: preview truncates one character short
  • retry_backoff_cap_001: backoff delay grows without bound
  • eligibility_and_or_001: check uses or where both conditions are required

Adversarial baseline

shallow-patch is a generic adversarial reviewer that works on any pack: it localizes the bug from the reference patch, then proposes a no-op change that applies cleanly but repairs nothing. It scores detection near 1.0 with validated_case_rate 0.0, the detection-versus-validation gap the harness exists to measure.

Commands

arena validate benchmark_sets/audit_v2
arena lint-cases benchmark_sets/audit_v2 --strict
arena certify-pack benchmark_sets/audit_v2 --allow-local-execution --strict certified
arena run benchmark_sets/audit_v2 --reviewer reference-patch --mode full --allow-local-execution
arena run benchmark_sets/audit_v2 --reviewer shallow-patch --mode full --allow-local-execution

Results

The rendered report for this pack lives at /reports/audit-v2: the verified reference patch against the generic adversarial baseline.

Primary metric

Use validated_case_rate in full mode. Detection metrics alone do not prove a working repair.