Audit Pack v2 is a 10-case pack that targets small but high-impact logic defects: the kind that pass a casual read yet change behavior in production. Every case is authored leak-free (no ground-truth vocabulary in the diff, comments, or test names) and fully certified (the buggy baseline fails, the reference fix passes, and the tests kill every viable mutant).
Cases
money_discount_rounding_001: per-unit reduction loses money on multi-unit ordersratelimit_window_boundary_001: fixed-window limiter admits one past the cappermission_precedence_001: boolean precedence bypasses a guardoverdraft_min_balance_001: lowest-balance tracker compares the wrong wayprogress_zero_division_001: completion percentage divides by zeropage_count_ceil_001: page count floors and drops the last pagetruthiness_default_001: fallback discards an explicit zeropreview_truncation_001: preview truncates one character shortretry_backoff_cap_001: backoff delay grows without boundeligibility_and_or_001: check uses or where both conditions are required
Adversarial baseline
shallow-patch is a generic adversarial reviewer that works on any pack: it localizes the bug from the reference patch, then proposes a no-op change that applies cleanly but repairs nothing. It scores detection near 1.0 with validated_case_rate 0.0, the detection-versus-validation gap the harness exists to measure.
Commands
arena validate benchmark_sets/audit_v2 arena lint-cases benchmark_sets/audit_v2 --strict arena certify-pack benchmark_sets/audit_v2 --allow-local-execution --strict certified arena run benchmark_sets/audit_v2 --reviewer reference-patch --mode full --allow-local-execution arena run benchmark_sets/audit_v2 --reviewer shallow-patch --mode full --allow-local-execution
Results
The rendered report for this pack lives at /reports/audit-v2: the verified reference patch against the generic adversarial baseline.
Primary metric
Use validated_case_rate in full mode. Detection metrics alone do not prove a working repair.