Audit Pack v1 is a curated 10-case pack for reviewers that must localize a defect, supply a patch, pass regression tests, and satisfy structural validators. Cases span security, distributed systems, RAG safety, concurrency, idempotency, API correctness, and pagination.
Cases
security_fastapi_multitenant_admin_bypass_001: tenant admin authorizationdistributed_kafka_duplicate_event_001: Kafka duplicate deliveryrag_fabricated_citation_001: RAG citation groundingasync_balance_race_001: async lost updatesidempotency_key_tenant_scope_001: tenant-scoped idempotencysecurity_sql_join_ownership_leak_001: SQL ownership leaksecurity_jwt_audience_validation_001: JWT audience and issuer checksdistributed_out_of_order_event_001: out-of-order eventsapi_pagination_cursor_skip_001: pagination cursor tiebreakerrag_prompt_injection_policy_override_001: RAG prompt injection
Reference patches
Each case ships a reference.patch. Run arena run benchmark_sets/audit_v1 --reviewer reference-patch --mode full --allow-local-execution to validate the static artifacts.
Commands
arena validate benchmark_sets/audit_v1 arena run benchmark_sets/audit_v1 --reviewer reference-patch --mode full --allow-local-execution arena run benchmark_sets/audit_v1 --reviewer control:perfect_patch --mode full --allow-local-execution arena leaderboard runs/ --metric validated_case_rate --beta 1.0 --include-unverified arena audit-report runs/ --output docs/reports/audit-v1-results.md
Primary metric
Use validated_case_rate in full mode. Detection metrics alone do not prove a working repair.