Audit Pack v1 is a curated 10-case pack for reviewers that must localize a defect, supply a patch, pass regression tests, and satisfy structural validators. Cases span security, distributed systems, RAG safety, concurrency, idempotency, API correctness, and pagination.

Cases

  • security_fastapi_multitenant_admin_bypass_001: tenant admin authorization
  • distributed_kafka_duplicate_event_001: Kafka duplicate delivery
  • rag_fabricated_citation_001: RAG citation grounding
  • async_balance_race_001: async lost updates
  • idempotency_key_tenant_scope_001: tenant-scoped idempotency
  • security_sql_join_ownership_leak_001: SQL ownership leak
  • security_jwt_audience_validation_001: JWT audience and issuer checks
  • distributed_out_of_order_event_001: out-of-order events
  • api_pagination_cursor_skip_001: pagination cursor tiebreaker
  • rag_prompt_injection_policy_override_001: RAG prompt injection

Reference patches

Each case ships a reference.patch. Run arena run benchmark_sets/audit_v1 --reviewer reference-patch --mode full --allow-local-execution to validate the static artifacts.

Commands

arena validate benchmark_sets/audit_v1
arena run benchmark_sets/audit_v1 --reviewer reference-patch --mode full --allow-local-execution
arena run benchmark_sets/audit_v1 --reviewer control:perfect_patch --mode full --allow-local-execution
arena leaderboard runs/ --metric validated_case_rate --beta 1.0 --include-unverified
arena audit-report runs/ --output docs/reports/audit-v1-results.md

Primary metric

Use validated_case_rate in full mode. Detection metrics alone do not prove a working repair.