- Create minimal working
before/code and buggyafter/code. - Generate a valid unified
pr.diffthat exposes the introduced defect. - Author
case.yamlground truth, localization, scoring, validation, execution, and beta settings. - Add regression tests when execution can demonstrate the repair.
- Select tolerant structural validators or implement a validator with unit tests.
- Run
arena validate benchmark_sets/v1and a deterministic mock-patch full run.
benchmark_sets/v1/new_case/ case.yaml before/ after/ pr.diff tests/