Runs / 2026-11-21_02-32-14 / Case trace
Did not complete
RAG unsupported citation emission
Ground truthGenerated citation IDs are not validated against retrieved context.
Six stages in order. Tests and validators both run after the patch applies. Outcome: Did not complete.
not reached
not recorded
The case stopped on an execution error
case_execution_error
RuntimeError: wrapper died
0 findings0 of 1 seeded bug matched0 false positives
No findings returned. The reviewer output was recorded as invalid, so there was nothing to score.
The case stopped on an execution error, so no stage recorded a result.
The case stopped on an execution error, so no stage recorded a result.
No structural validators ran. The case stopped on an execution error, so no stage recorded a result.
rag_fake_citation_001python, rag, llm
Seeded change11 lines
--- a/rag/answer.py +++ b/rag/answer.py @@ -1,6 +1,2 @@ def build_answer(generator, retrieved_chunks): - answer = generator(retrieved_chunks) - valid_ids = {chunk["id"] for chunk in retrieved_chunks} - if any(citation not in valid_ids for citation in answer["citation_ids"]): - raise ValueError("Answer contained an unsupported citation") - return answer + return generator(retrieved_chunks)
0.0/100
Scoring JSON17 lines
{
"scored_findings": [],
"breakdown": {
"concept_match": 0,
"file_match": 0,
"line_overlap": 0,
"severity_match": 0,
"fix_quality": 0,
"no_false_positives": 0,
"false_positive_penalty": 0,
"invalid_json_penalty": 0,
"total": 0
},
"failure_reasons": [
"case_execution_error: RuntimeError: wrapper died"
]
}parse status: invalid
The reviewer returned no output for this case.