Citations Were Not Enough for Safety-Critical Requirements RAG

The answerer was safe on perfect evidence.

Then I removed the supporting evidence, and it still answered.

That was the result that changed the direction of this RAG experiment.

The previous lab was about retrieval over structured software requirements. I used 13,244 sanitized automotive requirement records and found that flattening them into chunks destroyed field-to-record meaning. The better architecture preserved records, used exact execution for IDs, filters, counts, and trace links, and sent only semantic questions through retrieval.

That solved one problem.

It did not solve the more dangerous one:

Once the system has retrieved some records, can the LLM know when not to answer?

For ordinary document search, a weak answer may be merely annoying. For requirements engineering, it is different. A requirement is not just text. It is a structured engineering record used to make decisions about behavior, traceability, safety constraints, variants, status, and implementation scope.

In that setting, a confident unsupported answer is worse than an explicit “I do not know from the available evidence.”

So I tested the answer-generation part separately.

The result was negative. Not ambiguous. Not “needs a better threshold.” The experiment hit a preregistered STOP condition.

And that is exactly why the experiment was worth running.

The Question

The question was not:

Can an LLM write a plausible answer from retrieved requirements?

That is too low a bar.

The question was:

Can a requirements RAG produce a useful free-text answer only when every material claim is supported by cited requirement evidence, and abstain when the corpus or retrieved context is insufficient?

This distinction matters.

The system was allowed to answer only if the answer could be grounded in supplied requirement records. It had to cite exact evidence. If the context was insufficient, contradictory, out of scope, or supported only by inadmissible records, the correct behavior was abstention.

The experiment focused on the semantic path left over after routing:

1. retrieve requirement records;
2. freeze the context;
3. generate a cited answer;
4. validate the citations and evidence policy;
5. decide whether to publish the answer or abstain.

Structured questions were not part of this evaluation. If the user asks for an exact ID, a count, a field filter, or a trace relationship, that should go through deterministic execution. This experiment was about the remaining free-text semantic questions.

Why Requirements Make This Harder

Requirements are not prose articles.

They have fields: ID, module, status, variant, version, ASIL, description, rationale, and trace relationships. Some records are approved. Some are rejected. Some are under work. Some apply only in a specific scope.

Retrieving a record does not automatically make it valid evidence.

For the current normative-answer use case, the evidence policy was intentionally strict: approved records could support current behavior; rejected or otherwise inadmissible records could not support a normative answer. A rejected record could still be valid evidence for a historical-status question, but not for “what should the system currently do?”

This creates failure modes that ordinary RAG demos often hide:

• The right text appears only in a rejected record.
• An approved record is retrieved beside a contradictory rejected record.
• Two equally admissible records conflict.
• A question needs multiple claims from multiple records.
• The retriever finds related material but misses a critical requirement.
• The answerer cites a real quote but draws the wrong conclusion.

The experiment was designed around those cases because they are exactly the cases that matter in engineering review.

The Setup

The corpus was the same sanitized requirements corpus used in the retrieval lab:

• 13,244 requirement records
• 9 exported modules
• structured metadata and trace links
• a frozen evidence-admissibility policy
• semantic free-text query groups authored for answerability, unsupported cases, and stress cases

The graph was partial: one directly linked requirements module could not be exported. That limitation matters for absolute coverage claims, but it did not change the selective-answering question tested here.

The full authoring frame contained 275 query groups across development, calibration, and test splits. The development gate used two development splits: one to fit or select candidate signal forms, and one to evaluate the gate without refitting.

The answerer was deliberately constrained.

It had to return strict JSON. A valid answer contained only ordered claims and evidence objects. The user-visible text was rendered deterministically from validated claims. The model could not add uncited prose in a separate answer field.

The deterministic guards converted the response to abstention if:

• the JSON or schema was invalid;
• a cited record ID was not in the supplied context;
• a cited quote was not an exact substring of that record;
• a normative answer cited inadmissible evidence;
• an answer contained no cited claims;
• a claim had no valid evidence.

Those guards are necessary.

They were not sufficient.

Three Ways to Test the Same Answerer

The most useful part of the experiment was the three-condition diagnostic.

For each development query, the system was evaluated under different context conditions:

1. Oracle evidence: provide the supporting evidence that should be enough to answer.
2. Retrieved evidence: use the actual retrieved RAG context.
3. Evidence removed: remove the labeled supporting evidence and keep plausible non-supporting context.

This separates three failure modes.

If the answerer fails with oracle evidence, the problem is generation or composition. The model cannot reliably use perfect evidence.

If it succeeds with oracle evidence but fails with retrieved evidence, the problem is retrieval or context sufficiency. The retrieved context is not good enough, or the answerer cannot handle contamination.

If it answers after the supporting evidence was removed, the problem is refusal and grounding. The answerer is willing to produce an answer without enough evidence.

That last case is the dangerous one.

The Result

With oracle evidence, the generator looked usable:

With oracle evidence, 31 of 38 answers were safe-correct. There were no harmful answers. That passed the gate’s minimum generator capability requirement.

But the retrieved-context result exposed the real risk. In the retrieved condition, the answerer published 28 answers. Ten were harmful. That is a selective risk of 35.7 percent among published answers.

The evidence-removed condition was worse as a safety signal. After supporting evidence was removed, the system still produced 20 answers. Fourteen of those were harmful.

This is the core finding:

The model could use good evidence, but it did not reliably refuse when evidence was insufficient.

That is not a retrieval-only problem. It is not fixed by saying “add citations.” The system had citations, schemas, guards, and a frozen evidence policy. It still answered too often when it should have abstained.

The Sufficiency Gate Did Not Save It

The next idea was selective answering.

Could a runtime signal decide when the context was sufficient and allow answers only then?

The experiment tested preregistered policy families:

• confidence_only: a post-generation estimate of answer safety;
• sufficiency_only: a pre-generation estimate of whether the context was sufficient;
• combined: a frozen logistic combination of confidence and sufficiency.

The most attractive option was sufficiency gating. If the system can detect insufficient context before generation, it can avoid generation cost and reduce answer risk.

But in this experiment, the automatic sufficiency rater did not work.

On the development-validation split:

• AUROC: 0.5625
• one-sided AUROC lower bound: 0.4537
• false-sufficient rate: 0.6875
• one-sided false-sufficient upper bound: 0.8196

The safety-critical error is false sufficiency: the gate predicts that the context is enough when human reviewers judged it insufficient.

That happened too often.

A gate that lets most insufficient contexts through is not a safety mechanism. It is another source of confidence.

The sufficiency_only and combined families were removed from eligibility. The remaining confidence_only family still failed to find an operating point that met the required coverage and harmful-output reduction.

The STOP Decision

The development gate had five conditions.

Two passed:

• the ungated retrieved-context baseline demonstrated a real harmful-answer problem;
• the generator was strong enough on oracle evidence to justify testing the question.

Three failed:

• no confirmatory policy family reached the required coverage and harmful-output reduction;
• the automatic sufficiency rater failed the AUROC and false-sufficient-rate gates;
• malformed generator and signal outputs exceeded the 5 percent cap.

So the decision was STOP.

No calibrated threshold policy was implemented. No calibration split was opened. No test split was evaluated. The experiment was not rescued by changing prompts, models, thresholds, or signal definitions after seeing the result.

That discipline matters.

If a lab only accepts positive results, it is not a lab. It is a demo pipeline.

What This Does Not Mean

This result does not mean “RAG is unsafe.”

It does not mean “LLMs cannot answer requirements questions.”

It does not even mean this exact architecture could never be improved.

The result is narrower and more useful:

This free-text selective-answering path did not pass the frozen development gate for safety-critical requirements RAG.

The model could answer many oracle-evidence cases. Retrieval and routing still matter. Structured execution still works for exact queries. Deterministic citation guards caught some invalid outputs.

But the system did not earn the right to publish free-text answers by default.

For safety-critical requirements, that is the conclusion that matters.

What I Would Build From This

I would not make free-text semantic answering the default interface for critical requirement decisions.

I would keep the safer execution paths first:

1. exact lookup for requirement IDs and code-like identifiers;
2. structured filters for module, status, variant, version, and ASIL;
3. deterministic counts and absence checks;
4. trace graph traversal for dependencies and impact analysis;
5. record-level retrieval for semantic discovery, followed by cautious explanation.

For the free-text path, I would treat abstention as a first-class requirement, not as a prompt preference.

That means measuring:

• whether the context is actually sufficient;
• whether each claim is supported by admissible evidence;
• whether the model refuses when evidence is missing;
• whether a confidence or sufficiency signal reduces harmful output at acceptable coverage;
• whether malformed outputs stay below a predefined cap.

If those conditions fail, the product should not quietly lower the threshold.

It should change the interaction model.

For example, the system can show retrieved requirement records, summarize only deterministic fields, explain trace links, or ask for clarification. It can help the engineer inspect evidence without pretending that it knows more than the evidence supports.

That is less flashy than a chat answer.

It is also more honest.

The Bigger Lesson

The previous retrieval experiment taught me that structured requirements should not be flattened into generic chunks.

This experiment taught me the next lesson:

Grounded answer generation is not proven by citations. It is proven by refusal behavior under insufficient evidence.

Citations are necessary. So are schemas, exact-quote checks, admissibility rules, and deterministic guards.

But they do not answer the safety question by themselves.

The safety question is whether the system can stop when the evidence is not enough.

In this lab, it could not.

That is a negative result. It is also a useful architecture result.

For safety-critical requirements RAG, the answerer must earn the right to answer. This one did not.

If you are building RAG over engineering records, do not stop at retrieval metrics and do not trust a cited answer because it looks grounded. Test the cases where the evidence is missing, rejected, incomplete, or contradictory. The most important answer in a safety-critical system is often: “I do not know from the available evidence.”