The RAG Verifier Learned to Say No. It Still Missed a Hard Requirement Failure.

The verifier finally did the thing the previous system could not do.

When I removed the supporting evidence, it abstained every time.

That sounds like a clean win. It was not.

The held-out test still found harmful published answers. Not many, but enough to matter. The final result was positive as a research signal and limited as a product signal:

Claim-level verification made the requirements RAG system much safer than the old sufficiency gate, but it still failed on semantic completeness in multi-action requirements.

That is the most useful kind of result. It changes the architecture, but it does not let the system off the hook.

This was the next experiment after the previous selective-answering lab. In that run, a cited JSON answerer looked good when given perfect evidence, but it answered too often when evidence was missing or insufficient. The development gate ended in STOP.

The question this time was narrower:

If the system verifies each individual answer claim against admissible requirement evidence, can it make safer publish, partial-answer, and abstain decisions than a context-sufficiency gate?

The answer was yes.

But not “yes, deploy it.”

Why the Previous Gate Failed

The previous system tried to answer requirements questions from retrieved records, then decide whether the answer should be published.

It had strict JSON. It had citations. It had deterministic guards. A claim could cite only records present in the context. A quote had to be an exact substring of the cited record. Inadmissible evidence could not support a current normative answer.

Those checks were necessary.

They were not enough.

On the development-validation retrieved condition, the ungated answerer published 28 answers. Ten were harmful. That is a selective risk of 35.7 percent among published answers.

Then I tested whether an automatic sufficiency signal could act as a runtime gate. It did not pass:

Baseline signal	Value
ungated harmful published answers	10 / 50 = 20.0%
ungated selective risk	10 / 28 = 35.7%
sufficiency autorater AUROC	0.5625
sufficiency false-sufficient rate	68.75%
malformed generator rate	9.52%
malformed signal rate	8.78%

The dangerous error was false sufficiency: the signal said the context was enough when human reviewers judged it insufficient.

That is exactly the wrong failure mode for safety-critical requirements RAG.

So I did not tune a threshold until it looked acceptable. The experiment stopped.

The New Question

The failed gate judged the whole context.

The new experiment judged the answer claim by claim.

That changes the unit of safety. Instead of asking:

Is this retrieved context sufficient?

the system asks:

For each claim in the candidate answer, is it supported by admissible evidence, contradicted by admissible evidence, supported only by inadmissible evidence, or not supported at all?

Then a deterministic aggregator maps the result to one of three outcomes:

• answer
• partial_answer
• abstain

The goal was not to make the LLM more confident. The goal was to make it more willing to withhold an answer when the evidence did not support every material claim.

For requirements engineering, that distinction is central. A requirement record is not just text. It has status, scope, version, module, rationale, and trace relationships. A rejected or “under work” record may contain plausible text, but it is not valid support for a current normative answer.

The verifier therefore had an explicit admissibility policy:

• approved records can support current normative behavior;
• inadmissible records can explain conflicts or history, but cannot support a clean current answer;
• conflicting approved records require an explicit conflict answer or abstention;
• missing evidence should fail closed.

The Pipeline

The claim-level verifier has three stages.

First, the candidate answer is decomposed into factual claims.

Second, each claim is checked against records marked [ADMISSIBLE] and [INADMISSIBLE]. The verifier must assign one of six labels, such as supported by admissible evidence, refuted by admissible evidence, conflict, or insufficient evidence.

Third, deterministic aggregation decides whether the system may publish the answer, publish a partial answer, or abstain.

The important part is that the final decision is not free-form prose. The LLM performs bounded judgments. The surrounding code enforces the output contract, evidence policy, and fail-closed behavior.

This is still an LLM system.

But the LLM is no longer asked to both write an answer and decide, globally, whether its own answer is safe. It is asked to verify smaller claims against labeled evidence.

The Development Result

The development pilot passed the pre-registered gate.

On the double-labeled development_fit subset, the claim-level verifier met all four primary GO criteria. Distribution checks on development_validate did not show large anomalies.

That was the first positive signal.

It was not enough to open the final test immediately.

Calibration review found repeatable failure modes:

• incomplete AND-conjoined actions;
• boolean logic inversion;
• under-conditioned claims;
• approved-record conflict;
• a coverage-linking artifact where strict coverage and substantive answer harm diverged.

This is where the experiment could easily have become dishonest. If I had opened the held-out test and then decided how to interpret these failures, the result would have been too flexible.

So the policy was frozen before test.

The key frozen rule was strict AND completeness:

If an approved requirement consequent contains multiple mandatory AND-actions, a clean published answer must not present one sibling action as the complete mandated behavior unless the answer explicitly scopes itself as only one part of the wider AND-set.

That rule matters later.

Model Sensitivity Before Test

Before opening the held-out split, I also ran a targeted calibration batch across several local models.

The purpose was not to find the model with the highest answer rate. The purpose was to check whether the frozen failure modes were actually handled.

Model	Answer	Abstain	Malformed
qwen2.5:14b-instruct	15	3	0
llama3.1:8b	11	7	1 verifier malformed
mistral-nemo:12b	12	6	0
qwen3:14b	17	1	0
deepseek-r1:14b	14	4	4 extraction malformed

The newer same-family model, qwen3:14b, published more answers. That looked attractive until the control rows were inspected. It missed a boolean inversion and an approved-record conflict.

deepseek-r1:14b had a different problem. Some hard rows abstained because extraction malformed and failed closed, not because the model had demonstrated the right semantic judgment. It also treated an Under Work record as support for a current normative answer in one control case.

The selected runtime candidate remained qwen2.5:14b-instruct, with admissible records rendered before inadmissible records and with the frozen pre-test rules in the verifier prompt.

That is a useful reminder: higher answer rate is not necessarily better coverage. It can also be lower caution.

The Held-Out Test

The held-out test split contained 187 condition rows:

Condition	Rows
retrieved	75
oracle	56
evidence_removed	56

The three conditions test different things.

oracle gives the verifier the evidence that should be sufficient.

retrieved uses the actual RAG context.

evidence_removed removes the supporting evidence and leaves plausible non-supporting context.

The last condition is the safety diagnostic. A grounded system should not keep answering when the evidence has been removed.

The automatic results were strong:

Metric	Value
rows processed	187
extraction malformed rate	0 / 187 = 0.0%
verifier malformed count	0
inadmissible leakage count	0
mean latency over all rows	5757.4 ms
mean latency on LLM-call rows	9612.8 ms

The decision distribution showed the system was conservative:

Condition	Rows	Answer	Partial answer	Abstain
retrieved	75	24	1	50
oracle	56	37	2	17
evidence_removed	56	0	0	56

That last row is the big improvement.

In the previous experiment, evidence removal exposed the failure: the answerer continued to publish unsupported answers. In this held-out test, the claim-level verifier abstained on all 56 evidence-removed rows.

That is not a small prompt improvement. It is a different safety mechanism.

The Human Audit

Automatic metrics were not enough.

The test run could tell me whether the verifier malformed, leaked inadmissible evidence, or abstained in the evidence-removed condition. It could not prove that every published answer was substantively safe.

So every published held-out row was reviewed:

Reviewed published rows	Rows
answer	61
partial_answer	3
total	64

By condition:

Condition	Reviewed rows
retrieved	25
oracle	39

The final adjudicated audit found:

Metric	Value
strict row result	55 pass / 9 fail
substantive row result	60 pass / 4 fail
harmful published answers	4 / 64 = 6.25%
retrieved harmful published answers	1 / 25 = 4.0%
oracle harmful published answers	3 / 39 = 7.69%

On the primary retrieved condition:

Primary retrieved metric	Value
published rows	25 / 75 = 33.3%
harmful published rows	1 / 75 = 1.33%
selective risk among published rows	1 / 25 = 4.0%
non-harmful published rows	24 / 75 = 32.0%

That is the core result.

Compared with the previous retrieved baseline, selective risk fell from 35.7 percent to 4.0 percent. Harmful published output fell from 20.0 percent of retrieved rows to 1.33 percent.

At the same time, the system did not simply abstain on everything. It published non-harmful answers on roughly one third of retrieved rows.

For a research experiment, that supports the hypothesis.

For a safety-critical product, the remaining failures still matter.

The Failure That Remained

The human audit found four harmful published answers:

Query	Condition	Failure mode
qa-sa-032	oracle	antecedent incompleteness / commission error
qa-sa-145	oracle	AND-action incompleteness
qa-sa-243	oracle	AND-action incompleteness
qa-sa-243	retrieved	AND-action incompleteness

The dominant residual failure was semantic completeness for conjoined requirements.

This is a subtle failure. The model can state something true and supported, but still make the answer harmful by presenting it as complete.

Imagine a requirement whose consequent is:

When condition X holds, the system shall do A, B, and C.

If the answer says:

When X holds, the system shall do A.

that may be a supported claim. But if the question asks for the required behavior, the answer is incomplete. It omits sibling mandatory actions B and C. In a safety-critical setting, that can be harmful even though the published sentence is not obviously false.

This is where ordinary citation checking breaks down. The answer can cite a real approved record. The cited text can contain action A. The local claim can be supported.

The failure is in the missing structure of the requirement.

That is why the frozen strict AND policy mattered. Under a looser query-scoped interpretation, some of these rows would be strict failures but not substantive harms. I did not adopt that looser interpretation after seeing the test labels, because that would have relaxed the failure rule post hoc.

The Final Classification

The final classification was:

Limited / Mixed.

Not clean GO.

Not terminal STOP.

The verifier passed the automatic safety plumbing checks:

• no malformed verifier output;
• no inadmissible-evidence leakage;
• all evidence-removed rows abstained.

It also materially improved the harm and coverage tradeoff over the old sufficiency gate:

• selective risk among published retrieved answers fell from 35.7% to 4.0%;
• harmful retrieved output fell from 20.0% of rows to 1.33%;
• useful retrieved publication remained around one third of rows.

But the held-out human audit still found harmful published answers, including one in the primary retrieved condition.

That means the system is evidence-disciplined and much safer than the previous architecture, but it is not robust enough to claim product-ready success for safety-critical use.

What This Changes Architecturally

The experiment changes my default architecture for requirements RAG.

I would not rely on a whole-context sufficiency score as the main safety signal. In this setting, it was too weak and too prone to false sufficiency.

I would also not trust citations alone. Citations prove that the model pointed at something. They do not prove that every material claim is supported, that inadmissible records were ignored, or that mandatory sibling actions were not omitted.

The better shape is:

1. route exact IDs, filters, counts, and trace questions to deterministic execution;
2. use retrieval only for semantic discovery;
3. generate candidate answers under strict evidence constraints;
4. decompose answers into claims;
5. verify each claim against admissible and inadmissible evidence;
6. check required-claim coverage deterministically;
7. abstain aggressively when support is incomplete.

Even then, claim-level verification is not the end.

The remaining failure points to a more structured next step: requirements need to be decomposed into antecedents and consequent action sets. A verifier should not only ask whether claim A is supported. It should know when A is one member of an AND-set and whether the answer has represented the full set correctly.

That is not a bigger prompt. It is a stronger representation.

What I Would Build Next

I would target the residual failure mode directly.

The next version should extract requirement logic into a normalized structure:

• antecedent conditions;
• mandatory consequent actions;
• AND and OR relationships;
• status and admissibility;
• conflicts between approved records;
• scope of the user question.

Then the answer verifier can run deterministic checks that are currently too semantic:

• Did the answer omit sibling mandatory AND-actions?
• Did it drop required antecedents?
• Did it invert NOT, AND, OR, or Invalid logic?
• Did it present a scoped partial answer as a complete requirement?
• Did it handle approved-record conflicts explicitly?

The target is not a chatbot that sounds more certain.

The target is an engineering assistant that knows when the evidence supports a complete answer and when the only correct answer is abstention.

The Takeaway

This experiment did not produce a clean deployment claim.

It produced something more useful:

Claim-level verification substantially improves safety over context-sufficiency gating, but robust held-out success requires stronger handling of multi-action and under-conditioned requirement logic.

That is the right level of conclusion.

The system learned to say no when the evidence was removed. That is a real improvement.

It still missed a hard requirements failure where the answer was locally supported but globally incomplete. That is the next engineering problem.

For safety-critical requirements RAG, that is exactly the kind of distinction the evaluation has to expose.