The Requirements RAG Router Failed. Then the Benchmark Failed Too.

The router looked like the answer.

On the original requirements benchmark, it raised hit@10 from 0.7467 to 0.8933 and cut mean latency from 860 milliseconds to 364.

Then I tested it on independently paraphrased and adversarial questions.

It failed.

I added a deterministic query-grounding layer between the router and the requirements store.

That failed too.

When I investigated the worst query family, I found a more uncomfortable problem: the benchmark itself had grouped several different tasks under one label. Some rows were testing module aliases. Others were testing text and symbol search. A few gold result sets were contaminated by unsafe substring matching.

The system was wrong.

The measuring instrument was also wrong.

That changed the direction of the experiment.

Before a requirements RAG can choose the right retrieval primitive, the benchmark has to distinguish the tasks, and the system has to translate the question into an explicit evidence plan.

After repairing the taxonomy and manually reviewing the affected rows, a typed query-to-evidence executor passed a narrow 12-query gate with complete evidence recall.

That is not a claim that requirements RAG is solved.

It is a more useful result: a clearer boundary between query understanding, structured execution, retrieval, evidence assembly, and answer generation.

Where This Fits in the Series

This article continues a series of measured experiments on RAG over structured engineering requirements.

The earlier work established three boundaries: requirement records should remain atomic rather than being flattened into arbitrary chunks; exact filters, counts, absence checks, and trace relationships need different execution primitives; and moving retrieval into a hybrid database does not automatically preserve ranking or evidence-bundle quality.

If you are joining the series here, start with the structured requirements retrieval experiment, then read why hybrid databases were the wrong fix. The claim-level verification experiment explains the downstream verifier that this router was eventually expected to support.

This article tests the missing boundary: whether the router can reliably translate independently authored human questions into the right executable evidence path.

The Result I Was Trying to Validate

The previous requirements retrieval experiment ended with a simple architecture lesson:

Choose the retrieval policy first. Then choose the backend that implements it.

Postgres, ParadeDB, and OpenSearch could all host useful pieces of the requirements system. None worked as a drop-in quality replacement for the current retrieval stack. The strongest improvement came from routing each query family to a different primitive:

• exact lookup for requirement IDs;
• structured execution for filters, counts, and absence;
• graph traversal for trace relationships;
• lexical or semantic retrieval for discovery questions;
• reranking only where the evidence gain justified the latency.

In the earlier requirements track, that policy looked strong on the existing 75-query retrieval benchmark.

The automatic router reached:

Pipeline	Hit@10	MRR	Mean latency
universal R5, paired rerun	0.7467	0.4984	860 ms
automatic router	0.8933	0.6687	364 ms

But the hybrid database series had already exposed a warning.

The routed policy improved retrieval qrels and still produced worse downstream evidence bundles than the frozen R5 path:

Metric	Frozen R5 bundle	Routed bundle
required evidence coverage	0.5658	0.5132
admissible required evidence coverage	0.5658	0.5132

The router could look better at top 10 and still give the answerer worse evidence.

So the next question was not whether routing improved one retrieval score.

It was:

Can the router remain reliable on independently paraphrased, adversarial, structural, and multi-claim requirements questions before its evidence reaches the answerer and verifier?

The Independent Router Gate

I built a sealed 176-row evaluation set.

It included identifier lookups, filters, counts, absence questions, relationship questions, semantic requirements questions, and a family called module_or_scope. The set combined reused semantic questions with newly authored structural queries and harder paraphrases.

The gate was frozen before the run:

• overall route accuracy had to reach at least 0.85;
• each sufficiently large structured family had to reach at least 0.75;
• structured argument extraction had to reach at least 0.80.

The router failed all four headline conditions:

Gate	Required	Actual
overall route accuracy	0.85	0.778
field_filter route accuracy	0.75	0.125
module_or_scope route accuracy	0.75	0.000
structured argument extraction	0.80	0.471

The easy families remained easy.

Identifier lookup and relationship routing both stayed at 1.000. Semantic questions remained strong. The failures concentrated at the boundary between human language and executable structure.

The router recognized literal sanitized module IDs such as MOD_<hex>. It did not resolve names such as PBC or SW-System.

It could execute exact scalar fields. It did not know what to do with constraints such as:

• signal;
• unit;
• monitor;
• constant;
• action;
• term;
• source document;
• free-text scope.

Thirty-nine structured rows could not even enter the exact structured-correctness calculation.

That was not one regex bug.

It was a missing layer.

Query Grounding Was the Next Fix

The next mechanism was a deterministic query-grounding layer.

Instead of handing raw router arguments directly to the structured store, the grounder would build an explicit evidence plan.

Conceptually:

human query
    -> route decision
    -> resolved and unresolved constraints
    -> evidence plan
    -> structured, graph, text, or semantic execution

The plan made several distinctions explicit:

• exact scalar filters;
• module and source-document entities;
• text and symbol constraints;
• trace direction;
• semantic fallback;
• unsupported constraints.

That was an improvement in architecture even before it was an improvement in metrics. A constraint could no longer disappear silently into not_computed. The system had to say whether it had resolved, searched, deferred, or rejected it.

The broad grounding gate still failed.

Metric	Result
overall grounded-intent accuracy	0.790
module_or_scope grounded-intent accuracy	0.000
executable-plan coverage on structured rows	0.607
module/source entity-resolution accuracy	0.000, n=1
exact correctness on non-fallback computed plans	0.917

Only one row was eligible for the module/source entity-resolution metric. Its 0.000 result triggered the fail-closed gate, but the sample is too small to describe broad entity-resolution performance.

The last number is important but easy to misread.

When the mechanism produced a non-fallback plan, execution was usually correct. The problem was coverage: too many questions still could not become an honest executable plan.

The new layer resolved 29 of the previous 39 not_computed rows. Only four became genuine computed pass/fail rows. Twenty-five were instead classified as impossible_authoring_mismatch because their expected values did not exist in the corpus in the form the benchmark claimed.

The gate failed correctly.

Then the investigation moved from the matcher to the data.

The Alias Source Did Not Exist

The most visible failure was module_or_scope.

The working hypothesis was straightforward:

Human module abbreviations should be linked to sanitized MOD_<hex> identifiers through an alias index.

I tested the available sources.

Among the 16 values that the benchmark then treated as module names, the crosswalk resolved four unambiguously, found one ambiguous value, and left 11 unmatched.

The module_abbrev field added nothing. In the sanitized corpus, it repeated the opaque module identifier.

Raw pre-sanitization fields did not contain a reliable alias table.

Identifier naming conventions did not recover the expected names.

Free-text co-occurrence found mentions, but they were distributed across multiple modules. A term appearing in records from several modules is not a trustworthy alias-to-module mapping.

The conclusion was NO:

The available corpus did not contain a reliable, reproducible source that could map the expected human aliases to sanitized module IDs.

This matters because it defines an engineering boundary.

If the source of truth does not exist, a better fuzzy matcher does not create one.

It creates a more confident guess.

The Benchmark Was Mixing Different Tasks

A later provenance audit reconstructed how the 16 module_or_scope gold result sets had actually been authored.

The result changed the interpretation of the failed gate.

Only three of the 16 values were real module or source-document aliases.

Even those three queries were not asking for records inside the module. Their gold sets contained records that referenced the named artifact.

The other 13 values were text or symbol scope terms: component names, function abbreviations, signal families, parameter names, and similar domain vocabulary.

Three result sets were materially contaminated:

• ECD matched fragments inside sanitized hexadecimal identifiers;
• DAA had the same collision problem;
• CEP matched unrelated English-word substrings.

The old family label implied one task:

human module name -> module ID -> module members

The rows were actually testing several tasks:

module alias -> module membership
module alias -> records that reference the module
domain term -> matching requirement text
domain term -> matching structured metadata
unsupported term -> explicit failure

Those tasks need different gold semantics and different execution paths.

The previous failure should not be reported as “the alias matcher failed on 16 real module aliases.”

The correct conclusion is narrower:

The benchmark asked one matcher to solve several different tasks because the taxonomy had collapsed them into one family.

Repairing the Measuring Instrument

I did not edit the sealed dataset after seeing the result.

Instead, I added a hash-bound repair manifest that joined each legacy row to a corrected family and eligibility decision.

The 16 old rows became:

Repaired family	Rows	Initial eligibility
module_reference_scope	3	clean
text_symbol_scope	10	diagnostic only
text_symbol_scope_contaminated	3	excluded

The ten text and symbol rows still needed safe gold.

A bounded blinded review package contained 192 judgments across those ten queries. Candidate provenance and record IDs were hidden from the reviewer. Labels were relevant, not_relevant, and uncertain, with uncertainty handled fail-closed.

Nine rows passed their individual review gates.

The approved slice contained 88 reviewed safe candidates, 76 of them relevant:

Metric	Result
approved query rows	9
reviewed safe candidates	88
relevant safe candidates	76
safe precision	0.864
one-sided 95% Wilson lower bound	0.793

One row remained diagnostic-only. The three contaminated rows stayed excluded.

This was a partial GO, not a retroactive claim that the original benchmark had passed.

The frozen all-row gate remained STOP for audit transparency.

That distinction is important. Repairing a benchmark should preserve the history of why the original result was invalid.

Text Search Helped, but It Was Not the Answer

The first mechanism tested on the repaired text and symbol slice was safe-term field search.

It restricted matching to allowed requirement fields and applied conservative term rules intended to avoid the substring failures found in the legacy gold.

The result improved both recall and ranking:

Pipeline	Recall@20	MRR
field-aware BM25	0.054	0.167
allowed-fields BM25	0.059	0.081
safe-term then BM25	0.171	0.314

The safe-term branch improved Recall@20 by 0.112 over the strongest baseline and MRR by 0.147.

It passed its frozen gate.

It also missed three of the nine approved rows completely at Recall@20.

Those rows depended on structured scope facts rather than a prose-term match.

That failure reinforced the architecture lesson from the database experiment. One search endpoint was still the wrong abstraction. Even inside the repaired text_symbol_scope family, some questions required structured metadata and others required controlled field search.

The result was a qualified GO:

Safe-term search is useful as one typed evidence-plan branch, not as a universal retriever.

Typed Evidence Plans

The final mechanism compiled each approved query into a typed evidence plan.

The compiler received:

• the query text;
• a trusted local policy vocabulary.

The executor received the resulting plan and the canonical requirements corpus.

Neither stage received:

• the query ID;
• gold labels;
• expected record IDs.

The executor could choose among:

• structured metadata conditions;
• safe prose conditions over allowed fields;
• literal reference-scope search;
• OR alternatives;
• explicit unresolved reasons.

The gate combined the three clean reference-scope rows with the nine manually approved text and symbol rows.

Metric	Current baseline	Typed executor
executable-plan coverage	n/a	1.000
overall evidence-record recall	0.037	1.000
module_reference_scope recall	0.000	1.000
text_symbol_scope recall	0.049	1.000

All 12 rows had non-zero recall.

None was unresolved.

The gate passed.

This is the positive result in the series.

It is also deliberately narrow.

The policy vocabulary remains local because it contains domain values from the trusted environment. The 12-row slice covers two repaired families from one corpus. The result does not establish general module-alias resolution, broad query grounding, answer correctness, or portability to another requirements system.

The typed executor passed the task it was actually tested on.

Nothing more.

The Correct Output Was Not an LLM Answer

The last step revealed another boundary that the old RAG framing had hidden.

All 12 approved queries were set-valued commands such as:

List requirements matching this scope.

They contained expected record ID sets.

They did not contain required natural-language claims.

The executor materialized:

Metric	Result
queries	12
materialized queries	12
unresolved queries	0
total returned records	10,731
mean agreed-gold evidence recall	1.000

Turning those complete record sets into an LLM answer would make the result worse.

The 12 complete result sets contained 10,731 records in total. Feeding those set-valued outputs through a fixed 4,096-token answer context would necessarily truncate the larger sets. A prose summary would sample, compress, or omit records from a query whose expected result is exhaustive.

So answer generation was marked not_applicable_structured_result.

Claim verification was marked not_applicable_no_required_claims.

That was not a skipped downstream gate.

It was the correct terminal path.

The system now had to distinguish at least three output types:

1. a complete structured result;
2. a bounded evidence bundle for a natural-language answer;
3. an explicit unsupported or abstention result.

Calling all three “RAG answers” had obscured the contract.

The Final Decision

The broad router did not pass independent reliability validation.

The first deterministic grounding layer did not pass its broad gate.

The alias source required by the original interpretation did not exist.

The benchmark taxonomy was partially invalid.

After repairing the taxonomy and manually reviewing the affected slice, typed evidence plans passed a narrow structured-result gate.

The consolidated decision is:

Close the open-ended requirements RAG research phase. Do not open another unconstrained router, retriever, matcher, fusion, database, model, or agentic-RAG tuning series.

What This Does Not Mean

This result does not prove that typed query planning solves requirements RAG.

It does not validate natural-language answer generation on the repaired queries.

It does not show that the local policy vocabulary generalizes to a second corpus.

It does not repair the unsupported module-alias problem.

It does not convert the earlier router or grounding failures into passes.

And it does not justify a single global score for requirements RAG.

The result is narrower:

Once the benchmark distinguished the tasks, a typed executor could solve the approved structured-result slice by choosing explicit execution semantics instead of hiding every question behind one retrieval endpoint.

That is useful architecture evidence.

It is not a production claim.

The Takeaway

The tempting story after the database experiment was:

Different questions need different retrieval primitives, so build a router.

That story was incomplete.

The router still had to understand human constraints. The corpus still needed authoritative alias data. The benchmark still had to distinguish module membership from module references and text or symbol search. The output contract still had to distinguish an exhaustive result set from a bounded evidence bundle and a generated answer.

The deeper lesson is:

Before optimizing a requirements RAG, make the question, evidence semantics, and expected output type explicit — and test the measuring instrument as hard as the system.

The router failed.

The first grounding mechanism failed.

The benchmark failed too.

Repairing the benchmark did not make the whole system work.

It made the next result honest.