I Flattened 13,244 Requirements Into Chunks. 97% Lost Their Meaning.
The strongest universal retriever in my experiment reached 0.7467 hit@10.
Then I stopped using one retriever for every question.
The result reached 0.8933 hit@10 and cut mean latency from 860 milliseconds to 364.
The improvement did not come from a larger embedding model, a more expensive reranker, or a better prompt. It came from a less fashionable decision:
Some questions should not go through RAG retrieval at all.
That became the main result of my second retrieval lab.
The first lab used clean text documents. This time I used 13,244 sanitized software requirement records from nine source modules, with 9,235 resolved trace links between them. The records came from a structured requirements export, not from prose articles.
That difference changed almost everything.
A requirement is not just text. It has an ID, module, status, version, variant, safety classification, description, rationale, and relationships to other requirements. Flatten those fields into a long string and split it by token count, and the retrieval system can separate a value from the record it belongs to.
So the question was not simply:
Which retriever works best?
The better question was:
Which execution path is correct for each kind of engineering question?
The Setup
The evaluation corpus contained:
- 13,244 sanitized requirement records
- 9 source modules
- 9,235 resolved trace edges
- 75 manually reviewed queries with record-level relevance labels
- identifier, field-filter, module, relationship, and semantic query families
The primary embedding model was BAAI bge-m3. The strongest reranking path used BAAI bge-reranker-v2-m3.
I compared a progression of retrieval strategies:
- R0: flatten the export, chunk the resulting text, and run vector search
- R1: keep one complete requirement record as one retrieval unit
- R2: serialize each record with explicit field names
- R3: apply structured metadata constraints to retrieval candidates
- R4: fuse BM25 and vector rankings at requirement-record level
- R5: rerank R4 candidates with a cross-encoder
- R6: index smaller child units from long requirements, but return the full parent record
- R7: use structured filters, aggregation, and graph traversal where top-k retrieval is the wrong primitive
The evaluation was retrieval-first. I measured whether the system found the right requirement records before asking an LLM to generate an answer. That separation matters. If the evidence is wrong, a fluent answer only hides the retrieval failure.
Flatten-and-Chunk Failed Structurally
The most important baseline was also the worst one.
R0 flattened the structured export into text and split it into token-based chunks. It reached only 0.1200 hit@10.
More importantly, 3,150 of its 3,233 chunks contained a field-to-record association problem: visible fields could not be assigned safely to one requirement record. That is 97.43%.
This was not a small ranking regression. It was a data-model failure.
Imagine a chunk containing the end of requirement A, a status field, the beginning of requirement B, and one requirement ID. A vector database can retrieve that chunk successfully according to similarity. But the downstream system no longer knows which status belongs to which requirement.
For ordinary prose, chunk boundaries are often an optimization parameter. For structured engineering records, they can change the meaning of the data.
Keeping each requirement atomic immediately raised hit@10 from 0.1200 to 0.5467.
That single result justified a different default architecture:
Preserve the record first. Optimize retrieval second.
The Universal Retrieval Results
Here is the main comparison before query routing:
| Pipeline | Hit@10 | MRR | nDCG@10 | p95 latency |
|---|---|---|---|---|
| R0 flattened vector | 0.1200 | 0.0769 | 0.0771 | 13 ms |
| R1 record vector | 0.5467 | 0.2749 | 0.3201 | 17 ms |
| R2 field-aware vector | 0.5200 | 0.3274 | 0.3641 | 18 ms |
| R3 metadata post-filter | 0.5733 | 0.3723 | 0.4154 | 20 ms |
| R4 BM25+vector RRF | 0.5600 | 0.3784 | 0.4086 | 18 ms |
| R5 RRF + reranker | 0.7467 | 0.4996 | 0.5581 | 1.10 s |
Several results were less obvious than I expected.
Field-aware serialization was not universally better than plain record text. R2 improved MRR and nDCG@10, but hit@10 fell from 0.5467 to 0.5200. Explicit labels such as Status: and Module: helped some constrained questions, but they also changed the semantic representation enough to hurt others.
Metadata filtering helped field queries, but it was not a complete solution. R3 raised field-filter hit@10 from 0.75 to 0.95, yet semantic questions still needed semantic retrieval.
R4 was the strongest cheap general-purpose path. It combined exact lexical signals with semantic similarity and stayed near 20 milliseconds p95.
R5 produced the strongest universal ranking quality, but the reranker pushed p95 latency to roughly 1.1 seconds in this baseline run.
If I had evaluated only one mixed query set and insisted on choosing one pipeline, R5 would have looked like the answer.
It was not.
Different Questions Need Different Primitives
The failures grouped naturally by query intent.
An exact requirement ID is not a semantic concept. It should be resolved through an exact index.
A question such as “Which released requirements in module X have ASIL B?” is not ordinary top-k search. It is a structured filter.
“How many requirements have no outgoing trace link?” requires exhaustive aggregation. Failing to retrieve a record does not prove absence.
“What depends on this requirement?” is a graph-neighborhood query.
Only questions about functional intent are naturally semantic retrieval problems.
This boundary was measurable. I validated the structured executor against independent full-corpus scans:
| Structured operation | Exact checks |
|---|---|
| Field filters | 30/30 |
| Counts | 30/30 |
| Explicit absence | 8/8 |
| Outgoing graph neighborhoods | 25/25 |
| Incoming graph neighborhoods | 25/25 |
Twenty-four of the 30 filter queries had more than 20 matching records. Even a perfect top-20 retriever could not return the complete result set. The mean best-possible set recall for those cases was only 0.342.
That is not something a better embedding fixes.
Counts need aggregation. Absence needs an exhaustive check. Traceability needs graph traversal. A top-k retriever answers a different question.
The Router Architecture
Once the query families were clear, the architecture became straightforward.
The router uses deterministic parsing rather than an LLM classifier.
It extracts requirement and module identifiers, recognizes field constraints such as status and ASIL aliases, distinguishes incoming and outgoing relationship questions, and sends everything else to semantic retrieval.
The execution paths are deliberately unequal:
| Query intent | Execution path |
|---|---|
| Exact requirement ID | Exact ID index, with optional BM25 fallback |
| Module, status, version, variant, or safety constraint | Structured pre-filter |
| Count, coverage, or absence | Structured aggregation |
| Direct, incoming, or impact relationship | Trace graph traversal |
| Semantic requirement intent | Record vector or reranked hybrid retrieval |
| Semantic intent plus constraints | Structured pre-filter, then semantic ranking |
This is still a RAG system. But retrieval is now a toolbox, not one universal endpoint.
Routing Improved Quality and Latency
The automatic router matched the known-family oracle on all 75 current queries.
For the routing comparison, I reran universal R5 in the same experiment process as the automatic and oracle routers. That paired rerun kept hit@10 at 0.7467; MRR moved from 0.4996 to 0.4984, nDCG@10 from 0.5581 to 0.5571, and p95 latency from 1.10 seconds to 1.22 seconds. The small ranking differences and larger latency shift reflect variation between separate executions of the same configured pipeline. The table below uses the paired rerun so the router comparison shares one execution context.
| Pipeline | Hit@10 | MRR | nDCG@10 | Mean latency | p95 latency |
|---|---|---|---|---|---|
| Universal R5 (paired rerun) | 0.7467 | 0.4984 | 0.5571 | 860 ms | 1,220 ms |
| Automatic router | 0.8933 | 0.6687 | 0.7208 | 364 ms | 1,202 ms |
Routing fixed all ten relationship queries and one field-filter query, with no hit@10 regressions. Semantic quality stayed unchanged because semantic queries still used R5.
The latency improvement came from not reranking questions that did not need reranking:
| Query family | Universal R5 mean | Routed mean |
|---|---|---|
| Identifier lookup | 833 ms | 0.03 ms |
| Field filter | 886 ms | 20.9 ms |
| Module or scope | 483 ms | 0.28 ms |
| Relationship | 913 ms | 0.04 ms |
| Semantic requirement | 896 ms | 895 ms |
The p95 stayed near 1.2 seconds because semantic queries still invoked the expensive reranker. That is expected. Routing did not make the slow path faster. It stopped sending every query through it.
This is a useful architecture pattern beyond requirements engineering:
Do not optimize the expensive universal path until you have measured how many queries belong there.
Parent-Child Retrieval Helped, But It Was Not a Universal Default
Keeping one record atomic protects structure, but some requirement descriptions are long. A detail buried deep inside a long record can be difficult to match against one whole-record embedding.
R6 tested a compromise: index smaller child chunks, deduplicate results by record_id, and return the complete parent requirement as evidence.
On an independently written eight-query detail set, the best observed configuration used 64-token children with 12-token overlap:
| Pipeline | Hit@10 | All-relevant recall@10 | MRR | p95 latency |
|---|---|---|---|---|
| Whole record | 0.2500 | 0.1250 | 0.1502 | 16.5 ms |
| R6, 64 / 12 | 0.5000 | 0.4167 | 0.4062 | 24.0 ms |
| R6, 128 / 24 | 0.3750 | 0.3750 | 0.2812 | 20.8 ms |
The 64-token setup recovered details about release-timer conditions, actuator state after interrupted actuation, and counters retained after power loss.
But it also increased the number of indexed units from 13,244 to 29,420 and nearly doubled index size. The query set was small and contained repeated requirement templates.
So the conclusion is narrower than “smaller chunks are better.”
Parent-child retrieval is useful for long-record detail questions. It is not evidence that every requirement should be fragmented by default.
Aggregate Metrics Hid the Requirements Failure
I also placed the 13,244 requirements into one shared index with 5,000 wiki documents and evaluated 275 queries together.
The aggregate result looked healthy:
| Scope | Queries | Hit@10 | MRR | nDCG@10 |
|---|---|---|---|---|
| Aggregate | 275 | 0.8618 | 0.7909 | 0.8068 |
| Wiki | 200 | 0.9750 | 0.9470 | 0.9536 |
| Requirements | 75 | 0.5600 | 0.3747 | 0.4154 |
The requirements subset was not being destroyed by cross-corpus interference. Its result was almost unchanged from the standalone R4 run.
The problem was reporting.
The larger, easier wiki subset dominated the aggregate score and made the system look much stronger than it was on the engineering records.
For enterprise RAG, one global retrieval metric is not enough. Results need to be split by corpus and query family. Otherwise the easiest and largest content class can hide the system that matters most.
What I Would Build
For structured software requirements, I would not start with a universal vector database endpoint.
I would build five explicit capabilities:
- An exact index for requirement IDs, module IDs, and code-like identifiers.
- A structured store for module, status, version, variant, safety classification, and source fields.
- A trace graph for incoming and outgoing relationships.
- Record-level hybrid retrieval for semantic intent, with reranking only where its quality gain justifies the latency.
- Parent-child retrieval as an optional path for long-record detail questions.
Then I would put a small, inspectable router in front of them.
The LLM belongs after this layer, when the system has already selected the correct evidence and preserved the record identity. It should explain, summarize, or compare grounded records. It should not be asked to repair a retrieval architecture that mixed exact lookup, aggregation, graph traversal, and semantic matching into one similarity search call.
What This Experiment Does Not Prove
The automatic router’s 100% intent and argument accuracy is not a general classifier claim.
Several query families used generated templates. Field-filter phrasing covered a limited set of aliases. The same benchmark scope informed both implementation and evaluation. The next routing test should use independently authored paraphrases and adversarial near-miss questions.
The parent-child result is also directional because it used only eight detail queries.
And this was retrieval evaluation, not answer-generation evaluation. The study shows which records and structured results the system can recover. It does not yet measure whether an LLM turns that evidence into a correct final answer.
Those are real limitations. They do not change the retrieval conclusion.
The Bigger Lesson
The main mistake in enterprise RAG is often not choosing the wrong embedding model.
It is choosing the wrong abstraction.
We take records, tables, IDs, statuses, versions, and relationships. We flatten them into text. We split the text into chunks. Then we ask one similarity search endpoint to behave like a database, graph engine, full-text index, and semantic retriever at the same time.
When that fails, we tune chunk size, prompts, and embedding models.
Sometimes the correct fix is simpler:
Preserve the structure. Classify the question. Use the right primitive.
The strongest universal retriever in this lab was useful. It was not the final architecture.
The final architecture was a router that knew when retrieval was appropriate, when exact execution was required, and when a graph query was the only honest answer.
That is the result I would carry into a production requirements system.
If you are building RAG over engineering records, my practical advice is to test the flatten-and-chunk baseline, but treat it as a failure baseline. Label real query families, measure them separately, and make every exact or exhaustive question justify why it should pass through top-k retrieval at all.