Hybrid Databases Were the Wrong Fix for Requirements RAG

Three databases later, the answer was still no.

Postgres could hold the structured requirement records. pgvector could run vector search. ParadeDB and OpenSearch both improved some top-10 lexical coverage.

The replacement still failed.

The problem was not that the databases were useless. They were useful. The problem was that the system was asking one hybrid backend to solve several different engineering questions at once.

That was the result of the latest requirements RAG experiment:

The quality bottleneck was retrieval policy, not database colocation.

Moving BM25, vectors, and structured records into a database did not preserve the current ranking behavior. It did not preserve the downstream evidence bundles. And when routing improved retrieval, the improvement came from choosing the right primitive for each query family, not from using a hybrid database.

That is a useful negative result.

It closes one tempting architecture path.

The Question

The previous experiments left the requirements RAG system in an awkward but understandable shape.

The system had:

• structured requirement records;
• exact execution for IDs, filters, counts, absence, and trace links;
• BM25 retrieval;
• vector retrieval;
• reranking;
• answer candidates;
• evidence admissibility;
• claim-level verification.

That architecture was not arbitrary. Earlier experiments showed why each piece existed.

Flattening structured requirements into chunks broke field-to-record meaning. Universal top-k retrieval was the wrong primitive for counts, absence checks, and trace relationships. Citations were not enough for safe free-text answers. Claim-level verification improved safety, but only after the system had built a defensible evidence bundle.

Still, the operational shape looked too fragmented.

A hybrid database seemed like the obvious simplification:

Put records, fields, graph links, full-text search, vector search, answer candidates, and verifier artifacts in one backend.

The experiment asked a narrow question:

Can a hybrid database replace the current in-process requirements RAG stack without losing retrieval quality, evidence bundle quality, or verifier interpretability?

This was not a benchmark of every database.

It was a replacement test.

If the backend could not preserve retrieval behavior and evidence bundles, it was not a safe replacement for the answering pipeline.

The Baseline

The database candidates were compared against the existing requirements retrieval machinery, not against a toy demo.

The core baseline was the current shallow BM25 plus vector RRF over field-aware requirement records:

Pipeline	Hit@10	MRR	NDCG@10
vector only	0.5200	0.3274	0.3641
BM25 only	0.3733	0.3162	0.3276
in-process equal RRF	0.5600	0.3784	0.4086

This was not the strongest possible requirements architecture. The earlier router and reranker experiments reached higher retrieval quality.

But it was the right baseline for a backend replacement.

The candidate backend had to preserve the structured store, lexical ranking, vector ranking, hybrid fusion behavior, and the evidence bundles that feed the verifier.

If it failed there, it was not a drop-in replacement.

Postgres Plus pgvector Passed the Boring Parts

The first experiment used Postgres plus pgvector.

That was the conservative choice. Postgres has tables, joins, transactions, indexes, auditability, and operational familiarity. pgvector adds vector search. If one backend could replace the current in-process stack, this was the least exotic starting point.

The structured part worked.

Postgres matched the current Python structured store on 45 of 45 live parity checks. Exact filters, counts, and trace-link operations could be represented reliably.

Vector search was also not the core problem. HNSW matched exact pgvector Hit@10 in the tested setup. Snapshot reproducibility passed.

The failure was lexical and hybrid quality.

Axis	Result	Notes
structured parity	PASS	45 / 45 live parity tests
Postgres FTS vs current BM25	FAIL	Hit@10 0.307 vs 0.373; MRR 0.179 vs 0.316
exact pgvector vs FAISS	FAIL strict gate	top-1 matched 100%; Hit@10 differed by 1.33 pp
HNSW vs exact pgvector	PASS	Hit@10 delta 0.0 pp
Postgres hybrid vs RRF baseline	FAIL	MRR 0.245 vs 0.378
evidence bundle parity	DEFERRED	0 / 50 both-match
snapshot reproducibility	PASS	50 / 50 identical bundles

The interpretation was narrow:

Postgres can host the structured store and vector search, but native full-text search and naive database-level fusion did not preserve retrieval quality.

That did not kill the broader database idea.

It only killed the simplest version.

Maybe Postgres full-text search was the weak link. Maybe the architecture would work if the lexical arm used true BM25.

True BM25 Did Not Save It

The second experiment replaced Postgres full-text search with ParadeDB BM25 while keeping the Postgres and pgvector foundation.

The hypothesis became sharper:

Postgres architecture is viable if the lexical arm is true BM25-quality search, not native full-text ranking.

That was plausible.

It still failed.

ParadeDB improved lexical Hit@10 from 0.3733 to 0.4000. But MRR dropped from 0.3162 to 0.2931, which breached the gate.

The hybrid result also failed. Hit@10 matched the RRF baseline at 0.5600, but MRR dropped from 0.3784 to 0.3302.

The evidence bundle check was worse:

0 of 50 checked bundles matched both sides.

Axis	Result	Notes
structured parity	PASS	45 tests
ParadeDB BM25 vs current BM25	FAIL	Hit@10 0.3733 to 0.4000; MRR 0.3162 to 0.2931
exact pgvector vs FAISS	FAIL strict gate	top-1 matched 100%; Hit@10 differed by 1.33 pp
HNSW vs exact pgvector	PASS	Hit@10 0.5333
ParadeDB hybrid vs RRF baseline	FAIL	Hit@10 0.5600; MRR 0.3302 vs 0.3784
evidence bundle parity	FAIL	0 / 50 both-match
snapshot reproducibility	PASS	50 / 50 reproducible

That changed the diagnosis.

The problem was not simply that Postgres full-text search was weaker than BM25.

With true BM25, more relevant records sometimes entered the top 10, but the ordering changed in ways that mattered. A retriever can improve candidate presence and still degrade the evidence bundle that the answerer actually consumes.

The diagnostic cases made this concrete.

Some diverging queries had almost no overlap with the frozen five-record bundle. Some frozen records were absent from the ParadeDB BM25 top 25. Some vector-only records ranked too low to survive RRF when the lexical arm contributed strong literal matches.

Those are not tiny tie-breaker problems.

They are policy problems.

OpenSearch Repeated the Pattern

The third experiment tested OpenSearch as the lexical backend while keeping pgvector as the vector arm.

This was a useful sanity check.

If ParadeDB failed because its BM25 behavior was not mature enough, Lucene/OpenSearch might preserve lexical ranking better.

It did not.

Axis	Result	Notes
OpenSearch BM25 vs current BM25	FAIL	Hit@10 0.3733 to 0.4000; MRR 0.3162 to 0.2839
OpenSearch plus pgvector RRF vs baseline	FAIL	Hit@10 0.5600; MRR 0.3260 vs 0.3784

The pattern matched ParadeDB:

• lexical Hit@10 improved;
• MRR dropped;
• hybrid ranking failed to preserve the baseline ordering.

That was the point where trying another database under the same hypothesis stopped being useful.

A third backend had reproduced the same shape of failure. The question had moved from “which database?” to “which retrieval policy?”

Global Fusion Was the Wrong Abstraction

Before closing the database path, the experiment tested whether simple fusion tuning could recover quality.

No new database was used. The sweep reused the existing in-process BM25 and FAISS vector retrievers, then varied vector-to-lexical weights and source budgets.

The best overall configuration was still the current shallow equal RRF:

Config	Hit@10	MRR	NDCG@10
equal RRF, 20 vector and 20 lexical candidates	0.5600	0.3784	0.4086
vector only	0.5200	0.3274	0.3641
BM25 only	0.3733	0.3162	0.3276

Increasing source budgets sometimes preserved candidate presence, but it usually hurt MRR by pushing relevant records down.

The important signal appeared only after splitting by query family:

Query family	Best strategy
identifier lookup	BM25 only
semantic requirement	vector only
field filter	vector-heavy fusion
module or scope	vector-heavy fusion
relationship	no fusion setting recovered hits

That table explains the database failures better than any vendor comparison.

There was no global hybrid setting that matched every query family.

Lexical signals helped identifier lookups. They hurt semantic requirements. Relationship questions did not need BM25 or vectors at all. They needed exact trace graph traversal.

One endpoint was being asked to behave like a keyword index, a semantic retriever, a structured filter engine, and a graph query engine.

That was the wrong abstraction.

Routing Improved Retrieval, But Not Because of the Database

Once the family-specific pattern was visible, the next experiment routed each query family to the primitive that fit it.

No new database was required.

The in-process heuristic router used deterministic surface rules and exact trace_links traversal for relationship queries. It matched the oracle best-family router on the current qrels and strongly improved retrieval metrics:

Pipeline	Hit@10	MRR	NDCG@10
vector only	0.5200	0.3274	0.3641
BM25 only	0.3733	0.3162	0.3276
equal RRF	0.5600	0.3784	0.4086
heuristic router	0.7867	0.5962	0.6333

The OpenSearch plus pgvector backend showed the same pattern under routing:

Pipeline	Hit@10	MRR	NDCG@10
OpenSearch plus pgvector equal RRF	0.5867	0.3464	0.3948
OpenSearch plus pgvector heuristic router	0.8000	0.5932	0.6367

That result is useful.

It is also easy to misread.

It does not prove that the hybrid database was better. It proves that query-family routing was better.

The backend became interesting only after the retrieval policy changed:

• identifier lookups route to lexical search;
• semantic requirements route to semantic retrieval;
• field and module queries use vector-heavy or structured retrieval;
• relationship queries route to exact trace graph traversal.

The database did not unlock the quality.

The policy did.

The Downstream Bundle Check Blocked Replacement

Retrieval-qrels improvement was not enough.

The answering pipeline depends on evidence bundles, not only on whether a relevant record appears somewhere in the top 10. A record at rank 9 may help Hit@10 and still fail to enter the five-record context bundle used by the verifier.

So the routed policy was checked against frozen selective-answering context rows.

It did not beat the old frozen R5 bundle path:

Metric	Frozen old bundle	Routed plus R5 semantic
required evidence coverage	0.5658	0.5132
admissible required evidence coverage	0.5658	0.5132

There were four improved queries and eight regressed queries.

That blocked the replacement.

For this system, the evidence bundle is the contract between retrieval and claim-level verification. A backend replacement has to preserve or improve that contract.

This one did not.

The Final Decision

The final decision was:

No-Go for hybrid databases as a drop-in replacement for the current retrieval stack.

The consolidated findings were:

• Postgres can reliably host structured requirement records and pgvector search.
• HNSW/vector reproducibility was not the problem.
• BM25 inside a database was not enough to preserve ranking or evidence-bundle behavior.
• Simple global hybrid fusion was the wrong abstraction for mixed query families.
• Query-family routing improved retrieval, but that is a RAG policy result, not a database result.
• Selective-answering bundle coverage remained better with the existing frozen R5 path than with the routed variants tested here.

The last point matters most.

If the goal is a search demo, the routed OpenSearch result looks attractive.

If the goal is safety-critical requirements answering, the replacement has to preserve the evidence path into verification.

It did not.

What This Does Not Mean

This result does not mean hybrid databases are bad.

It does not mean Postgres, OpenSearch, ParadeDB, Vespa, Qdrant, Weaviate, Neo4j, SurrealDB, or any other backend cannot be useful in a requirements system.

It means something narrower:

Moving the same retrieval problem into hybrid databases did not preserve quality as a drop-in replacement.

That distinction matters.

Postgres may still be the right place for structured records, trace metadata, audit trails, snapshots, and reproducible evidence bundles. OpenSearch may still be the right lexical sidecar for a production system. A graph database may still be useful for traceability. A vector database may still be the right operational choice for semantic search.

But those are infrastructure choices after the retrieval policy is known.

They are not substitutes for choosing the right primitive for the question.

What I Would Build From This

The architecture decision should be split into two layers.

First, choose the retrieval policy:

1. exact lookup for requirement IDs and code-like identifiers;
2. structured execution for filters, counts, absence, and variants;
3. trace graph traversal for dependency and impact questions;
4. semantic retrieval for functional-intent questions;
5. reranking only where it improves the evidence bundle enough to justify latency;
6. claim-level verification before publishing generated answers.

Second, choose infrastructure to implement that policy:

1. a structured database for records, metadata, trace links, snapshots, and auditability;
2. a lexical engine if exact term ranking needs production-grade behavior;
3. a vector index for semantic search;
4. evidence-bundle persistence so verifier inputs can be reproduced;
5. monitoring that reports metrics by query family and downstream bundle coverage, not only global Hit@10.

That second layer may still use a hybrid database.

But the reason should be operational simplicity, auditability, latency, reproducibility, or deployment constraints.

Not the hope that database colocation will fix retrieval quality.

The Takeaway

The tempting story was:

This RAG stack is complex because the data lives in too many places. Put the data in one hybrid database and the system will get cleaner and better.

The experiment showed a less convenient story:

The stack is complex because the questions are different. Some are lookup, some are filtering, some are graph traversal, and some are semantic retrieval.

A single hybrid search endpoint did not make those differences go away.

The best result in the series came from admitting the differences and routing accordingly.

For engineering requirements RAG, that is the architecture lesson:

Do not ask a database to compensate for an unclear retrieval policy.

Choose the policy first. Then pick the backend that implements it cleanly.