I Tested a Simple RAG Idea: One Summary per Document. It Was Useful, But Not Enough.

Most RAG discussions start too late.

They start with the LLM. Which model should answer? How large should the context window be? Which prompt template should we use? How do we make the answer sound confident and helpful?

Those questions matter. But they are not the first questions.

The first question is simpler and less glamorous:

Did retrieval find the right document?

If retrieval brings the wrong evidence, the generator is already in trouble. A stronger LLM may produce a smoother answer, but it is still answering from the wrong material. In a RAG system, the LLM is the last step. The system lives or dies much earlier.

That is why I built a small retrieval lab.

The goal was not to produce an academic benchmark. The goal was to test one practical idea before turning it into an architecture decision.

The idea was this:

Instead of indexing many chunks from each document, generate one LLM summary per document, embed that summary, retrieve the document through the summary, and then use the full document as context for the LLM.

It is an attractive idea. It sounds clean. It reduces the size of the vector database. It gives document-level attribution. It avoids returning five unrelated chunks from the same document. It may also help when the document text is noisy, because the summary can normalize the content into something easier to search.

It is also exactly the kind of idea that should not be accepted because it sounds reasonable.

So I tested it.

The Setup

The corpus was deliberately modest: 5,000 SimpleWiki-like text documents.

The evaluation set contained 200 manually written queries with document-level relevance labels. Each query had a known relevant document. The retrieval task was simple: does the pipeline return that document near the top?

I measured:

• hit@10: did the relevant document appear in the top 10?
• MRR: how high did the first relevant document appear?
• NDCG@10: rank-sensitive relevance quality in the top 10
• mean retrieval latency

This was retrieval-only evaluation. I did not ask an LLM to generate final answers yet. That was intentional. I wanted to isolate the instrument. If the retriever misses the relevant document, the rest of the RAG chain cannot fix that reliably.

The main pipelines were:

• title-only retrieval: one vector per document, using only the article title
• summary retrieval: one LLM-generated summary per document
• entity-focused summary retrieval: same idea, but with a prompt tuned toward names, dates, places, and key facts
• BM25
• vector search over the original text
• document-level BM25+vector RRF
• reranking

The title-only pipeline was a control. If titles performed close to summaries, the summary was not adding much. If summaries performed much better than titles, then document representation quality clearly mattered.

The Results

Here is the compressed version.

The original-summary latency is not directly comparable with the entity-summary latency. The original summary run used the DocumentSummaryIndex retriever path, while the entity-summary run used a plain vector index over summary nodes. I care about the retrieval-quality comparison here more than the latency difference between those two summary rows.

The summary retriever was not bad.

That is important. It would be easy to frame this as “summary retrieval failed”. That is not what happened.

The title-only control was weak: 0.610 hit@10. The LLM-generated summaries were much stronger: 0.920 and 0.945 hit@10 depending on the prompt and implementation. The summary clearly carried useful retrieval signal.

But the strong baselines were better.

Document-level BM25+vector RRF reached 0.990 hit@10 and 0.944 MRR. Reranking had slightly lower hit@10 than doc-level RRF in this run, but the best MRR and NDCG@10.

That changed the decision.

One summary per document is useful. It is not the primary retriever I would choose for this corpus.

What Surprised Me

The surprising part was not that summary retrieval lost to BM25+vector RRF.

The surprising part was how good it was before losing.

A single generated summary per document reached 0.945 hit@10. That is strong enough to be tempting. Without baselines, I might have accepted it as “working” and moved on to building the rest of the RAG pipeline around it.

That would have been a premature architecture decision.

The lab changed the question from:

Does summary retrieval work?

to:

Does summary retrieval work better than the boring baselines?

That second question is the one that matters.

Prompt Quality Matters, But It Did Not Save the Strategy

I wanted to make sure I was not dismissing the idea too early.

Maybe the first summary prompt was simply bad. Maybe a better prompt would preserve the retrieval cues that matter: names, dates, roles, places, events, aliases. So I ran an entity-focused prompt and compared it with the original summary prompt.

The entity-focused prompt changed the behavior.

It recovered some misses from the original summary run, especially entity-heavy questions. But it also lost other hits. On the 200-query set:

• 16 queries improved in rank
• 22 queries got worse
• 162 queries stayed the same
• 6 original-summary top-10 misses were recovered
• 11 original-summary top-10 hits were lost

The confidence intervals did not support a stable claim that the entity-focused prompt was better than the original summary prompt.

For example, the original summary missed this query in the top 10:

Who was the teenage serial killer who terrorized Chicago in 1945 and 1946, known for communicating with police through messages left at murder scenes?

The relevant document was William Heirens. BM25 found it at rank 1, vector search at rank 3, and the reranker at rank 1. The original summary found it only at rank 14. The entity-focused prompt recovered this case.

That is exactly why I ran the prompt-quality experiment. Some summary failures are prompt-sensitive.

That is a useful result. It means prompt design matters, but prompt tuning did not remove the structural limitation in this setup.

The limitation is compression.

A single generated summary has to decide what the document is “about”. Retrieval queries do not always ask for the main idea. They ask for a side fact, a name, a date, a role, a small relationship, a phrase that appears once, or a connection that the summary did not preserve.

BM25 and vector search over the original text do not have to compress the document into one representation. They can exploit lexical and semantic signals distributed across the text. RRF can combine those signals at the document level. A single summary vector cannot reliably carry all of them.

That does not make summaries useless. It gives them a ceiling.

A Few Concrete Cases

The misses were not random. They were exactly the kind of misses I would expect from aggressive document compression.

One query asked:

What is the feast day of the patron saint of England?

The relevant document was April 23. BM25 found it at rank 1, vector search at rank 4, doc-level RRF at rank 1, and the reranker at rank 2. Both summary prompts missed it in the top 20.

That is a useful failure. The document contains the evidence, and the standard retrievers can reach it. The summary representation did not preserve enough of the specific retrieval cues from the query.

Another query asked:

Which Japanese international footballer played as a midfielder for Fiorentina in Serie A and won the Asian Footballer of the Year award?

The relevant document was Kōji Nakata. Vector search found it at rank 3, doc-level RRF at rank 2, and the reranker at rank 3. Both summary prompts missed it in the top 20.

This is the uncomfortable case for summary retrieval: the relevant document is not hard to retrieve from the original text, but it becomes hard after compressing the document into one generated representation.

But summary retrieval was not always worse.

For the query:

Which Swedish economist chaired the Nobel Prize in Economics committee for several decades?

the summary retriever ranked Assar Lindbeck at rank 1, while the best baseline had it at rank 2.

And for:

What 1999 Bollywood romantic film starred Ronit Roy and Rambha in the lead roles?

the summary retriever ranked Deewane Tere Pyar Ke at rank 3, while the best baseline had it at rank 9.

Those cases are why I would not throw summaries away completely. They can capture a useful document-level signal. The problem is that the signal is not reliable enough to replace stronger retrieval over the original text.

The Methodological Trap I Had to Fix

One important correction came out of the experiment.

An earlier hybrid RRF run used chunk-level fusion. The evaluation labels were document-level. That mismatch made the interpretation muddy.

Chunk-level RRF is not “bad”. It is measuring a different object.

If relevance is labeled at the document level, the clean comparison is document-level fusion:

1. Run BM25 and vector search.
2. Deduplicate each source ranking to one candidate per document ID.
3. Apply RRF over document rankings.
4. Evaluate against document-level qrels.

After making that correction, doc-level BM25+vector RRF became the strong cheap baseline.

This is exactly why a lab is useful. The mistake was not in the code crashing or the model hallucinating. The risk was quieter: measuring with a slightly wrong instrument and then drawing an architectural conclusion from it.

That is the kind of error I want to catch before a 12-hour run becomes a production recommendation.

Does Summary Help as an Extra Fusion Signal?

After summary-only retrieval lost to BM25+vector RRF, the next question was obvious:

What if summary retrieval is not the main retriever, but an additional signal?

So I tested BM25+vector+summary fusion.

The result was not convincing.

The best BM25+vector RRF run had:

• hit@10: 0.990
• MRR: 0.944
• NDCG@10: 0.955
• mean latency: 0.0099s

Adding summaries could recover one additional top-10 hit in a shallow configuration, but it reduced MRR and NDCG. It also made retrieval roughly 8.7 times slower in that comparison.

An offline union analysis found no query where summary top-10 recovered a relevant document that was absent from the BM25+vector union@20. On this eval set, the summary mostly duplicated signal already present in the two cheaper baselines.

So for this corpus, summary retrieval did not earn a place as the third fusion source either.

What I Would Build From This

For this corpus and query set, I would start with document-level BM25+vector RRF.

It is simple, fast, cheap, and strong. It also has a nice engineering property: BM25 and vector search fail differently. BM25 is good at exact words, names, and rare terms. Vector search is good at semantic similarity. RRF gives a practical way to combine them without overfitting a complex scoring model too early.

If ranking quality matters more than latency, I would add a reranker after candidate retrieval.

I would not use one-summary-vector-per-document as the primary retrieval strategy here.

I would still keep summaries in the system, but for different jobs:

• human-readable document previews
• debugging and explainability
• metadata or tag generation
• routing for much longer or more heterogeneous documents
• compression after stronger retrieval has selected evidence
• possible separate experiments for slides, PDFs, and mixed document types

That last point matters. This experiment was done on clean text documents. Presentations, scanned PDFs, tables, and mixed media are different problems. Summary-like representations may become more useful there, especially when the raw extraction quality is poor or when the document is not naturally searchable as plain text.

But that is a different experiment, not a reason to ignore the current result.

The Bigger Lesson

The most useful result of this lab was not the final ranking table.

The useful result was the discipline.

Before deciding on a RAG architecture, I now want to see:

• a representative query set
• relevance labels
• retrieval metrics before generation metrics
• simple baselines
• latency per stage
• error analysis
• confidence intervals on the differences
• a clear distinction between chunk-level and document-level evaluation

Without that, it is too easy to confuse a convincing demo with a working retrieval system.

RAG depends on many parameters: the corpus, extraction quality, chunking, metadata, embedding model, BM25 configuration, fusion, reranking, prompt construction, context budget, and only then the LLM. A vendor-default pipeline or a fashionable architecture may work. It may also be wrong for the actual documents and questions.

You do not know until you measure.

That is the point I want to keep from this experiment.

The summary idea was plausible. It was worth testing. It produced useful signal. It also lost to a stronger, simpler baseline.

That is not a disappointing result. That is exactly what a retrieval lab is for.

If you are building RAG over internal documents, my practical advice is simple: do not start with the architecture diagram. Start with the measuring instrument. Build a small retrieval lab, label real queries, compare boring baselines, and only then decide which pipeline deserves implementation effort.