Alpesh Kumar

Contextual Retrieval addresses a common RAG failure. A system can return a chunk that looks relevant but still misses the information the user needs. Lost context is often the culprit. For example, a sentence survives chunking, but the document, product, customer, date, or definition that makes it meaningful does not.

Anthropic calls its answer Contextual Retrieval. In its engineering write-up, the company reports a 49% reduction in failed retrievals in its evaluation. It reports a 67% reduction when teams combine the approach with reranking. These are evaluation results, not a promise that every RAG system will improve by the same amount. Instead, the central idea is simpler. Give each chunk enough context to stand on its own before a retrieval system tries to find it.

Key takeaways

• Contextual Retrieval adds a short explanation of where a chunk sits in its source document and what it refers to.
• The added context supports both embedding-based retrieval and BM25-based retrieval.
• Anthropic’s 67% figure applies to Contextual Retrieval combined with reranking in its reported evaluation.
• A small knowledge base may work better in the model prompt than behind a RAG pipeline.

Why ordinary RAG can retrieve the wrong chunk

Retrieval-augmented generation, or RAG, finds relevant material in a knowledge base and appends it to the user’s prompt. First, a system normally splits source material into smaller chunks. Next, it indexes those chunks, retrieves promising ones for a question, and sends them to the model.

Chunking is necessary, but it can turn a clear passage into an ambiguous fragment. Consider a chunk that says, Revenue increased by 12%. On its own, it does not identify the company or reporting period. It also omits the product line, geography, and definition of revenue. As a result, a query about a business unit may retrieve it because the wording closely matches. However, the chunk may answer a different question.

Embedding retrieval finds conceptual similarity. Meanwhile, BM25 is a keyword retrieval method that scores documents by the query terms they contain. Both approaches can help. However, neither can recover details that chunking removed before indexing. Anthropic identifies this loss of context during encoding as one reason traditional RAG misses relevant information.

What Contextual Retrieval changes before indexing

Contextual Retrieval enriches each chunk with a concise description of its place in the larger source. For example, instead of indexing the isolated revenue sentence, a system could prepare an enriched version like this:

This chunk is from Acme’s Q2 2025 investor update. It describes revenue for the UK subscription product line.

Revenue increased by 12%.

The example is illustrative, but it shows the key move. The original text remains available. Meanwhile, the added prefix supplies references that would otherwise be missing. As a result, a query about Acme’s UK subscription business has more signals to match.

Anthropic describes two parts of the method: Contextual Embeddings and Contextual BM25. The system embeds the contextualized chunk for semantic retrieval. It also indexes that chunk for BM25. Therefore, both retrieval paths use the same disambiguating information instead of the bare fragment alone.

Why use embeddings, BM25, and reranking together?

Semantic and keyword retrieval make different mistakes. For instance, semantic search can surface similar meanings when the terms differ. In contrast, keyword search helps when a user supplies an exact identifier, product name, or phrase. Together, they create a broader set of candidate chunks for the answer stage.

Reranking adds another selection step. Rather than accepting the initial retrieval order, the system re-evaluates the candidate set. It then promotes the chunks that best match the query. Anthropic reported a 49% reduction in failed retrievals for Contextual Retrieval alone. By comparison, it reported the 67% reduction when it combined the technique with reranking.

This distinction matters when teams plan an implementation. Contextualizing chunks can improve the material that enters retrieval. Reranking can improve the final choice from the retrieved candidates. Still, neither step replaces an evaluation that represents the documents and questions an organisation cares about.

Make the implementation decision with an evaluation

Contextual Retrieval adds a preprocessing step. A team must generate contextual descriptions and rebuild the index. It must also decide how much extra text helps without becoming repetitive. Consequently, these choices add costs in latency, model usage, and operational complexity.

Start with an A/B retrieval evaluation. First, collect real or carefully designed questions. Then define what successful retrieval looks like. Next, compare the baseline with contextualized chunks. Also, review missed results as well as successful ones. An answer can sound fluent even when retrieval selected the wrong evidence. For that reason, measure retrieval quality separately.

This approach is most promising for corpora with many similarly worded fragments, long documents, repeated terminology, or facts that depend on surrounding metadata. However, it is less compelling when a team has not observed a retrieval problem. It may also be unsuitable when the additional indexing workflow is unaffordable.

When RAG may be unnecessary

More retrieval machinery is not always the right answer. Anthropic notes that a knowledge base below 200, 000 tokens may fit directly in the prompt. In that case, teams may avoid RAG or a similar retrieval system. Direct inclusion removes the chunk-selection problem. However, it still must meet the product’s cost, latency, and context requirements.

The decision rule is practical. Start with the simplest approach that reliably gives the model the needed evidence. Then add retrieval when the knowledge base or task makes direct inclusion impractical. Finally, add contextualization when chunk-level retrieval demonstrably loses important meaning.

The verdict

Contextual Retrieval can help teams with large or context-heavy knowledge bases when relevant chunks are being missed. Its contribution is not a mysterious new search algorithm. Instead, it preserves a chunk’s meaning before embedding and BM25 indexing.

It is not a universal 67% improvement. Treat Anthropic’s figure as a result from Anthropic’s evaluation. Then validate the method against your own documents and queries. A safe first action is to audit failed searches. Next, run a small A/B test with contextualized chunks and the existing baseline.

Categories: Artificial Intelligence, Software Architecture

Tags: Anthropic, LLM, Artificial Intelligence, Claude