Proposal: Add Advanced Retrieval Processors (CommunityToolkit.DataRetrieval)

[luisquintanilla]

Proposal: Add Advanced Retrieval Processors (CommunityToolkit.DataRetrieval)

Summary

This issue proposes adding advanced retrieval pipeline processors to the AI Community Toolkit, building on the Microsoft.Extensions.DataRetrieval abstractions (proposed in dotnet/extensions#7507). These processors enable sophisticated RAG (Retrieval-Augmented Generation) patterns — query expansion, hypothetical document embeddings, LLM reranking, corrective RAG, and more — through a fluent DI-friendly builder API.

Motivation

What This Package Provides

The Microsoft.Extensions.DataRetrieval packages (proposed in dotnet/extensions) give you the abstractions — the pipeline structure, base classes, and data types. But abstractions alone don't help you build a better RAG system. You still need concrete implementations of the techniques that actually improve retrieval quality.

That's what this package provides: production-ready, tested implementations of proven retrieval techniques, wired up through a fluent builder that handles DI registration automatically.

The Techniques Explained

Pre-Search: Making Your Query Better Before Searching

Multi-Query Expansion (MultiQueryExpander)

The problem: A single query embedding may not capture all aspects of what the user is asking. "How do I handle errors in my API?" is one question, but it relates to exception handling, middleware, HTTP status codes, and validation — each might be stored in different chunks.

The solution: Ask an LLM to generate N variant phrasings of the original query. Search with ALL of them, then merge results using Reciprocal Rank Fusion (RRF) — a technique that boosts chunks appearing in multiple result sets. A chunk that appears for 3 out of 5 query variants is almost certainly relevant.

HyDE — Hypothetical Document Embeddings (HydeQueryTransformer)

The problem: Users ask short questions ("How do I configure CORS?") but documents contain long explanations. Short questions and long answers live in different regions of vector space — their embeddings aren't as close as you'd expect.

The solution: Ask an LLM to generate a hypothetical answer to the question (it doesn't need to be correct). Then search using the hypothetical answer's embedding instead. Since a hypothetical answer looks like a document chunk, its embedding is naturally closer to the real answer chunks in vector space.

Adaptive Routing (AdaptiveRouter)

The problem: Not all queries benefit from the same retrieval strategy. A simple factual question ("What's the default timeout?") needs basic vector search. A broad architectural question ("How does the authentication system work?") needs hierarchical tree search. A question about a specific person or technology might benefit from metadata-filtered search.

The solution: Ask an LLM to classify the query's complexity and type, then route to the optimal search paradigm. Simple → standard vector search. Thematic/broad → tree search through summaries. Entity-specific → filtered vector search.

Post-Search: Improving Results After Search

LLM Reranking (LlmReranker)

The problem: Vector similarity is an approximation. The "closest" vectors aren't always the most relevant for answering the specific question. A chunk about "configuring CORS headers" might be geometrically closer than one about "CORS middleware setup," even though the latter actually answers the question better.

The solution: Retrieve more candidates than you need (say 20), then ask an LLM to rank them by actual relevance to the question. The LLM reads each chunk and the query, then orders them by how well each chunk answers the question. Return only the top N.

Corrective RAG (CragValidator)

The problem: Sometimes vector search returns chunks that are topically related but don't actually answer the question. If you pass these to the generation LLM, it may hallucinate an answer that sounds plausible but isn't grounded in the retrieved content.

The solution: Before using the results, ask an LLM to score retrieval quality (1-5). Based on the score, route through three paths:

• Correct (4-5): Results are relevant, proceed normally
• Ambiguous (2-3): Results are partially relevant, attempt refinement
• Incorrect (1): Results are irrelevant, fall back to web search or acknowledge uncertainty

Advanced Orchestrators: Multi-Step Patterns

Self-RAG (SelfRagOrchestrator)

A self-reflective loop: retrieve → generate a draft answer → evaluate whether the draft is well-grounded in the retrieved chunks → if not, refine the query and try again. This catches hallucination before it reaches the user.

Speculative RAG (SpeculativeRagOrchestrator)

A parallel drafting approach: generate multiple candidate answers from different chunk subsets simultaneously, then use an LLM to select the best one. Trades compute for quality — useful when accuracy matters more than latency.

Why a Fluent Builder?

Instead of manually wiring processors and DI registrations:

// Without builder (manual, error-prone)
services.AddSingleton<RetrievalPipeline>(sp => {
    var pipeline = new RetrievalPipeline();
    pipeline.QueryProcessors.Add(new MultiQueryExpander(sp.GetRequiredService<IChatClient>()));
    pipeline.ResultProcessors.Add(new LlmReranker(sp.GetRequiredService<IChatClient>()));
    return pipeline;
});

// With builder (discoverable, concise)
services.AddRetrievalPipeline()
    .UseQueryExpansion()
    .UseLlmReranking()
    .AsRetriever<string, Article>(
        sp => sp.GetRequiredService<VectorStoreCollection<string, Article>>(),
        record => record.Content);

The builder is discoverable via IntelliSense, handles DI resolution, and the AsRetriever() terminal registers a ready-to-inject IRetriever singleton.

Proposed Packages

Package: CommunityToolkit.DataRetrieval

LLM-powered processors + fluent builder for composing retrieval pipelines via dependency injection.

Fluent Builder API

using CommunityToolkit.DataRetrieval;

builder.Services.AddRetrievalPipeline()
    .UseQueryExpansion(o => o.VariantCount = 5)
    .UseHyDE()
    .UseLlmReranking(o => o.MaxResults = 5)
    .UseCrag()
    .AsRetriever<string, Article>(
        sp => sp.GetRequiredService<VectorStoreCollection<string, Article>>(),
        record => record.Content);

Query Processors (pre-search)

Type	Technique	What It Does
MultiQueryExpander	Multi-Query Expansion	Generates N query variants via LLM, searches with all of them, then merges results using Reciprocal Rank Fusion — chunks appearing across multiple variant results are boosted to the top
HydeQueryTransformer	Hypothetical Document Embeddings	Generates a hypothetical answer to the query, then searches using that answer's embedding — bridges the "short question ↔ long document" vector space gap
AdaptiveRouter	Adaptive Routing	Classifies query complexity (simple/thematic/entity-specific) and routes to the optimal search strategy — not all questions benefit from the same retrieval approach
TreeSearchRetriever	RAPTOR Tree Search	Top-down hierarchical traversal starting from corpus-level summaries down to leaf chunks — ideal for broad or thematic questions where you need context at multiple abstraction levels

Result Processors (post-search)

Type	Technique	What It Does
LlmReranker	LLM Reranking	Retrieves extra candidates, then asks an LLM to rank them by actual relevance (not just vector distance) — returns only the top N most relevant passages
CragValidator	Corrective RAG	Scores retrieval quality 1-5 and routes through correct/ambiguous/incorrect paths — catches cases where retrieved chunks are topically related but don't actually answer the question

Orchestrators (advanced multi-step)

Type	Technique	What It Does
SelfRagOrchestrator	Self-RAG	Self-reflective loop: retrieve → generate draft → evaluate groundedness → refine if needed — catches hallucination before it reaches the user
SpeculativeRagOrchestrator	Speculative RAG	Generates multiple candidate answers from different chunk subsets in parallel, then selects the best one — trades compute for answer quality

DI Extensions

public static class DataRetrievalServiceCollectionExtensions
{
    // No-arg: creates default pipeline, returns builder for composition
    public static RetrievalPipelineBuilder AddRetrievalPipeline(
        this IServiceCollection services);

    // Factory overload: bring your own pipeline instance
    public static RetrievalPipelineBuilder AddRetrievalPipeline(
        this IServiceCollection services,
        Func<IServiceProvider, RetrievalPipeline> pipelineFactory);
}

Builder Terminal: AsRetriever

// Registers IRetriever singleton binding pipeline → collection
builder.Services.AddRetrievalPipeline()
    .UseQueryExpansion()
    .UseLlmReranking()
    .AsRetriever<string, Article>(
        sp => sp.GetRequiredService<VectorStoreCollection<string, Article>>(),
        record => record.Content);

// Then inject IRetriever anywhere:
public class ChatService(IRetriever retriever)
{
    public async Task<string> AnswerAsync(string question)
    {
        var results = await retriever.RetrieveAsync(question, topK: 5);
        // ... use results.Chunks for grounding
    }
}

Package: CommunityToolkit.DataRetrieval.OnnxReranker

Local ONNX cross-encoder reranker for zero-cloud-dependency reranking.

using CommunityToolkit.DataRetrieval.OnnxReranker;

// Use with pipeline directly
pipeline.ResultProcessors.Add(new CrossEncoderReranker(new CrossEncoderRerankerOptions
{
    ModelPath = "models/ms-marco-MiniLM-L-6-v2.onnx",
    TokenizerPath = "models/tokenizer.json",
    MaxResults = 5
}));

// Or as IReranker for IReranker-consuming APIs
IReranker reranker = new CrossEncoderReranker(options);

Features:

• Supports ms-marco-MiniLM, BGE-reranker, and any ONNX cross-encoder model
• Text-pair tokenization → ONNX inference → sigmoid normalization
• Lazy initialization (model loaded on first use)
• IDisposable for native ONNX resource cleanup
• Implements both RetrievalResultProcessor AND IReranker (dual interface)

Dependencies: Microsoft.ML.OnnxRuntime.Managed + Microsoft.ML.Tokenizers

Relationship to Upstream

Package	Layer	Description
Microsoft.Extensions.DataRetrieval.Abstractions	Contracts	RetrievalQuery, RetrievalChunk, processor base classes, IRetriever
Microsoft.Extensions.DataRetrieval	Pipeline	RetrievalPipeline, VectorStoreRetriever, AsRetriever()
CommunityToolkit.DataRetrieval	Implementations	Concrete processors + builder + DI
CommunityToolkit.DataRetrieval.OnnxReranker	Local ML	ONNX cross-encoder reranking

This follows the same layering as document processing:

• Microsoft.Extensions.DataIngestion (abstractions in dotnet/extensions)
• CommunityToolkit.DocumentProcessing.PdfPig.* (implementations in CommunityToolkit)

Design Decisions

Decision	Rationale
Fluent builder (not manual registration)	Discoverability via IntelliSense; reduces DI boilerplate; matches ASP.NET patterns (AddAuthentication().AddJwtBearer())
All processors use IChatClient	Works with any M.E.AI provider (OpenAI, Azure, Ollama, etc.)
AsRetriever() as terminal	Natural pipeline → endpoint transition; registers IRetriever in DI for downstream injection
ONNX reranker in separate package	Heavy native dependencies (OnnxRuntime); not everyone needs local ML
Options pattern for configuration	Action<TOptions>? on each builder method; familiar .NET pattern
Orchestrators as standalone classes	Self-RAG/Speculative RAG are full orchestration patterns, not single-stage processors

Dependencies

CommunityToolkit.DataRetrieval:

• Microsoft.Extensions.DataRetrieval (≥ 10.5.0-preview)
• Microsoft.Extensions.DataRetrieval.Abstractions
• Microsoft.Extensions.AI.Abstractions
• Microsoft.Extensions.DependencyInjection.Abstractions
• Microsoft.Extensions.VectorData.Abstractions
• Microsoft.Extensions.Logging.Abstractions

CommunityToolkit.DataRetrieval.OnnxReranker:

• All of the above, plus:
• Microsoft.ML.OnnxRuntime.Managed
• Microsoft.ML.Tokenizers

Implementation

A reference implementation exists on the feature/data-retrieval branch including:

• Both packages with full XML documentation
• Fluent builder with IntelliSense-friendly method signatures
• All processors tested against real retrieval scenarios in the advanced-rag reference application