Add VisionLMOcrClient: an IOcrClient backed by a vision-capable IChatClient

[luisquintanilla]

Summary

Add a VisionLMOcrClient — a provider-neutral IOcrClient implementation backed by any
Microsoft.Extensions.AI.IChatClient that supports image input (a vision LM). It turns "give me an
IChatClient that can see" into a first-class OCR provider that any MEDI reader (e.g. the PdfPig
reader) can compose, alongside dedicated OCR services (Mistral Document AI, Azure Document
Intelligence).

This is the vision-LM path from the current PdfPig VisionOcrEnricher, relocated and reshaped:
instead of a downstream document processor that mutates placeholder elements, it becomes an
IOcrClient provider the reader calls inline. Same mechanism (prompt + page image → text), correct
layer.

Why (motivation)

• A vision LM is an OCR engine — but today the only way to use one for OCR in MEDI is the PdfPig
  VisionOcrEnricher, which is coupled to the reader's placeholder/page_image stash and lives at
  the wrong layer (post-read processor, not an OCR provider).
• The IOcrClient abstraction (dotnet/extensions #7588) is the provider-neutral seam for
  "document → structured text". A vision LM should be one provider behind that seam, swappable
  with Mistral OCR / Azure DI at composition time — no reader changes.
• Keeping the concrete impl in CommunityToolkit (not dotnet/extensions) respects the layering rule:
  extensions holds only the abstraction; concrete providers live in the toolkit, none
  privileged.

Package

Proposed: CommunityToolkit.DataIngestion.VisionLMOcr (name open to discussion — the point is a
satellite package whose only extra dependency footprint is IChatClient; it does not force a
vision dependency onto the base ingestion package).

Dependencies: Microsoft.Extensions.AI.Abstractions + the IOcrClient/OcrResult abstractions
only. No provider SDKs. Neutral by construction.

Targets: net10.0;net8.0 (match sibling packages).

The IOcrClient contract to implement

Note: IOcrClient and its result types are currently unpublished — they live in
dotnet/extensions PR #7588. Until that merges, vendor the shape below into this package
(internal copy). When #7588 ships, delete the vendored copy and reference the real
Microsoft.Extensions.DataIngestion.Abstractions types. This is a tracked, deliberate temporary
duplication.

public interface IOcrClient
{
    Task<OcrResult> GetTextAsync(
        Stream document,
        string mediaType,
        OcrOptions? options = null,
        IProgress<OcrProgress>? progress = null,
        CancellationToken cancellationToken = default);

    object? GetService(Type serviceType, object? serviceKey = null);
}

public sealed class OcrResult(IReadOnlyList<OcrPage> pages)
{
    public IReadOnlyList<OcrPage> Pages { get; } = pages;
    public string Markdown => string.Join("\n\n", Pages.Select(p => p.Markdown));
    public string? OcrSource { get; init; }
    public string? ModelId { get; init; }
    public OcrUsage? Usage { get; init; }
    // + RawRepresentation, AdditionalProperties
}

public sealed class OcrPage(int index, string markdown)
{
    public int Index { get; }
    public string Markdown { get; }
    public IReadOnlyList<OcrBlock> Blocks { get; init; } = [];
    public IReadOnlyList<OcrTable> Tables { get; init; } = [];
    public double? Confidence { get; init; }
}
// OcrTable { RowCount, ColumnCount, OcrTableCell[,]? Cells, string MarkdownRepresentation, ToMarkdown() }
// OcrBlock { Text, Kind?, OcrBoundingRegion?, Confidence? }
// OcrBoundingRegion { PageNumber, Polygon, FromRectangle(...), GetBounds() }
// OcrOptions { ModelId?, IncludeImages, AdditionalProperties? }
// OcrProgress { PagesProcessed?, TotalPages?, Status? }
// OcrUsage { PagesProcessed?, AdditionalProperties? }

Behavior spec

VisionLMOcrClient(IChatClient chatClient, VisionLMOcrOptions? options = null) : IOcrClient

1. Input. GetTextAsync(document, mediaType, ...) receives a document stream + media type.
   • If mediaType is an image (image/png, image/jpeg, …): send the bytes directly as one page.
   • If it's a rendered-page image handed in by a reader (the common PdfPig FallbackForEmptyPages
     case): one image → one OcrPage.
   • (PDF rasterization is out of scope — the reader renders pages to images and calls this per
     page. This client OCRs images.)
2. Prompt. Reuse the proven VisionOcrEnricher prompt:
   • System: "You are a precise OCR engine. Extract all visible text from the provided image
     exactly as it appears. Preserve line breaks and formatting. Output only the extracted text, no
     commentary."
   • User: DataContent(imageBytes, mediaType) + TextContent("Extract all text from this image.")
3. Call chatClient.GetResponseAsync(messages, ct); map response.Text → OcrPage.Markdown.
4. Result. Build OcrResult with OcrSource = "vision_lm", ModelId from options/response.
   Populate OcrPage.Tables only if the model returns a table representation (optional; leave
   empty otherwise — do not invent a separate table pass; tables at the OCR layer come from the
   provider, and a chat LM generally returns them inline in markdown).
5. Progress. Emit OcrProgress per page when IProgress<OcrProgress> is supplied.
6. GetService. Return this for IOcrClient/VisionLMOcrClient; delegate to the inner
   IChatClient.GetService otherwise.

Options

public sealed class VisionLMOcrOptions
{
    public string? ModelId { get; set; }
    public string? SystemPrompt { get; set; }   // default = the OCR prompt above
    public string? UserPrompt { get; set; }      // default = "Extract all text from this image."
}

DI extension

services.AddVisionLMOcrClient();                 // resolves IChatClient from DI
services.AddVisionLMOcrClient(o => o.ModelId = "gpt-4o");

Register as IOcrClient so readers pick it up by composition.

Tests (unit, no network)

Use a fake IChatClient (returns canned ChatResponse):

• Image input → one OcrPage whose Markdown == the fake's returned text; OcrSource == "vision_lm".
• The fake receives a DataContent with the right media type + the OCR system prompt.
• Custom SystemPrompt/UserPrompt/ModelId from options are threaded through.
• IProgress<OcrProgress> fires once per page.
• GetService(typeof(IOcrClient)) returns the client; unknown types delegate to inner chat client.
• Build + tests green on net8.0 and net10.0.

Acceptance criteria

• New package CommunityToolkit.DataIngestion.VisionLMOcr (or agreed name), net10.0;net8.0.
• VisionLMOcrClient : IOcrClient over IChatClient, prompt/behavior per spec.
• VisionLMOcrOptions + AddVisionLMOcrClient(...) DI extension.
• Vendored IOcrClient shape (internal) with a clear TODO to un-vendor when #7588 merges.
• Unit tests above, green net8/net10.
• No provider SDK dependencies — IChatClient + OCR abstractions only.

Dependency / sequencing note

• Gated on dotnet/extensions #7588 for the final form (real IOcrClient). Ship now with the
  vendored shape; swap when #7588 lands.
• Sibling PR replatforms the PdfPig reader onto IOcrClient (PdfPigOcrReader + OcrPolicy) and
  removes the old VisionOcrEnricher; this client is that vision path's new home.

Reference: the logic being relocated

The current VisionOcrEnricher (PdfPig reader PR) already implements the vision-LM OCR call — prompt,
DataContent image message, GetResponseAsync, text extraction. This issue moves that mechanism
behind IOcrClient and drops the placeholder/page_image-stash coupling. A junior dev can port the
message-construction + response-mapping code almost verbatim.