graph view

Author: KSemenenko

AGENTS.md

@@ -140,6 +140,7 @@ Rule format:
 - Core AI infrastructure should stay minimal and predefined: prefer one factory that assembles known agents and known workflows, and avoid extra catalog or registry layers for skills, agents, or workflows unless a later task explicitly requires dynamic discovery.
 - Core AI skill and article wiring should use official Microsoft Agent Framework context primitives such as `AgentSkillsProvider` and `AIContextProvider`; do not manually concatenate embedded skill markdown into an agent system prompt.
 - AI, agent context, script graph extraction, and assistant action-matching paths must not use ad-hoc language heuristics for meaning, intent, or similarity. Use an available LLM path for semantic understanding; if no LLM is available, only an explicit tokenizer/vector similarity fallback is allowed, including for action search by tokenizing action descriptions and the user query into comparable token vectors.
+- Script graph tokenizer/vector fallback must use the tokenizer/vector primitives shipped by `ManagedCode.MarkdownLd.Kb` when available; do not keep a parallel PrompterOne-owned tokenizer implementation beside the graph library.
 - AI Spotlight suggestions must behave like a compact action search, not a full tool dump: show only a few top route-relevant actions before the user types, cap search matches to the best small set, and hide the suggestion list when there are no matches.
 - Agent tool safety metadata must classify the actual operation: ordinary revision-bound script edits are mutable but not destructive because browser history/undo owns recovery; reserve `Destructive` and required approval for operations such as deletion, irreversible replacement, publishing, recording, or external open-world effects.
 - Agent/MCP tool coverage is part of the feature contract: when adding a user-visible feature, action, setting, media operation, streaming operation, or editor capability, add or update the corresponding MCP-style tool descriptor in the same task, or document explicitly why the feature is not agent-callable.
@@ -152,6 +153,7 @@ Rule format:
 - The AI spotlight is a command/search palette plus agent chat/execution surface, not a planning screen: it should suggest route-aware commands and navigation targets while typing, send the entered prompt to the agent on submit, show execution progress only as live process/status feedback, and use a macOS Spotlight-style overlay where the underlying route remains visible with only a very light dim and no dark blurred full-page backdrop.
 - The AI spotlight prompt input must receive focus immediately when the spotlight opens so users can start typing without an extra click.
 - Script graph views must prioritize a readable writer knowledge map over decorative circular placement or raw TPS/source dumps: highlight the main/root document node, group meaningful document structure, ideas, people, themes, story references, and relationships by graph value/kind, expose node and edge hover tooltips with source detail, provide zoom/fit/layout controls, avoid noisy duplicated node-list strips under the graph, and never present source lines, WPM/pace, timing, cue, archetype, or raw TPS attribute values as primary graph nodes or primary edge labels. Graph viewing must support both a full graph-only mode and a split source/graph mode with a user-draggable divider that resizes the zones. Graph extraction must be useful to scriptwriters at a high-level glance: it should show what the text is about, how blocks and ideas connect, which references or entities recur, and where the user can jump back into the source; regex, stop-word, capitalization, keyword, and hardcoded-domain semantic heuristics are strictly forbidden. The only non-LLM fallback is an explicit user-requested tokenizer/vector similarity mode based on TikToken-style token vectors and distance calculations.
+- Script graph labels, details, semantic scopes, tokenizer chunks, and knowledge-graph markdown input must use compiled TPS display text from the TPS SDK, matching the clean text shown in the prompter; raw TPS source may be used for editor ranges and metadata only, not as a post-hoc string-cleaned source for visible graph prose.
 - The AI spotlight action inventory and agent tool inventory are related but separate surfaces: Spotlight suggestions are a macOS-style UI function search over routes and app actions, while agent MCP-style tools are the callable agent contract. Keep both fed from typed descriptors, and every supported agent tool must carry stable names, descriptions, scope, input parameters, MCP safety hints (`read-only`, `idempotent`, `destructive`, `open-world`), and an explicit approval requirement when a call can mutate app or document state.
 - Assistant context must be route-aware. On editor routes, the assistant context must include the current script content, cursor location, selected lines or ranges, and document metadata so rewrite actions can target the intended selection or paragraph instead of replacing the whole script.
 - AI document-editing tools must operate through explicit range-based contracts such as line or character insert, replace, and delete operations; whole-file regeneration is forbidden for partial rewrite requests.

Directory.Packages.props

@@ -3,10 +3,10 @@
     <ManagePackageVersionsCentrally>true</ManagePackageVersionsCentrally>
   </PropertyGroup>
   <ItemGroup>
-    <PackageVersion Include="Azure.Identity" Version="1.20.0" />
+    <PackageVersion Include="Azure.Identity" Version="1.21.0" />
     <PackageVersion Include="bunit" Version="2.7.2" />
     <PackageVersion Include="LLamaSharp" Version="0.26.0" />
-    <PackageVersion Include="ManagedCode.MarkdownLd.Kb" Version="0.0.1" />
+    <PackageVersion Include="ManagedCode.MarkdownLd.Kb" Version="0.1.0" />
     <PackageVersion Include="ManagedCode.MarkItDown" Version="10.0.7" />
     <PackageVersion Include="ManagedCode.MimeTypes" Version="10.0.1" />
     <PackageVersion Include="ManagedCode.Storage.Browser" Version="10.0.5" />
@@ -28,18 +28,16 @@
     <PackageVersion Include="Microsoft.Extensions.Localization" Version="10.0.5" />
     <PackageVersion Include="Microsoft.Extensions.Logging.Abstractions" Version="10.0.5" />
     <PackageVersion Include="Microsoft.Extensions.Logging.Configuration" Version="10.0.5" />
-    <PackageVersion Include="Microsoft.ML.Tokenizers" Version="2.0.0" />
-    <PackageVersion Include="Microsoft.ML.Tokenizers.Data.O200kBase" Version="2.0.0" />
-    <PackageVersion Include="Microsoft.Playwright" Version="1.58.0" />
+    <PackageVersion Include="Microsoft.Playwright" Version="1.59.0" />
     <PackageVersion Include="ModelContextProtocol.Core" Version="1.2.0" />
     <PackageVersion Include="OllamaSharp" Version="5.4.25" />
-    <PackageVersion Include="Sentry.AspNetCore.Blazor.WebAssembly" Version="6.3.0" />
-    <PackageVersion Include="Sentry" Version="6.3.0" />
+    <PackageVersion Include="Sentry.AspNetCore.Blazor.WebAssembly" Version="6.3.1" />
+    <PackageVersion Include="Sentry" Version="6.3.1" />
     <PackageVersion Include="PDFtoImage" Version="5.2.1" />
     <PackageVersion Include="Shouldly" Version="4.3.0" />
-    <PackageVersion Include="TUnit" Version="1.28.7" />
-    <PackageVersion Include="TUnit.Assertions" Version="1.28.7" />
-    <PackageVersion Include="TUnit.Core" Version="1.28.7" />
-    <PackageVersion Include="TUnit.Playwright" Version="1.28.7" />
+    <PackageVersion Include="TUnit" Version="1.33.0" />
+    <PackageVersion Include="TUnit.Assertions" Version="1.33.0" />
+    <PackageVersion Include="TUnit.Core" Version="1.33.0" />
+    <PackageVersion Include="TUnit.Playwright" Version="1.33.0" />
   </ItemGroup>
 </Project>

src/PrompterOne.Core/AGENTS.md

@@ -43,6 +43,8 @@ It owns TPS parsing, compilation, export, RSVP helpers, preview and workspace st
 - Do not let UI-specific shortcuts leak into domain parsing or RSVP behavior.
 - AI agent infrastructure belongs under `AI/*`; keep one focused class per concrete agent type, colocate that agent's system prompt and skill references with the agent class, prefer one factory that assembles predefined agents and workflows over separate catalog layers, and wire skills or article context through official `AgentSkillsProvider` or `AIContextProvider` support instead of hand-built prompt concatenation.
 - AI script graph extraction must prioritize writer-facing knowledge from the document text: themes, terms, entities, characters, concepts, claims, recurring references, and block-to-block relationships should be explicit graph concepts, while TPS/source mechanics stay metadata unless they add semantic meaning. The graph model should help a scriptwriter understand what the script is about, how ideas and blocks connect, and where to jump back into the source.
+- AI script graph labels, descriptions, semantic scopes, tokenizer chunks, and knowledge-graph markdown input must come from compiled TPS display text through the TPS SDK so they match the clean prompter text; raw TPS source is allowed only for source ranges and structural metadata, not post-hoc string-cleaned visible prose.
 - AI script graph semantic extraction must use an available LLM/chat-client path as the primary extractor. Regex, stop-word, capitalization, keyword, or hardcoded-domain semantic heuristics are strictly forbidden; when no LLM graph extraction is configured, the app may either show no semantic graph or run only an explicit user-requested tokenizer/vector similarity fallback based on `Microsoft.ML.Tokenizers` token vectors and distance calculations.
+- Script graph tokenizer/vector fallback must use the tokenizer/vector primitives exposed by `ManagedCode.MarkdownLd.Kb` when that package provides them; do not keep a duplicate Core-owned tokenizer implementation.
 - AI and agent services must not use ad-hoc language heuristics for semantic meaning, intent matching, or action similarity. Prefer LLM extraction; the only non-LLM fallback allowed for similarity or search is explicit tokenizer/vector similarity based on `Microsoft.ML.Tokenizers`.
 - Respect root maintainability limits. Large parser/compiler edits need explicit decomposition or documented exceptions.

src/PrompterOne.Core/AI/Services/ScriptKnowledgeGraphCompiledDocument.cs

@@ -0,0 +1,107 @@
+using ManagedCode.Tps;
+using ManagedCode.Tps.Models;
+using System.Text;
+
+namespace PrompterOne.Core.AI.Services;
+
+internal sealed class ScriptKnowledgeGraphCompiledDocument
+{
+    private ScriptKnowledgeGraphCompiledDocument(
+        TpsCompilationResult compilation,
+        string displayText,
+        string displayMarkdown,
+        IReadOnlyDictionary<string, string> segmentTextById,
+        IReadOnlyDictionary<string, string> blockTextById)
+    {
+        Compilation = compilation;
+        DisplayText = displayText;
+        DisplayMarkdown = displayMarkdown;
+        SegmentTextById = segmentTextById;
+        BlockTextById = blockTextById;
+    }
+
+    public TpsCompilationResult Compilation { get; }
+
+    public string DisplayText { get; }
+
+    public string DisplayMarkdown { get; }
+
+    public IReadOnlyDictionary<string, string> SegmentTextById { get; }
+
+    public IReadOnlyDictionary<string, string> BlockTextById { get; }
+
+    public static ScriptKnowledgeGraphCompiledDocument Create(string sourceContent, string? title)
+    {
+        var compilation = TpsRuntime.Compile(sourceContent);
+        var segmentTextById = compilation.Script.Segments.ToDictionary(
+            static segment => segment.Id,
+            static segment => BuildWordText(segment.Words),
+            StringComparer.Ordinal);
+        var blockTextById = compilation.Script.Segments
+            .SelectMany(static segment => segment.Blocks)
+            .ToDictionary(
+                static block => block.Id,
+                static block => BuildWordText(block.Words),
+                StringComparer.Ordinal);
+        var displayText = BuildWordText(compilation.Script.Words);
+        var displayMarkdown = BuildDisplayMarkdown(compilation.Script, title);
+
+        return new ScriptKnowledgeGraphCompiledDocument(
+            compilation,
+            displayText,
+            displayMarkdown,
+            segmentTextById,
+            blockTextById);
+    }
+
+    public string GetSegmentText(string segmentId) =>
+        SegmentTextById.TryGetValue(segmentId, out var text) ? text : string.Empty;
+
+    public string GetBlockText(string blockId) =>
+        BlockTextById.TryGetValue(blockId, out var text) ? text : string.Empty;
+
+    private static string BuildDisplayMarkdown(CompiledScript script, string? title)
+    {
+        var builder = new StringBuilder();
+        if (!string.IsNullOrWhiteSpace(title))
+        {
+            builder.Append("# ").AppendLine(title.Trim());
+        }
+
+        foreach (var segment in script.Segments)
+        {
+            builder.Append("## ").AppendLine(segment.Name);
+            if (segment.Blocks.Count == 0)
+            {
+                AppendScopeText(builder, BuildWordText(segment.Words));
+                continue;
+            }
+
+            foreach (var block in segment.Blocks)
+            {
+                builder.Append("### ").AppendLine(block.Name);
+                AppendScopeText(builder, BuildWordText(block.Words));
+            }
+        }
+
+        return builder.ToString().Trim();
+    }
+
+    private static void AppendScopeText(StringBuilder builder, string text)
+    {
+        if (string.IsNullOrWhiteSpace(text))
+        {
+            return;
+        }
+
+        builder.AppendLine(text).AppendLine();
+    }
+
+    private static string BuildWordText(IEnumerable<CompiledWord> words) =>
+        string.Join(
+            ' ',
+            words
+                .Where(static word => word.Metadata.IsPause == false)
+                .Select(static word => word.CleanText)
+                .Where(static text => !string.IsNullOrWhiteSpace(text)));
+}

src/PrompterOne.Core/AI/Services/ScriptKnowledgeGraphDocumentBuilder.cs

@@ -12,6 +12,7 @@ public static void AddDocumentGraph(
         string containsEdgeLabel,
         ScriptKnowledgeGraphBuildRequest request,
         string content,
+        string displayContent,
         ICollection<ScriptKnowledgeGraphSemanticScope> scopes,
         IDictionary<string, ScriptKnowledgeGraphNode> nodes,
         IDictionary<string, ScriptKnowledgeGraphEdge> edges,
@@ -28,7 +29,7 @@ public static void AddDocumentGraph(
                 ["source"] = "script",
                 ["documentId"] = request.DocumentId ?? string.Empty
             });
-        scopes.Add(new ScriptKnowledgeGraphSemanticScope(documentNodeId, "Document", content));
+        scopes.Add(new ScriptKnowledgeGraphSemanticScope(documentNodeId, "Document", displayContent));
 
         AddSectionNodes(documentNodeId, containsEdgeLabel, content, nodes, edges, ranges);
     }

src/PrompterOne.Core/AI/Services/ScriptKnowledgeGraphService.cs

@@ -20,9 +20,13 @@ public async Task<ScriptKnowledgeGraphArtifact> BuildAsync(
         ArgumentNullException.ThrowIfNull(request);
 
         var content = request.Content ?? string.Empty;
+        var compiledDocument = ScriptKnowledgeGraphCompiledDocument.Create(content, request.Title);
         var pipeline = new MarkdownKnowledgePipeline();
         var kbResult = await pipeline
-            .BuildFromMarkdownAsync(content, CreateSourcePath(request.DocumentId), cancellationToken: cancellationToken)
+            .BuildFromMarkdownAsync(
+                compiledDocument.DisplayMarkdown,
+                CreateSourcePath(request.DocumentId),
+                cancellationToken: cancellationToken)
             .ConfigureAwait(false);
 
         var nodes = new Dictionary<string, ScriptKnowledgeGraphNode>(StringComparer.Ordinal);
@@ -35,6 +39,7 @@ public async Task<ScriptKnowledgeGraphArtifact> BuildAsync(
             ContainsEdgeLabel,
             request,
             content,
+            compiledDocument.DisplayText,
             semanticScopes,
             nodes,
             edges,
@@ -43,16 +48,32 @@ public async Task<ScriptKnowledgeGraphArtifact> BuildAsync(
             DocumentNodeId,
             ContainsEdgeLabel,
             content,
+            compiledDocument,
             semanticScopes,
             nodes,
             edges,
             ranges);
         AddKnowledgeBankGraph(kbResult.Graph.ToSnapshot(), content, nodes, edges, ranges);
-        var semanticStatus = await TryAddModelSemanticGraphAsync(request, content, semanticScopes, nodes, edges, ranges, cancellationToken)
+        var semanticStatus = await TryAddModelSemanticGraphAsync(
+                request,
+                compiledDocument.DisplayMarkdown,
+                semanticScopes,
+                nodes,
+                edges,
+                ranges,
+                cancellationToken)
             .ConfigureAwait(false);
         if (semanticStatus != ScriptKnowledgeGraphSemanticStatus.Model &&
             request.SemanticMode == ScriptKnowledgeGraphSemanticMode.TokenizerSimilarity &&
-            _tokenizerSimilarityExtractor.AddTokenizerSimilarity(content, semanticScopes, nodes, edges, ranges))
+            await _tokenizerSimilarityExtractor
+                .AddTokenizerSimilarityAsync(
+                    content,
+                    compiledDocument.DisplayMarkdown,
+                    nodes,
+                    edges,
+                    ranges,
+                    cancellationToken)
+                .ConfigureAwait(false))
         {
             semanticStatus = ScriptKnowledgeGraphSemanticStatus.TokenizerSimilarity;
         }