Log-template mining (Drain-style clustering)

[Hirogen]

This is an draft Idea for a feature based on the now removed patternwindow. The idea of this dialog was a statistical analyses of a given logfile, but this idea was never full developed. I dont want to loose the idea, so this issue should cover it.

Background

The abandoned Pattern analysis feature (removed in cleanup/remove-pattern-analysis) tried to answer a real question: "which multi-line event sequences repeat in this log, how often, and where?" Its approach — a per-line character-sum hash plus Levenshtein comparison, O(N²) over the file — could never work on the large files where the answer matters.

The modern equivalent is log-template mining: cluster log lines into templates by treating the stable tokens as the template and the variable tokens (timestamps, IDs, paths, numbers) as parameters. The reference algorithm is Drain (He et al., ICWS 2017), a fixed-depth parse tree
that clusters lines online in roughly O(N · depth) — a single pass, suitable for exactly the large files LogExpert targets.

Proposed feature

A "Templates" tool window (or tab) that, on demand, mines the current log and shows:

• One row per discovered template, e.g. Connection to <*> lost after <*> ms, with an occurrence count and first/last line number
• Selecting a template lists its matching lines in a second grid; double-click jumps the main log view to that line (same navigation contract the old Pattern window intended)
• Sort by count to surface noisy templates; rare templates (count 1–2) are often the anomalies worth reading first

Possible follow-ups (out of scope for the first cut)

• Filter integration: "hide/show all lines matching this template"
• Diff two logs by template histogram (e.g. before/after a deploy)
• Multi-line block detection on top of the template stream (the old feature's original goal): recurring sequences of template IDs can be found cheaply once lines are template-classified

Design notes / constraints

• Algorithm: Drain (fixed-depth prefix tree keyed on token count + leading tokens, similarity threshold on token overlap). Simple enough to implement in-repo; no external service. A port is small (~300 lines); check licenses before borrowing code from Drain3 (MIT) or logparser (MIT).
• Preprocessing: mask obvious variables before clustering (numbers, hex, GUIDs, IPs) — Drain3's masking-regex approach. Reuse the columnizer: mine the message column only, not timestamps.
• Performance target: single pass over the file via LogfileReader, streaming, cancellable via the existing cancel-handler registry (same pattern as filter runs), progress bar updates. Memory bounded by template count, not line count; cap tree width (maxChildren) as Drain does.
• Placement: engine in LogExpert.Core (testable, no WinForms deps — follow the filter-engine extraction precedent from PR refactor: extract Filter execution into a pure Core module (Filter Engine) #653), UI in LogExpert.UI. Entry point: context/menu item in LogTabWindow — unlike the old feature, wire the menu item from day one.
• Testing: engine gets unit tests with known corpora (loghub samples or synthetic); assert template extraction and counts. No UI-driven logic in the engine.

Acceptance criteria (first cut)

• Menu entry opens the Templates view for the active log window
• Mining a 1M-line file completes in seconds, streams progress, and is cancellable with ESC
• Templates show masked parameters as <*>, with occurrence counts and line navigation
• Engine lives in LogExpert.Core with unit tests; UI project contains no clustering logic
• Localized UI strings (en/de/zh-CN) from the start

References

• Drain paper: He, Zhu, Zheng, Lyu — "Drain: An Online Log Parsing Approach with Fixed Depth Tree" (ICWS 2017)
• Drain3 (maintained Python implementation, MIT): https://github.com/logpai/Drain3
• logparser benchmark suite: https://github.com/logpai/logparser