Skip to content

feat: add native and automatic semantic updates #2941

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nyldn wants to merge 3 commits into
base: v8
Choose a base branch
from
Open

feat: add native and automatic semantic updates #2941

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
4d90975
feat: add native semantic update pipeline
nyldn Aug 21, 2026
6f74478
docs(skill): reuse native semantic update conductor
nyldn Aug 21, 2026
b413b19
feat(hooks): automate opted-in semantic updates
nyldn Aug 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 28 additions & 3 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -390,7 +390,7 @@ graphify export callflow-html # Mermaid architecture/call-flow HTML (auto-r
/graphify add https://arxiv.org/abs/1706.03762 # fetch a paper and add it
/graphify add <youtube-url> # transcribe and add a video

graphify hook install # auto-rebuild on git commit
graphify hook install # auto-rebuild on git commit (AST; optional semantics)
graphify merge-graphs a.json b.json # combine two graphs

graphify prs # PR dashboard: CI state, review status, worktree mapping
Expand Down Expand Up @@ -440,8 +440,24 @@ graphify-out/cost.json # local only
**Workflow:**
1. One person runs `/graphify .` and commits `graphify-out/`.
2. Everyone pulls — their assistant reads the graph immediately.
3. Run `graphify hook install` to auto-rebuild after each commit (AST only, no API cost). This also sets up a git merge driver so `graph.json` is never left with conflict markers — two devs committing in parallel get their graphs union-merged automatically.
4. When docs or papers change, run `/graphify --update` to refresh those nodes.
3. Run `graphify hook install` to auto-rebuild after each commit (AST only, no API cost by default). This also sets up a git merge driver so `graph.json` is never left with conflict markers — two devs committing in parallel get their graphs union-merged automatically.
4. To keep documents, papers, and images current without a manual step, opt the project into semantic updates:

```ini
# .graphifyrc
semantic_update=on_commit
semantic_backend=kimi
semantic_env_file=.env.local
# semantic_model=kimi-k2.6 # optional
# semantic_google_workspace=true # optional
```

The semantic worker runs detached after a commit, shares the graph rebuild lock,
and coalesces overlapping commits. Failed or interrupted work stays queued for a
later commit. The env file is parsed as data, never sourced as shell code; only
credential variables for the selected backend are loaded, existing process
credentials win, and endpoint variables are ignored. Keep that file untracked.
Without `semantic_update=on_commit`, hooks remain AST-only and incur no LLM cost.

---

Expand Down Expand Up @@ -778,6 +794,7 @@ graphify --version # print installed version
graphify watch ./src
graphify check-update ./src
graphify update ./src
graphify update ./my-workspace --semantic --backend kimi # refresh changed code + documents, then regenerate report/community outputs
graphify update ./src --no-cluster # skip reclustering, write raw AST graph only
graphify update ./src --force # overwrite even if new graph has fewer nodes
graphify cluster-only ./my-project
Expand All @@ -792,6 +809,14 @@ graphify label ./my-project # (re)name commun
graphify label ./my-project --backend=openai --model gpt-4o # force a specific backend and model
```

Use `graphify update <path> --semantic` for a mixed code/document corpus when
you want one incremental command to finish the whole pipeline. It reuses the
semantic cache, prunes replaced or deleted sources, and regenerates
`graph.json`, community labels, and `GRAPH_REPORT.md`. Changed documents use
the selected LLM backend and may incur API cost; plain `graphify update` stays
AST-only and makes no LLM calls. `--semantic` cannot be combined with
`--code-only` or `--no-cluster`.

> **Community names:** inside an agent (Claude Code, Gemini CLI) the agent names communities itself. When you run the bare CLI, `cluster-only` auto-names them with the configured backend (built-in or custom OpenAI-compatible provider) — pass `--no-label` to keep `Community N`, or run `graphify label` to (re)generate names on demand.

---
Expand Down
3 changes: 3 additions & 0 deletions graphify/__main__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -535,6 +535,9 @@ def _run_cli() -> None:
print(" --dir <path> target directory (default: ./raw)")
print(" watch <path> watch a folder and rebuild the graph on code changes")
print(" update <path> re-extract code files and update the graph (no LLM needed)")
print(" --semantic also refresh changed documents with an LLM, then regenerate the report")
print(" --backend=<name> semantic extraction and community-labeling backend")
print(" --model=<name> model for semantic extraction and community labeling")
print(" --force overwrite graph.json even if the rebuild has fewer nodes")
print(" (also: GRAPHIFY_FORCE=1 env var; use after refactors that delete code)")
print(" --no-cluster skip clustering, write raw extraction only")
Expand Down
162 changes: 154 additions & 8 deletions graphify/cli.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -829,7 +829,138 @@ def _reenter_main() -> None:
main()


def _semantic_update_extract_argv(update_argv: list[str]) -> list[str]:
"""Normalize update's flag-anywhere syntax for the extract parser."""
value_options = {
"--backend",
"--model",
"--mode",
"--out",
"--output",
"--as",
"--max-workers",
"--token-budget",
"--max-concurrency",
"--api-timeout",
"--resolution",
"--exclude-hubs",
"--exclude",
"--postgres",
"--batch-size",
"--min-community-size",
}
flag_options = {
"--dedup-llm",
"--no-dedup",
"--google-workspace",
"--no-gitignore",
"--global",
"--cargo",
"--force",
"--allow-partial",
"--timing",
"--no-viz",
"--no-label",
"--missing-only",
}
options: list[str] = []
paths: list[str] = []
args = update_argv[2:]
i = 0
while i < len(args):
arg = args[i]
if arg == "--semantic":
i += 1
elif arg in value_options:
if i + 1 >= len(args):
raise ValueError(f"semantic update option requires a value: {arg}")
options.extend((arg, args[i + 1]))
i += 2
elif arg in flag_options or any(
arg.startswith(f"{option}=") for option in value_options
):
options.append(arg)
i += 1
elif arg.startswith("-"):
raise ValueError(f"unknown semantic update option: {arg}")
else:
paths.append(arg)
i += 1

if len(paths) > 1:
raise ValueError("update accepts at most one path argument")
if paths:
update_root = paths[0]
else:
saved_root = Path(_GRAPHIFY_OUT) / ".graphify_root"
update_root = (
saved_root.read_text(encoding="utf-8").strip()
if saved_root.exists()
else "."
)
return [update_argv[0], "extract", update_root, *options]


def _semantic_update_cluster_argv(extract_argv: list[str], out_root: Path) -> list[str]:
"""Build the report-stage argv for a semantic update.

Extraction and ``cluster-only`` intentionally have separate parsers. Keep
only their shared/report-specific options here so extraction-only flags do
not leak into the final stage.
"""
value_options = {
"--backend",
"--model",
"--resolution",
"--exclude-hubs",
"--max-concurrency",
"--batch-size",
"--min-community-size",
}
flag_options = {"--no-viz", "--no-label", "--missing-only", "--timing"}
cluster_args: list[str] = []
args = extract_argv[2:]
i = 0
while i < len(args):
arg = args[i]
if arg in value_options and i + 1 < len(args):
cluster_args.extend((arg, args[i + 1]))
i += 2
elif arg in flag_options or any(
arg.startswith(f"{option}=") for option in value_options
):
cluster_args.append(arg)
i += 1
else:
i += 1
return [extract_argv[0], "cluster-only", str(out_root), *cluster_args]


def dispatch_command(cmd: str) -> None:
semantic_update = cmd == "update" and "--semantic" in sys.argv[2:]
if semantic_update:
incompatible = next(
(flag for flag in ("--no-cluster", "--code-only") if flag in sys.argv[2:]),
None,
)
if incompatible:
print(
f"error: --semantic and {incompatible} cannot be combined; "
"a semantic update refreshes documents and completes the report "
"and community outputs",
file=sys.stderr,
)
sys.exit(2)
# Reuse the native incremental extractor instead of maintaining a
# second mixed-corpus update pipeline. The report stage is completed at
# the end of the extract branch below.
try:
sys.argv = _semantic_update_extract_argv(sys.argv)
except ValueError as exc:
print(f"error: {exc}", file=sys.stderr)
sys.exit(2)
cmd = "extract"

if cmd == "provider":
from graphify.llm import _custom_providers_path, BACKENDS
import json as _json
Expand Down Expand Up @@ -4121,14 +4252,29 @@ def _invalidate_file_manifest_for_db_graph() -> None:
f"{merged['output_tokens']:,} out, "
f"est. cost (~{backend}): ${cost:.4f}"
)
# extract intentionally stops at graph.json + analysis; the report and
# community labels are produced by `cluster-only` (or an agent's Step 5).
# Point standalone users at it so communities get named (#1097).
print(
"[graphify extract] next: run "
f"`graphify cluster-only {graphify_out.parent}` "
"to generate GRAPH_REPORT.md and name communities"
)
if semantic_update:
# Complete the native update in-process so extraction monkeypatches,
# custom providers, and exit handling stay consistent with the
# command that produced graph.json.
extract_argv = sys.argv
try:
sys.argv = _semantic_update_cluster_argv(extract_argv, out_root)
dispatch_command("cluster-only")
finally:
sys.argv = extract_argv
print(
"[graphify update] semantic update complete: "
"graph, communities, and report are current"
)
else:
# extract intentionally stops at graph.json + analysis; the report
# and community labels are produced by `cluster-only` (or an agent's
# Step 5). Point standalone users at it so communities get named.
print(
"[graphify extract] next: run "
f"`graphify cluster-only {graphify_out.parent}` "
"to generate GRAPH_REPORT.md and name communities"
)
stages.total()

elif cmd == "cache-check":
Expand Down
Loading
Loading