feat: web domain allowlist and markdown report rendering (#15)

* feat(wire): add ToolExecutionStarted and ToolOutputPart events for live tool feedback

Introduces two new wire protocol events so the UI can distinguish the approval/hooks
phase from actual tool execution, and stream incremental shell output before the final
ToolResult arrives.

- ToolExecutionStarted: emitted once per tool call after approval completes, before
  the tool body runs; file/MCP/external tools emit from the approval path while Shell
  and RunAgents emit it themselves at the right moment
- ToolOutputPart: streamed from the Shell tool as stdout/stderr lines arrive

TUI blocks now hold the execution-started spinner until ToolExecutionStarted lands
(showing a calm "preparing" row before that), render streamed output as a live tail
preview, and the Bash card shows "running" status when partial output is present.
The composing _ContentBlock also gains a live Markdown preview while the model writes.

* fix(agents): guard against out-of-policy lint findings in explore/plan agents; pad markdown code blocks

explore.yaml: require supplemental ruff checks (e.g. --select C901) to be
labeled "outside project lint policy" so callers can distinguish enforced
violations from advisory findings.

plan.yaml: add context-gate rule to verify any lint/complexity finding is in
the project's active select list before proposing a refactor — prevents plans
driven by rules the project intentionally does not enforce.

markdown.py: yield a blank_row() above and below each bordered code block so
panels read as distinct sections rather than crowding surrounding prose.
Test added to assert the blank-row framing is present.

* docs: add agent live tool stream design spec

* docs: add agent live tool stream implementation plan

* feat(blocks): add subagent output tracking state and methods to _ToolCallBlock

Add three new state fields (_subagent_output_parts, _subagent_output_had_stderr,
_subagent_execution_started), two new public methods (mark_sub_execution_started,
append_sub_output_part), and cleanup in finish_sub_tool_call for the new fields.
Also update append_sub_tool_call and append_sub_tool_call_part to recompose after
mutations, readying the block for live subagent tool streaming in Task 2.

* test: sync default agent-spec snapshot with explore.yaml lint policy

The committed explore.yaml carries lint/complexity-policy guidance (do not
flag rules such as C901 that are absent from the project's configured rule
set) that its inline snapshot in test_agent_spec.py never received, so the
test failed on a clean checkout. Regenerate the snapshot to match HEAD.

* fix(tui): refine markdown report rendering

- Frame fenced code blocks with a blank row above and below so they read as
  a distinct section instead of crowding surrounding prose.
- Add a conservative pre-parse normalizer that repairs malformed GFM tables
  the model sometimes emits (header glued to prose, a blank line before the
  |---| delimiter, data rows crammed onto the delimiter line) which
  markdown-it would otherwise render as raw text. Anchored on the delimiter
  row, it rebuilds only regions whose header and data cell counts validate,
  preserves column alignment, and leaves ambiguous or fenced content
  untouched.

* feat(agent): guide output formatting and anchor date awareness

- Add an Output Formatting section to the default system prompt: emit
  well-formed Markdown tables, reserve code fences for actual code (never
  wrap prose reports, finding lists, or ASCII boxes in a fence), and use
  status icons sparingly. This stops the model from producing the malformed
  tables and emoji-laden boxes that rendered poorly in the TUI.
- Reword the Date and Time section so the injected ${PYTHINKER_NOW} is
  framed as the authoritative present, anchoring the agent's sense of 'now',
  recency, and 'latest' to it instead of a training-era year.
- Refresh the prompt snapshot and add coverage for both.

* refactor(blocks): align subagent output methods with peer guards, add cap test

* feat(web): add fetch/search domain allowlist

* test(web): add domain-allowlist fetch tests and tool-description/config docs

* fix(typos): unparseable -> unparsable in tasks/todo.md to satisfy spell-check gate

* fix: address CodeRabbit review findings on web allowlist and markdown

- config.py: reject `allowed_domains` entries that are dots-only (would
  normalize to empty → silently unrestricted) and any entry containing
  whitespace (newlines previously slipped past the space/tab check).
- tools/web/_allowlist.py: strip trailing dots when normalizing entries so
  `example.com.` matches `example.com` hosts.
- tools/web/search.py + ui/shell/tool_renderers/web.py: emit a structured
  `returned_results=0` signal on the all-filtered path and have the search
  renderer prefer it, so an all-filtered result reports "0 results" instead of
  misreading the prose notice as one result.
- ui/shell/components/markdown.py: prefix rebuilt table rows with the captured
  delimiter-line indent so normalization never promotes an indented table to
  top level (defensive; the guard already bails on non-empty indent today).
- agents/default/system.md: reword the code-fence guidance to use inline code
  spans for language names (markdownlint MD038).
- Tests: reject `.`/whitespace allowlist entries, trailing-dot entry matching,
  and an all-results-filtered renderer regression; refresh the default-agent
  system-prompt snapshot.

Author: elkaix

docs/en/configuration/config-files.md

@@ -85,6 +85,9 @@ api_key = "sk-xxx"
 base_url = "https://api.pythinker.com/coding/v1/fetch"
 api_key = "sk-xxx"
 
+[web]
+allowed_domains = ["example.com", "docs.python.org"]
+
 [mcp.client]
 tool_call_timeout_ms = 60000
 ```
@@ -199,6 +202,16 @@ Configures web fetch service. When enabled, the `FetchURL` tool prioritizes usin
 When configuring the Pythinker platform using the `/login` command, search and fetch services are automatically configured.
 :::
 
+### `web`
+
+`web` configures policy shared by the `FetchURL` and `SearchWeb` tools.
+
+| Field | Type | Default | Description |
+| --- | --- | --- | --- |
+| `allowed_domains` | `array<string>` | _unset_ | When set, web fetch and search may only touch these domains and their subdomains. `FetchURL` rejects URLs on other hosts before making any request — including redirect targets, which are re-validated on every hop — and `SearchWeb` drops results from other domains. Unset or empty means unrestricted. Entries must be bare hostnames (e.g. `example.com`), not URLs, paths, or `host:port`. |
+
+This is a coarse governance control layered on top of the existing SSRF protections (which always block private, loopback, link-local, multicast, and reserved addresses); it does not replace them. Matching is label-aware: `example.com` matches `example.com` and `docs.example.com`, but not `notexample.com`.
+
 ### `mcp`
 
 `mcp` configures MCP client behavior.

src/pythinker_code/agents/default/system.md

@@ -188,7 +188,7 @@ The operating environment is not in a sandbox. Any actions you do will immediate
 
 ## Date and Time
 
-The current date and time in ISO format is `${PYTHINKER_NOW}`. This is only a reference for you when searching the web, or checking file modification time, etc. If you need the exact time, use Shell tool with proper command.
+The current date and time in ISO format is `${PYTHINKER_NOW}`. Treat this as the authoritative present — it reflects the real "now", which is later than your training data suggests. Anchor all reasoning about the current date, the year, recency, and what counts as the "latest" version or release to `${PYTHINKER_NOW}`; do not fall back on an earlier year you might assume from training. Use it as your reference when searching the web or checking file modification times. If you need the exact time, use the Shell tool with a proper command.
 
 ## Working Directory
 
@@ -261,6 +261,15 @@ Identify the skills that are likely to be useful for the tasks you are currently
 
 Only read skill details when needed to conserve the context window.
 
+# Output Formatting
+
+Your responses are rendered as Markdown in a terminal. Emit well-formed Markdown so it renders cleanly:
+
+- **Tables:** put the header row on its own line, the `|---|---|` delimiter row on the immediately following line (no blank line between them), and one row per line. Never glue a table onto adjacent prose (e.g. `Findings| Col |`) and never cram multiple rows onto one line. Leave a blank line before and after the table.
+- Prefer a short bullet list over a table when there are only a few items or any cell is long; reserve tables for genuinely tabular data with short cells.
+- **Code fences are for code only.** Use triple-backtick blocks tagged with a language (for example, `python` or `toml`) solely for source, config, or commands — one snippet per block. Never wrap a prose report, finding list, checklist, or ASCII box in a fence to align or frame it; write it as normal Markdown (headings, bullets, tables) so it renders cleanly.
+- **Status icons sparingly.** A check/cross/dot can mark a single headline result, but do not prefix every line with one. Use plain words for severity and outcomes (e.g. `High`, `PASS`, `0 findings`). The terminal renders icons as calm monochrome glyphs only outside code fences — another reason not to box reports.
+
 # Ultimate Reminders
 
 At any time, you should be HELPFUL, CONCISE, and ACCURATE. Be thorough in your actions — test what you build, verify what you change — not in your explanations.

src/pythinker_code/config.py

@@ -14,6 +14,7 @@
     SecretStr,
     ValidationError,
     field_serializer,
+    field_validator,
     model_validator,
 )
 from tomlkit.exceptions import TOMLKitError
@@ -217,6 +218,41 @@ class Services(BaseModel):
     """Pythinker AI Fetch configuration."""
 
 
+class WebConfig(BaseModel):
+    """Web fetch/search policy."""
+
+    allowed_domains: list[str] | None = Field(
+        default=None,
+        description=(
+            "If set, web fetch and search may only touch these domains and their "
+            "subdomains. None or empty means unrestricted (default)."
+        ),
+    )
+
+    @field_validator("allowed_domains")
+    @classmethod
+    def _validate_allowed_domains(cls, value: list[str] | None) -> list[str] | None:
+        for entry in value or []:
+            cleaned = entry.strip()
+            if not cleaned:
+                raise ValueError(
+                    "Invalid allowed_domains entry: empty or whitespace-only hostname. "
+                    "Remove it, or omit allowed_domains entirely to leave web access "
+                    "unrestricted."
+                )
+            if cleaned.strip(".") == "":
+                raise ValueError(
+                    f"Invalid allowed_domains entry {entry!r}: hostname must contain "
+                    "domain labels, not only dots."
+                )
+            if any(char.isspace() for char in cleaned) or any(char in cleaned for char in "/:"):
+                raise ValueError(
+                    f"Invalid allowed_domains entry {entry!r}: use a bare hostname "
+                    "like 'example.com', not a URL, path, or host:port."
+                )
+        return value
+
+
 class FeedbackConfig(BaseModel):
     """User-submitted feedback endpoint configuration."""
 
@@ -364,6 +400,7 @@ class Config(BaseModel):
     )
     services: Services = Field(default_factory=Services, description="Services configuration")
     memory: MemoryConfig = Field(default_factory=MemoryConfig, description="Memory configuration")
+    web: WebConfig = Field(default_factory=WebConfig, description="Web fetch/search policy")
     feedback: FeedbackConfig = Field(
         default_factory=FeedbackConfig,
         description="User-submitted feedback endpoint configuration",

src/pythinker_code/tools/web/_allowlist.py

@@ -0,0 +1,27 @@
+"""Domain allowlist matching shared by the web fetch and search tools."""
+
+from __future__ import annotations
+
+
+def _normalize(entry: str) -> str:
+    return entry.strip().strip(".").lower()
+
+
+def host_in_allowlist(host: str | None, allowed: list[str] | None) -> bool:
+    """Return whether *host* is permitted by the *allowed* domain list.
+
+    A ``None`` or empty allowlist imposes no restriction (returns ``True``),
+    preserving unconfigured behavior. Otherwise a host matches when it equals an
+    allowlist entry or is a subdomain of one. Matching is label-aware and
+    case-insensitive: ``example.com`` matches ``example.com`` and
+    ``docs.example.com`` but not ``notexample.com``.
+    """
+    entries = [normalized for entry in (allowed or []) if (normalized := _normalize(entry))]
+    if not entries:
+        return True
+
+    host = (host or "").strip().rstrip(".").lower()
+    if not host:
+        return False
+
+    return any(host == entry or host.endswith(f".{entry}") for entry in entries)

src/pythinker_code/tools/web/fetch.md

@@ -1 +1 @@
-Fetch a web page from a URL and extract main text content from it.
+Fetch a web page from a URL and extract main text content from it. Requests may be restricted to a configured set of allowed domains; fetching a disallowed host (including via a redirect) returns an error.

src/pythinker_code/tools/web/fetch.py

@@ -15,6 +15,7 @@
 from pythinker_code.soul.agent import Runtime
 from pythinker_code.soul.toolset import get_current_tool_call_or_none
 from pythinker_code.tools.utils import ToolResultBuilder, load_desc
+from pythinker_code.tools.web._allowlist import host_in_allowlist
 from pythinker_code.utils.aiohttp import new_client_session
 from pythinker_code.utils.logging import logger
 
@@ -23,13 +24,16 @@
 _REDIRECT_STATUSES = frozenset({301, 302, 303, 307, 308})
 
 
-def _validate_fetch_url(url: str) -> str | None:
+def _validate_fetch_url(url: str, allowed_domains: list[str] | None = None) -> str | None:
     parsed = urlparse(url)
     if parsed.scheme not in {"http", "https"}:
         return "Only http and https URLs are supported."
     if not parsed.hostname:
         return "URL must include a host."
 
+    if not host_in_allowlist(parsed.hostname, allowed_domains):
+        return "URL host is not in the configured web allowlist."
+
     try:
         infos = socket.getaddrinfo(parsed.hostname, parsed.port, type=socket.SOCK_STREAM)
     except socket.gaierror:
@@ -76,23 +80,29 @@ def __init__(self, reason: str) -> None:
 
 
 async def _get_revalidating_redirects(
-    session: aiohttp.ClientSession, url: str, headers: dict[str, str]
+    session: aiohttp.ClientSession,
+    url: str,
+    headers: dict[str, str],
+    allowed_domains: list[str] | None = None,
 ) -> aiohttp.ClientResponse:
     """GET ``url``, following redirects manually and re-validating every hop.
 
     aiohttp follows redirects internally without re-checking the destination, so
     a public URL that 30x-redirects to a private/link-local address (e.g. a cloud
     metadata endpoint at 169.254.169.254) would otherwise sail past
     ``_validate_fetch_url``. We disable automatic redirects and validate each
-    ``Location`` before following it.
+    ``Location`` against both the SSRF guard and the configured domain allowlist
+    before following it.
 
     Returns the final, open, non-redirect response (the caller owns closing it).
     Raises ``_FetchBlocked`` if any hop is blocked or the redirect limit is hit.
     """
     current = url
-    for _ in range(MAX_FETCH_REDIRECTS + 1):
-        if reason := _validate_fetch_url(current):
-            raise _FetchBlocked(reason)
+    for hop in range(MAX_FETCH_REDIRECTS + 1):
+        if reason := _validate_fetch_url(current, allowed_domains):
+            if hop == 0:
+                raise _FetchBlocked(reason)
+            raise _FetchBlocked(f"redirect to a disallowed location: {reason}")
         response = await session.get(current, headers=headers, allow_redirects=False)
         location = response.headers.get(aiohttp.hdrs.LOCATION)
         if response.status in _REDIRECT_STATUSES and location:
@@ -116,6 +126,7 @@ def __init__(self, config: Config, runtime: Runtime):
         super().__init__()
         self._runtime = runtime
         self._service_config = config.services.pythinker_ai_fetch
+        self._allowed_domains = config.web.allowed_domains
 
     @override
     async def __call__(self, params: Params) -> ToolReturnValue:
@@ -134,11 +145,18 @@ async def __call__(self, params: Params) -> ToolReturnValue:
                 return ret
             logger.warning("Failed to fetch URL via service: {error}", error=ret.message)
             # fallback to local fetch if service fetch fails
-        return await self.fetch_with_http_get(params)
+        return await self.fetch_with_http_get(params, self._allowed_domains)
 
     @staticmethod
-    async def fetch_with_http_get(params: Params) -> ToolReturnValue:
+    async def fetch_with_http_get(
+        params: Params, allowed_domains: list[str] | None = None
+    ) -> ToolReturnValue:
         builder = ToolResultBuilder(max_line_length=None)
+        # Validate the initial URL up front so a disallowed host is rejected
+        # before any network session is opened; the redirect helper below
+        # re-validates every subsequent hop.
+        if reason := _validate_fetch_url(params.url, allowed_domains):
+            return builder.error(f"Failed to fetch URL: {reason}", brief="URL blocked")
         headers = {
             "User-Agent": (
                 "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
@@ -150,7 +168,9 @@ async def fetch_with_http_get(params: Params) -> ToolReturnValue:
             fetch_timeout = aiohttp.ClientTimeout(total=180, sock_read=60, sock_connect=15)
             async with new_client_session(timeout=fetch_timeout) as session:
                 try:
-                    response = await _get_revalidating_redirects(session, params.url, headers)
+                    response = await _get_revalidating_redirects(
+                        session, params.url, headers, allowed_domains
+                    )
                 except _FetchBlocked as blocked:
                     return builder.error(
                         f"Failed to fetch URL: {blocked.reason}", brief="URL blocked"
@@ -245,7 +265,7 @@ async def _fetch_with_service(self, params: Params) -> ToolReturnValue:
                 "Fetch service is not configured. You may want to try other methods to fetch.",
                 brief="Fetch service not configured",
             )
-        if reason := _validate_fetch_url(params.url):
+        if reason := _validate_fetch_url(params.url, self._allowed_domains):
             return builder.error(f"Failed to fetch URL: {reason}", brief="URL blocked")
 
         headers = {

src/pythinker_code/tools/web/search.md

@@ -1 +1 @@
-WebSearch tool allows you to search on the internet to get latest information, including news, documents, release notes, blog posts, papers, etc.
+WebSearch tool allows you to search on the internet to get latest information, including news, documents, release notes, blog posts, papers, etc. Results may be limited to a configured set of allowed domains.

src/pythinker_code/tools/web/search.py

@@ -1,5 +1,6 @@
 from pathlib import Path
 from typing import override
+from urllib.parse import urlparse
 
 import aiohttp
 from pydantic import BaseModel, Field, ValidationError
@@ -12,6 +13,7 @@
 from pythinker_code.soul.toolset import get_current_tool_call_or_none
 from pythinker_code.tools import SkipThisTool
 from pythinker_code.tools.utils import ToolResultBuilder, load_desc
+from pythinker_code.tools.web._allowlist import host_in_allowlist
 from pythinker_code.utils.aiohttp import new_client_session
 from pythinker_code.utils.logging import logger
 
@@ -53,6 +55,7 @@ def __init__(self, config: Config, runtime: Runtime):
         self._api_key = config.services.pythinker_ai_search.api_key
         self._oauth_ref = config.services.pythinker_ai_search.oauth
         self._custom_headers = config.services.pythinker_ai_search.custom_headers or {}
+        self._allowed_domains = config.web.allowed_domains
 
     @override
     async def __call__(self, params: Params) -> ToolReturnValue:
@@ -145,6 +148,26 @@ async def __call__(self, params: Params) -> ToolReturnValue:
                 brief="Search request failed",
             )
 
+        if self._allowed_domains:
+            kept = [
+                result
+                for result in results
+                if host_in_allowlist(urlparse(result.url).hostname, self._allowed_domains)
+            ]
+            dropped = len(results) - len(kept)
+            results = kept
+            if dropped:
+                builder.extras(allowlist_filtered=dropped)
+            if not results:
+                # Structured zero-result signal so the renderer reports "0
+                # results" instead of misreading the prose below as one result.
+                builder.extras(returned_results=0)
+                return builder.ok(
+                    f"All {dropped} search result(s) were outside the configured "
+                    "web allowlist and have been omitted.",
+                    brief="Filtered by allowlist",
+                )
+
         for i, result in enumerate(results):
             if i > 0:
                 builder.write("---\n\n")