U27-S03 // Context Engine

Context Engine converts repositories into governed, token-aware context packages for AI agents.

U27-S03
CONTEXT ENGINE

CLASS: SYSTEM
OPERATING_POSITION: 02/08
FUNCTION: Context Ingestion + Token Intelligence
REF_ID: U27-S03-CONTEXT-ENGINE

Release Status

SOURCE_STATUS: PUBLIC_PACKAGE ACCESS_STATUS: CLEARED_FOR_EXTERNAL_USE

This repository is a released Unit27 field kit: visible, inspectable, and intended for orientation, testing, and practical use. Controlled protocol materials remain outside this source package.

It answers one narrow question:

What context should an AI agent receive, and what was included or excluded?

Why Use It

Use Context Engine before handing a repository to an AI agent when the agent needs the right files, proof of what was included, and a token budget that will not drift silently.

It is useful when a repo is too large for a clean prompt, when generated context needs to be inspectable, or when the difference between core files and background files matters.

Example:

Problem: The agent gets the whole repo and misses the important files.
Result: Context Engine produces a scored, token-aware manifest before the handoff.

Demo

The current public release is GitHub-first. Run it from a local checkout:

git clone https://github.com/unit27research/unit27-context-engine
cd unit27-context-engine
pip install -e .
context-engine demo
context-engine scan . --max-tokens 50000

CLI

context-engine demo
context-engine scan . --max-tokens 50000
context-engine inspect context_report.json

Scan options:

context-engine scan PATH \
  --output context_manifest.md \
  --report context_report.json \
  --max-tokens 50000 \
  --max-file-bytes 1000000 \
  --trim \
  --fail-on-overage \
  --exact-tokens \
  --exclude "*.lock" \
  --config context-engine.yaml

Architecture

repo -> scan -> classify -> count -> enforce -> output

The scanner applies default excludes with directory pruning, nested .gitignore rules, custom excludes, binary extension checks, null-byte checks, file-size guards, priority classification, centrality scoring, token counting, static model pricing, and budget enforcement before writing the manifest and JSON report.

Token counts use exact tiktoken counting by default. Context Engine packages the cl100k_base cache file so normal scans do not need to fetch encoding data at runtime. If an operator deliberately wants fallback behavior, --approximate-tokens records TOKEN_MODE: APPROXIMATE in terminal output and reports.

Trim mode uses deterministic context scoring inside priority bands. It favors entrypoints, project contracts, repository overviews, inbound imports, dependency context, test relevance, docs proximity, and recency before spending the token budget.

Reliability

Context Engine is released as part of the Unit27 public tooling channel. CI verifies the test suite, CLI acceptance path, and wheel contents before changes are considered ready.

System Position

Stack Engine -> Context Engine -> Knowledge Readiness -> Handoff Engine -> Eval Bench -> Proof Ledger -> Boundary Engine -> u27-check

Context Engine sits after system shaping and before Knowledge Readiness classification. It packages repository context so prepared material can be classified before it becomes memory, onboarding material, automation context, handoff material, or proof.

What It Does Not Do

Context Engine does not:

1. Decide which project should be built
2. Generate agent handoff packets
3. Run implementation evals
4. Record durable proof
5. Check public claims
6. Perform launch QA

Exit Codes

0 = success
1 = token budget exceeded
2 = scan error
3 = config error

Acceptance

context-engine demo
context-engine scan tests/fixtures --max-tokens 50000
context-engine scan tests/fixtures --max-tokens 100 --fail-on-overage
context-engine scan tests/fixtures --max-tokens 100 --trim
context-engine inspect demo_outputs/sample_report.json
pytest