Spike: 3-layer pedagogical signal architecture — Reflection lesson type + inline Knowledge Check primitive (IDT v2.0.0)

[lapc506]

Type: Spike

Size: M (research only — implementation goes to a follow-up Spec issue post-findings)

Strategy: Explore

Components: Database, Backend

Flags: Epic (parent of follow-up Spec + Implementation issues, gated on findings)

---

👤 HUMAN LAYER

User Story

As a course instructor consuming an IDT-powered LMS, I want per-chapter funnel signal layered with three pedagogical depths — engagement, comprehension, and mastery so that I can identify where learners drop off with confidence, distinguishing between "didn't open the lesson", "opened but didn't engage with comprehension checks", and "engaged but failed the formal evaluation".

As an IDT toolkit maintainer, I want this signal-layering exposed as reusable primitives so that multiple LMS consumers can adopt the same pedagogical depth without each one re-implementing schema, render logic, and signal extraction from scratch.

Background / Why

A common pedagogical pattern surfaces three layers of learner progression signal per learning unit (chapter / module / lesson group):

1. Engagement — did the learner open and traverse the content?
2. Comprehension — did the learner answer the embedded knowledge checks correctly during the unit?
3. Mastery — did the learner pass the formal evaluation (quiz / assessment) at the end of the unit?

Each layer is independently informative. A unit might have 95% engagement, 80% comprehension, 60% mastery — and that funnel shape tells the instructor that learners ARE engaging with the content but not internalizing it well enough to pass the assessment. Different shapes (e.g. 95/95/95 vs 95/40/40) imply different pedagogical interventions.

IDT today provides lesson primitives + xAPI/cmi5 emission. What's NOT yet provided as a reusable primitive:

• Reflection lesson type — a lesson that prompts the learner with an open-ended question and records the response. The "completed unit" signal (Layer 1 in pedagogical terms, distinct from "opened" — it captures intentional traversal vs passive scrolling).
• Inline Knowledge Check primitive — embeddable per-lesson checkpoint with structured answer state. Layer 2 signal source.
• Signal extraction utilities — shared TS helpers that consumers call to compute per-chapter funnel data from the new primitives + their own course/lesson schema.

A current consumer (DojoOS Pathways v1.7.x) is shipping a Course Manager → Completion Rates surface using a schema-existing 3-layer model (lesson_started → lesson_completed → quiz_passed). This is Ship-Mode-clean and produces actionable signal immediately, but it lacks the pedagogical richness that explicit reflection capture + structured comprehension checks would provide.

This spike validates feasibility of porting the pedagogical pattern into IDT primitives BEFORE committing to a v2.0.0 implementation that consumers (Pathways v2.0.0 Pedagogical Optimization milestone, future LMSes) can adopt.

Analogy

Think of a 3-stage car wash with sensors: water on (engagement), soap applied (comprehension), drying complete (mastery). Most LMSes today expose only stages 1 and 3 sensors via lesson-completed + quiz-passed flags. The middle "soap applied" sensor — did the learner actually grapple with the content during the lesson, not just scroll past it? — is the IDT primitive this spike scopes.

UX / Visual Reference

• DojoOS course schema (representative consumer, target mapping): courses → modules → chapters → lessons. Chapter is the pedagogical equivalent of a self-contained learning unit with ~5-15 lessons. Per-chapter funnel surfaces attrition.
• Mockup of a DojoOS lesson with embedded KC — TBD via this spike.
• Mockup of the per-chapter funnel chart with three-layer color coding — TBD via this spike.

Known Pitfalls & Gotchas

• Reflection lesson type touches the course builder — every consumer's authoring tool needs UI for creating + editing reflection prompts. IDT exposes the data model + render/playback primitives; consumers plug them into their own builder UX.
• Cross-consumer primitive design is harder than single-consumer. A reflection lesson type designed for one consumer might miss requirements that another needs. Spike must surface what each consumer's schema constraints are BEFORE locking in the IDT primitive.
• xAPI / cmi5 alignment — IDT v1 already emits xAPI statements for existing lesson types. New lesson types should align with the existing verb taxonomy, not introduce a parallel signal pipeline. Spike must trace which xAPI verbs map to "reflection submitted" + "KC answered correctly".
• Signal extraction API surface stability — consumers will hard-code against this API; breaking changes are expensive. The contract must be small, pure, and versioned from day 1.
• License — IDT is BSL 1.1; new primitives inherit. Spike confirms no transitive dep brings in incompatible licensing.

---

🤖 AGENT LAYER

Objective

Validate (or refute) the feasibility of providing a 3-layer pedagogical signal architecture (Reflection + inline Knowledge Checks + dedicated Quiz signal extraction) as IDT primitives that LMS consumers can adopt without re-implementing schema, render logic, or signal extraction. Output: a written GO/NO-GO recommendation + (if GO) a follow-up Spec issue with implementation scope.

Context Files

• openspec/project.md — current architecture + conventions for IDT primitives.
• openspec/specs/ — IDT spec library (lesson types, signal emission, xAPI verbs).
• Existing xAPI verb taxonomy in IDT — locate via grep -rln "xapi\|cmi5" openspec/specs/.
• Reference: representative consumer's course schema (DojoOS Pathways) — read-only audit of courses / chapters / lessons / lesson_progress / quiz_results shape.

Acceptance Criteria

• Document the stack-agnostic primitive contract: which signals are pedagogically universal (carry over to any LMS) vs which are consumer-specific (live in adapters, not core IDT).
• Define a Reflection lesson type schema proposal: prompt, response storage, completion criteria, xAPI verb mapping.
• Define an Inline Knowledge Check primitive schema proposal: question types supported (MC, fill-in-blank, matching), answer state storage, completion criteria, xAPI verb mapping.
• Define a signal extraction TS API: function signatures consumers call to compute per-chapter funnel data (e.g. computeChapterFunnel(courseId, chapterId, userId): { engagement, comprehension, mastery }).
• Validate against at least 2 consumer integration cases (real or hypothetical): different course schema shapes, different authoring stacks. Identify where the primitive holds vs where it leaks.
• Identify migration cost for existing courses (do they need authoring rework, or can the new types ship side-by-side?).
• Identify schema drift risk if N consumers all consume the same IDT primitive but evolve their own funnel UX in parallel. Recommend versioning policy.
• GO/NO-GO recommendation with explicit rationale (what blocks GO, what blocks NO-GO).
• If GO: follow-up Spec issue opened in this repo with detailed implementation scope. Linked from this spike's closing comment.
• If NO-GO: alternative recommendation documented (e.g. consumers continue using schema-existing 3-layer models, or invest in a different signal layer like time-on-task / scroll depth).

Technical Constraints

• IDT must remain stack-agnostic (no Postgres / Supabase / specific-LMS-specific dependencies in core primitives — those live in consumer adapters).
• xAPI verb choice must align with existing IDT vocabulary (no parallel verb taxonomy).
• Signal extraction TS API must be pure-function — no I/O in core, consumers wire in their own DB adapter.
• BSL 1.1 license inheritance.

Verification Commands

# Inspect IDT current primitives
ls openspec/specs/
grep -rln \"lesson type\\|signal\\|xapi\\|cmi5\" openspec/specs/

# Identify existing xAPI verb taxonomy
grep -rln \"xapi\" openspec/specs/ | head -10

# Inspect consumer schema (read-only — example: DojoOS Pathways)
# (Requires consumer repo access; adapter audit only.)

Agent Strategy

Mode: Explore

Investigation questions:

1. What's the minimal cross-LMS data shape for a "reflection submitted" event?
2. Does IDT v1's xAPI verb taxonomy have an answered / responded verb that fits, or do we need a new verb (and is that an xAPI extension, not just a vocabulary pick)?
3. Is the inline-KC primitive better authored as a lesson sub-component (one lesson contains N KCs) or as N separate lessons (one KC = one lesson)? Which model does the existing IDT lesson taxonomy prefer?
4. Can the signal extraction API cleanly serve BOTH a per-chapter funnel chart AND a per-learner roster table (the Course Manager Roster sub-sub-tab in the consumer reference) without forking?
5. What's the minimum viable signal we can ship that improves on the schema-existing 3-layer model in current consumers? Reflection alone? KCs alone? Both?

Read-only phase:

• IDT repo: full openspec/ + docs/ + existing primitives.
• Representative consumer (DojoOS Pathways): courses + chapters + lessons + lesson_progress + quiz_results schema (read-only, adapter-level audit).

Decision point:
After 1-2 days of read-only investigation, the spike produces:

1. A written feasibility memo (~1-2 pages) covering each acceptance criterion.
2. A GO/NO-GO recommendation with explicit rationale.
3. If GO: a Spec issue with implementation scope, opened in this same repo and linked from this spike's closing comment.
4. If NO-GO: an alternative recommendation captured here.

The spike CLOSES at this point. Implementation (if GO) lives in the follow-up Spec + its child Implementation issue(s).

---

🔀 Parallelization Recommendation

Recommended mechanism: Subagents (read-only investigation in parallel)

Reasoning:

The spike is read-only across IDT + 1-2 representative consumers. Each repo's investigation is independent — no cross-write dependencies. Subagents are the right fit: cheap, parallel, results-only, no risk of file conflicts.

• Subagent 1 — IDT current state audit (existing primitives, xAPI taxonomy, openspec/changes already addressing similar pedagogical signal work).
• Subagent 2 — Representative consumer schema audit (course/chapter/lesson tables + lesson_progress + quiz_results).
• Subagent 3 — xAPI verb taxonomy audit + cross-LMS standardization research (which verbs exist for reflection / KC / quiz across xAPI / cmi5 / TinCan).
• Synthesizer subagent: produces the unified feasibility memo + GO/NO-GO recommendation.

Cost estimate: ~1.5x base token cost. Cheap relative to a 1-week implementation that turns out to be NO-GO.

---

📋 Spec Recommendation (Mode B follow-up if GO)

Per the standard /spec-recommend skill template, the post-spike Spec issue should follow this scaffold (open ONLY if this spike returns GO):

Title: Spec: IDT v2.0.0 reflection lesson type + inline KC primitive + signal extraction TS API

Body sections (Bilingual Format):

1. Human Layer (user story, background incorporating spike findings, analogy, UX reference, pitfalls confirmed by spike).
2. Agent Layer:
   • Objective: ship the 3 IDT primitives (Reflection lesson type, Inline KC, Signal extraction TS API).
   • Current State Audit: derived from spike's read-only audit.
   • Acceptance criteria: from spike's feasibility memo.
   • Technical constraints: BSL 1.1, stack-agnostic, xAPI-aligned, pure-function signal extraction.
   • Implementation Plan: OpenSpec change proposal → tasks → archive.
3. Consumer rollout mapping (which consumers adopt the new primitives in which milestone).

Type: Feature
Size: L (cross-consumer toolkit work — likely needs decomposition into 3 sub-issues per primitive)
Strategy: Team (frontend authoring + backend signal extraction + xAPI alignment all need attention)

The Spec issue is NOT opened with this spike. It's the spike's final deliverable if findings are GO. If findings are NO-GO, no Spec issue is opened; an alternative recommendation is documented here instead.

---

Synthesis Additional Comments

MECE Logical Validation

Mutually Exclusive: This spike does NOT overlap with:

• Existing IDT v1 xAPI/cmi5 work — this builds ON it, doesn't replace it.
• Consumer-side funnel UX work — that's a consumer concern; this spike scopes only the toolkit primitives that consumers consume.

Collectively Exhaustive: The 9 acceptance criteria cover: primitive contract, 2 primitive designs (Reflection + Inline KC) × schema each, signal extraction API, ≥2 consumer integration validations, migration cost, drift risk + versioning policy, GO/NO-GO, follow-up Spec scaffolding. Nothing pedagogically meaningful from the 3-layer pattern is silently dropped.

Executive Synthesis (Minto Pyramid)

Lead with the Answer: This spike asks "can IDT host the cross-LMS pedagogical primitives that consumers need to surface engagement / comprehension / mastery as separate signals, without forcing each consumer to build it from scratch?"

Supporting Arguments:

1. Toolkit reuse: if YES, every future LMS consumer gets the primitives for free.
2. Pedagogical depth: consumers using schema-existing 3-layer models today derive their layers from existing data. Native primitives give richer pedagogical signal (intentional reflection capture + structured comprehension checks) without each consumer reinventing.
3. xAPI alignment: IDT already emits xAPI verbs; the new primitives extend that vocabulary cleanly.

Data & Evidence: Pedagogical literature on retention has long held that active recall (KCs) + reflective integration (open-ended response) outperforms passive content traversal. The cost of NOT having native primitives is that each consumer either skips the layer or builds proprietary extraction — which fragments the cross-LMS data picture.

Pareto 80/20 Efficiency Review

The minimum viable IDT primitive set for v2.0.0 is likely Reflection lesson type only (the strongest "unit fully consumed" signal). Inline KCs are a richer signal but require more authoring + question-type-handling complexity. If the spike confirms 80% of the value comes from reflection alone, it can ship FIRST and KCs follow in a later release.

Second-Order Thinking & Risk Assessment

Scalability: If 3+ LMS consumers consume IDT primitives, schema evolution requires breaking-change discipline. Recommend: from day 1, version the primitive schemas (e.g. reflection.v1.json) and adopt a deprecation policy.

Downstream Effects: If IDT primitives change pedagogically (e.g. Reflection becomes "Reflection + AI feedback"), existing data must migrate. Spike must surface this as a risk.

Future Maintenance: The signal extraction TS API surface must be small and stable. Recommend: a single computeChapterFunnel() function with a clear input/output contract — no leaky abstractions.

---

Created by Claude Code on behalf of @lapc506