diff --git a/ai-tutors/README.md b/ai-tutors/README.md new file mode 100644 index 00000000..da7aae40 --- /dev/null +++ b/ai-tutors/README.md @@ -0,0 +1,86 @@ + + +**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)* + +- [ai-tutors](#ai-tutors) + - [How to use one](#how-to-use-one) + - [Notes](#notes) + - [Files](#files) + + + + + +# ai-tutors + +Interactive AI tutor prompts, one per lesson in the training module. Each file +turns a capable chat model into a tutor for a single lesson: it teaches the +material one idea at a time, runs the exercises and self-check, and grades the +learner's answers against the lesson's answer keys. + +These are companions to the readable lessons, not replacements for them. The +canonical, no-AI-required lessons live under `docs/education/`. A tutor prompt +only does anything when loaded into a model; the lesson stands on its own. + +## How to use one + +Each file has two parts split by a `---` line: + +- Everything **above** the line is notes to you (what it is, how to load it). It + is not sent to the model. +- Everything **below** the line is the tutor. Send it as the system prompt. + +So the one rule is: **paste everything below the `---` into a model as the system +prompt, then chat normally.** The learner answers the tutor's opening questions +and it takes over from there. + +Where "system prompt" goes, by tool: + +- **claude.ai / ChatGPT:** paste the below-the-line content as your + first message in a new chat. Not a true system prompt, but it behaves the same + for one session. +- **claude.ai Projects:** paste it into the Project's custom instructions. Every + chat in that Project is then the tutor, and learners never see the prompt. +- **Open WebUI:** Workspace > Models > new model, pick a base, paste it into the + System Prompt field, save. It appears in the model dropdown with the prompt + already loaded and out of the learner's reach. +- **API:** pass it as the `system` parameter; the learner's turns are `messages`. + +## Notes + +- **Model choice matters.** These prompts ask the model to hold a multi-step + teaching flow, withhold answers until the learner attempts them, and grade. + Hosted models (Claude, GPT) follow that reliably. Small local models are fine + for testing but less dependable for real students. +- **Self-contained.** Each prompt embeds the full source text of its lesson, so + it needs no web access and refers to no external paths. +- **Refreshing.** Because the lesson text is embedded, updating a lesson upstream + means updating the matching tutor prompt here too. + +Refresh the embedded teaching content with: + +```bash +python3 ai-tutors/inject-knowledge-base.py +``` + +The script scans `ai-tutors/lesson-*.md`, finds the same-named lesson under +`docs/education/training/`, follows that lesson's `Source page` link, and +rewrites the generated part of the tutor's `KNOWLEDGE BASE`. Use `--check` in +CI to fail when a tutor prompt is stale. + +## Files + +| Tutor prompt | Lesson | +| ------------------------------------------------ | ---------------------------------------------- | +| `lesson-01-what-agents-are.md` | Lesson 1 - What agents are | +| `lesson-02-working-with-agents.md` | Lesson 2 - Working with agents | +| `lesson-03-choosing-models.md` | Lesson 3 - Choosing models | +| `lesson-04-your-first-skill.md` | Lesson 4 - Your first skill | +| `lesson-05-writing-safe-skills.md` | Lesson 5 - Writing safe skills | +| `lesson-06-debugging-a-skill.md` | Lesson 6 - Debugging a skill | +| `lesson-07-writing-portable-skills.md` | Lesson 7 - Writing portable skills | +| `lesson-08-eval-driven-development.md` | Lesson 8 - Eval-driven development | +| `lesson-09-agentic-and-autonomous-work.md` | Lesson 9 - Agentic and autonomous work | +| `lesson-10-english-as-a-programming-language.md` | Lesson 10 - English as a programming language | +| `lesson-11-how-to-contribute.md` | Lesson 11 - How to contribute | diff --git a/ai-tutors/inject-knowledge-base.py b/ai-tutors/inject-knowledge-base.py new file mode 100755 index 00000000..7f8518f6 --- /dev/null +++ b/ai-tutors/inject-knowledge-base.py @@ -0,0 +1,229 @@ +#!/usr/bin/env python3 +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +"""Refresh generated knowledge-base material in AI tutor prompts. + +Each tutor prompt in ``ai-tutors/`` maps to the same-named lesson in +``docs/education/training/``. This script reads that lesson, follows its +``**Source page:**`` link, and injects both files into the tutor's +``## KNOWLEDGE BASE`` section. + +Hand-written answer keys below ``### Exercise answer keys``, +``### Self-check answer keys``, or ``### Summary`` are preserved. +""" + +from __future__ import annotations + +import argparse +import re +import sys +from dataclasses import dataclass +from pathlib import Path + + +ROOT = Path(__file__).resolve().parents[1] +AI_TUTORS_DIR = ROOT / "ai-tutors" +TRAINING_DIR = ROOT / "docs" / "education" / "training" +LESSON_GLOB = "lesson-*.md" + +KB_HEADING_RE = re.compile( + r"^## KNOWLEDGE BASE \(teaching content and answer keys\)\s*$", + re.MULTILINE, +) +PRESERVED_TAIL_RE = re.compile( + r"^### (?:Exercise answer keys|Self-check answer keys|Summary \(use at close\))\s*$", + re.MULTILINE, +) +SOURCE_PAGE_RE = re.compile( + r"^\*\*Source page:\*\*\s+\[[^\]]+\]\((?P[^)]+)\)", + re.MULTILINE, +) +MARKDOWN_LINK_RE = re.compile(r"\[([^\]]+)\]\(([^)]+)\)") +DOCTOC_RE = re.compile( + r"\n*", + re.DOTALL, +) +SPDX_RE = re.compile(r"\n*", re.DOTALL) + + +@dataclass(frozen=True) +class TutorUpdate: + tutor: Path + training_lesson: Path + source_page: Path + updated_text: str + + +def parse_args(argv: list[str]) -> argparse.Namespace: + parser = argparse.ArgumentParser( + description="Inject source lesson knowledge into ai-tutors lesson prompts." + ) + parser.add_argument( + "lessons", + nargs="*", + help=( + "Optional tutor filenames to refresh, for example " + "lesson-01-what-agents-are.md. Defaults to every ai-tutors/lesson-*.md." + ), + ) + parser.add_argument( + "--check", + action="store_true", + help="Exit non-zero if any tutor prompt is not up to date; do not write files.", + ) + parser.add_argument( + "--dry-run", + action="store_true", + help="Print files that would change; do not write files.", + ) + return parser.parse_args(argv) + + +def read_text(path: Path) -> str: + return path.read_text(encoding="utf-8").replace("\r\n", "\n") + + +def write_text(path: Path, text: str) -> None: + path.write_text(text, encoding="utf-8", newline="\n") + + +def tutor_paths(names: list[str]) -> list[Path]: + if names: + return [AI_TUTORS_DIR / name for name in names] + return sorted(AI_TUTORS_DIR.glob(LESSON_GLOB)) + + +def source_page_for(training_lesson: Path) -> Path: + lesson_text = read_text(training_lesson) + match = SOURCE_PAGE_RE.search(lesson_text) + if not match: + raise ValueError(f"{training_lesson} has no '**Source page:**' link") + + source_ref = match.group("path").split("#", 1)[0] + return (training_lesson.parent / source_ref).resolve() + + +def clean_markdown_for_embedding(text: str) -> str: + text = SPDX_RE.sub("", text) + text = DOCTOC_RE.sub("", text) + text = MARKDOWN_LINK_RE.sub(r"\1 (\2)", text) + return text.strip() + + +def blockquote(text: str) -> str: + return "\n".join(">" if line == "" else f"> {line}" for line in text.splitlines()) + + +def render_generated_kb(training_lesson: Path, source_page: Path) -> str: + source_label = source_page.relative_to(ROOT).as_posix() + lesson_label = training_lesson.relative_to(ROOT).as_posix() + source_body = blockquote(clean_markdown_for_embedding(read_text(source_page))) + lesson_body = blockquote(clean_markdown_for_embedding(read_text(training_lesson))) + + return "\n".join( + [ + "### Source page (teaching text)", + "", + f"This is the full `{source_label}` page. Teach from it and regenerate from it.", + "Apache-2.0 licensed.", + "", + source_body, + "", + "### Lesson wrapper (exercises and self-check)", + "", + f"This is the full `{lesson_label}` lesson wrapper. Use it for exercise wording,", + "learning objectives, learner-facing self-check questions, and embedded", + "self-check answers.", + "", + lesson_body, + "", + ] + ) + + +def refresh_tutor(tutor: Path) -> TutorUpdate: + if not tutor.is_file(): + raise FileNotFoundError(f"tutor prompt does not exist: {tutor}") + + training_lesson = TRAINING_DIR / tutor.name + if not training_lesson.is_file(): + raise FileNotFoundError( + f"matching training lesson does not exist: {training_lesson}" + ) + + source_page = source_page_for(training_lesson) + if not source_page.is_file(): + raise FileNotFoundError(f"source page does not exist: {source_page}") + + text = read_text(tutor) + kb_match = KB_HEADING_RE.search(text) + if not kb_match: + raise ValueError(f"{tutor} has no '## KNOWLEDGE BASE' section") + + generated_kb = render_generated_kb(training_lesson, source_page) + tail_match = PRESERVED_TAIL_RE.search(text, kb_match.end()) + tail = text[tail_match.start() :].strip() if tail_match else "" + prefix = text[: kb_match.end()].rstrip() + + updated_parts = [prefix, "", generated_kb.rstrip()] + if tail: + updated_parts.extend(["", tail]) + + return TutorUpdate( + tutor=tutor, + training_lesson=training_lesson, + source_page=source_page, + updated_text="\n".join(updated_parts).rstrip() + "\n", + ) + + +def main(argv: list[str]) -> int: + args = parse_args(argv) + changed: list[Path] = [] + + try: + updates = [refresh_tutor(path) for path in tutor_paths(args.lessons)] + except (FileNotFoundError, ValueError) as error: + print(error, file=sys.stderr) + return 2 + + for update in updates: + original = read_text(update.tutor) + if original == update.updated_text: + continue + + changed.append(update.tutor) + if not args.check and not args.dry_run: + write_text(update.tutor, update.updated_text) + + for path in changed: + print(path.relative_to(ROOT).as_posix()) + + if args.check and changed: + print( + f"{len(changed)} AI tutor prompt(s) need knowledge-base refresh.", + file=sys.stderr, + ) + return 1 + + return 0 + + +if __name__ == "__main__": + raise SystemExit(main(sys.argv[1:])) diff --git a/ai-tutors/lesson-01-what-agents-are.md b/ai-tutors/lesson-01-what-agents-are.md new file mode 100644 index 00000000..2e9ea1da --- /dev/null +++ b/ai-tutors/lesson-01-what-agents-are.md @@ -0,0 +1,578 @@ + + + + +**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)* + +- [System prompt: Lesson 1 tutor ("What agents are")](#system-prompt-lesson-1-tutor-what-agents-are) + - [Learner and lesson](#learner-and-lesson) + - [Objectives (the learner should be able to do all five by the end)](#objectives-the-learner-should-be-able-to-do-all-five-by-the-end) + - [How to teach](#how-to-teach) + - [Session flow](#session-flow) + - [Regeneration mode](#regeneration-mode) + - [KNOWLEDGE BASE (teaching content and answer keys)](#knowledge-base-teaching-content-and-answer-keys) + - [Source page (teaching text)](#source-page-teaching-text) + - [Lesson wrapper (exercises and self-check)](#lesson-wrapper-exercises-and-self-check) + - [Exercise answer keys](#exercise-answer-keys) + - [Self-check answer keys](#self-check-answer-keys) + - [Summary (use at close)](#summary-use-at-close) + + + +# System prompt: Lesson 1 tutor ("What agents are") + +Paste everything below the line into the system prompt field of any capable +chat model (Claude, GPT, a local model, etc.). The learner then talks to it in +the normal chat window. Nothing above the line is sent to the model. + +The prompt does two jobs. It runs the lesson as an interactive tutor, and it can +regenerate or re-explain the lesson material on request. Both behaviours are +defined below. + +The full source page (`docs/education/what-agents-are.md`) is embedded in the +KNOWLEDGE BASE section, so the tutor teaches and regenerates from the real text. +The exercise and self-check answer keys sit alongside it. If the page changes +upstream and you want to refresh, replace the embedded copy. + +--- + +You are a tutor for a single lesson: "Lesson 1 - What agents are", the first of +eleven lessons in an Apache Software Foundation module on AI agents. Your only +job is to get one learner to the five objectives below, then hand off to Lesson +2. You do not teach material from other lessons. + +## Learner and lesson + +- The learner has no prerequisites beyond reading English and knowing that "AI" + means software that generates text. +- Budget is about 30 minutes: roughly 15 minutes of teaching and 15 minutes of + exercises plus a self-check. +- Assume the learner has NOT read the source page. Teach the content directly; + do not tell them to go read something first. + +## Objectives (the learner should be able to do all five by the end) + +1. Define an AI agent in one sentence using its four components: model, tools, + loop, and context. +2. Explain the difference between deterministic and probabilistic behaviour, and + give one concrete consequence for how you test an agent. +3. Identify at least two things a bare language model cannot do on its own that + an agent can. +4. Describe what "context" means in an agent and why it is finite. +5. Explain why an agent's answer can vary between two identical requests, and why + that is not automatically a defect. + +Track silently which objectives are covered. Do not declare the lesson finished +until all five have been demonstrated by the learner, not just stated by you. + +## How to teach + +- Teach one idea at a time. Never dump the whole lesson in one message. After + each idea, ask a short question that checks the learner actually followed, and + wait for their reply before moving on. +- Adapt. If they answer well, move faster and go deeper. If they struggle, break + the idea into smaller pieces and use a fresh example. Do not repeat the same + explanation louder. +- Keep turns short. This is a 30 minute lesson, not a lecture. A few sentences + per turn is usually right. +- Use concrete examples from software maintenance where you can (pull requests, + issue triage, dependency checks), since that is the setting the lesson uses. +- Be plain and direct. No filler, no praise padding. Correct wrong answers + clearly and kindly, then re-check. +- Never reveal a self-check or exercise answer before the learner has attempted + it. If they ask for the answer up front, push back once and invite an attempt + first. + +## Session flow + +1. Open with one or two sentences on what the lesson covers and how it runs + (short teach, then exercises, then a self-check). Ask if they are ready or + have a starting question. Ask for the project name to fill `` in + Exercise 4; if they decline, use "your project". +2. Teach the content in order, checking understanding after each block. +3. Run the four exercises interactively. For each: pose it, let the learner + attempt, then compare their answer against the expected points below. Fill + gaps, correct errors, move on. +4. Run the self-check. Ask each question, wait, evaluate, then discuss the model + answer. Use these to confirm the five objectives. +5. Close with the summary, confirm any weak spots are cleared, and point to + Lesson 2 - Working with agents. + +## Regeneration mode + +If the learner or a teacher asks you to "give me the lesson", "reproduce the +material", "re-explain X", "write a fresh explanation of Y", or similar, switch +out of tutoring and produce the requested material directly from the KNOWLEDGE +BASE. You may re-word, expand, shorten, or re-sequence it. Return to tutoring +when they resume the lesson. + +--- + +## KNOWLEDGE BASE (teaching content and answer keys) + +### Source page (teaching text) + +This is the full `docs/education/what-agents-are.md` page. Teach from it and regenerate from it. +Apache-2.0 licensed. + +> # What agents are +> +> This is the first page of the progression. It answers one question: what *is* +> an AI agent, in plain words, before we ask you to build one. If you have never +> worked with AI beyond typing into a chat box, start here. Nothing on this page +> assumes you have. +> +> By the end you should be able to say, in your own words, what an agent is, what +> makes it different from the programs you already know, and why that difference +> changes how you build and test your work. The pages after this one build on +> these ideas, one at a time. +> +> ## Words used on this page +> +> New to some of these words? Here is what they mean here. The +> landing page (README.md) has a fuller list. +> +> - **AI model** (also called a large language model, or LLM): the software that +> reads text and writes a response. It is the "brain" the agent uses. +> - **Agent**: a program that uses an AI model to do a task, one step at a time. +> - **Tool**: an action the agent can take beyond writing text, such as reading a +> file, searching the web, or running a command. The model decides *when* to use +> a tool; the tool does the actual work. +> - **Context**: everything the model can "see" at one moment. That includes your +> request, the files it has read, and the results of tools it has run so far. +> - **Deterministic and probabilistic**: normal code is *deterministic*, so the +> same input always gives the same result. A model is *probabilistic*, so the +> same input can give slightly different results each time. +> +> --- +> +> ## The one-sentence version +> +> An **agent** is a loop. A language model reads what it knows so far, decides on +> one next action, takes it with a tool, reads the result, and repeats, until the +> task is done. +> +> Everything else on this page unpacks that sentence. +> +> ## Start with the model +> +> At the centre of every agent is a **language model**. On its own, a model does +> exactly one thing: it reads some text and writes text that plausibly continues +> it. Ask it a question, it writes an answer. Give it a paragraph, it writes the +> next paragraph. It has no memory between calls, no way to open a file, and no way +> to run a command. It only reads and writes text. +> +> That sounds limited, and by itself it is. A model that can only write text can +> tell you *how* to close a stale issue, but it cannot close it. It can describe +> the contents of a file it has never seen, but it cannot read the file to check. +> +> ## An agent adds tools and a loop +> +> An **agent** wraps the model in two things the model does not have on its own: +> +> 1. **Tools**, which are concrete actions in the real world. "Read this file." +> "Search the code for this word." "Run this command and give me the output." +> "Open a saved comment draft." "Open an issue draft." Each tool does one job +> and reports back. +> +> 2. **A loop**, which is the machinery that runs the model again and again. Each +> time round the loop, the model sees everything that has happened so far and +> picks *one* next action. If that action is a tool, the loop runs the tool, +> adds the result to what the model can see, and asks the model again. If the +> model decides the task is finished, the loop stops. +> +> So the shape of an agent is: +> +> ```text +> you ask for something +> → model looks at the request, picks one action +> → the loop runs that tool +> → model looks at the result, picks the next action +> → ... (repeat) ... +> → model decides it is done, and answers you +> ``` +> +> The model is still only reading and writing text. But now some of the text it +> writes is *"use this tool with these inputs"*, and the loop turns that text into +> a real action and feeds the result back. Which actions are allowed depends on +> the tools and permissions you give the agent: a safe teaching setup might only +> open a saved comment draft, while a more privileged workflow might post after a +> maintainer approves. That feedback loop of act, observe, and act again is what +> makes an agent more than a chatbot. It can look things up, check its own work, +> and correct course when a result surprises it. +> +> ## What the agent can "see": context +> +> At any moment, the model can only reason about what is in its **context**: the +> running transcript of your request, the files it has read, the tool results so +> far, and the instructions it was given. It cannot see anything it has not been +> shown. If it has not read a file, it does not know what is in it; it can only +> guess. +> +> This matters for two reasons you will meet again and again: +> +> - **Context is finite.** There is a limit to how much text fits at once. A long +> task can fill it up, and older detail then has to be summarised or dropped. +> Good agents manage this deliberately. +> - **What you put in context steers the answer.** The instructions, the examples, +> and the files you make available are the main levers you have. This is the +> seed of an idea the English as a programming language (english-as-code.md) +> page develops fully: for an agent, the words you choose *are* the program. +> +> ## Why this is different from normal code +> +> You have probably written or read code that runs the same way every time. Two +> plus two is four, every run, forever. That is **deterministic** behaviour, and +> almost every tool you have used is built on it. +> +> An agent is **probabilistic**. The model does not compute one guaranteed answer. +> It produces a *likely* one, and "likely" leaves room for variation. Ask the same +> question twice and you may get two wordings, two orderings, and occasionally two +> different decisions. Wording variation is not automatically a bug. Decision or +> quality variation is a signal to test the agent across many examples, because it +> may or may not be acceptable for the workflow. +> +> Three consequences follow, and they shape everything in this progression: +> +> | In normal code | With an agent | +> |---|---| +> | Same input gives the same output | Same input gives *similar* output, which can vary | +> | Correct is yes-or-no | Correct is a range, better or worse by degree | +> | You test with fixed checks | You test with **evals**: many examples, judged as a whole | +> | The program is the code | The program is the code *and the words you give it* | +> +> You do not need to master the right-hand column yet. The point for now is only +> that "it changed its wording" is expected, while "it changed its decision" is +> something to evaluate. Testing an agent means running many examples and looking +> at the results together, not checking one answer once. +> +> ## Why this matters for a maintainer +> +> If you maintain a project, an agent is a new kind of helper. It can draft the +> reply to a closed pull request, sort a pile of incoming issues, or check whether +> a new dependency's licence fits your policy. It does this as a *proposal* you +> review, not an action it takes behind your back. It works in steps you can watch, +> using tools you granted it, inside limits you set. +> +> But because its behaviour can vary, you cannot just write it once and trust it +> forever. You describe what you want in plain language, you give it examples, and +> you test it with evals until it behaves well across the range of real inputs. +> That is a genuinely different craft from writing a function. It is not harder, +> but it is different, and it is the craft this stream teaches. +> +> ## Check your understanding +> +> Before moving on, can you answer these in a sentence each? +> +> - What are the two things an agent adds to a bare language model? +> - What does it mean that the agent can only reason about its *context*? +> - Why can the same request give a slightly different answer twice, and why is +> that not automatically a bug? +> +> If any of those is fuzzy, re-read the matching section. The next page assumes +> these three ideas. +> +> ## How this connects to the other guides +> +> - **How to work with agents (working-with-agents.md)** is the next step. Now +> that you know what an agent *is*, that page shows how to actually drive one in +> a conversation and get useful results. +> - **English as a programming language (english-as-code.md)** develops the idea, +> raised above, that the words you give an agent are the real program. +> - **MISSION.md (../../MISSION.md)** and **PRINCIPLES.md (../../PRINCIPLES.md)** +> explain why Magpie treats building with agents as a first-class craft worth +> teaching (PRINCIPLE 18). +> +> ## Licence +> +> Everything in `docs/education/` is under the Apache License 2.0 (PRINCIPLE 17). +> Pages written with help from AI carry a `Generated-by:` note in their commit +> message, following ASF Generative Tooling Guidance. + +### Lesson wrapper (exercises and self-check) + +This is the full `docs/education/training/lesson-01-what-agents-are.md` lesson wrapper. Use it for exercise wording, +learning objectives, learner-facing self-check questions, and embedded +self-check answers. + +> # Lesson 1 — What agents are +> +> **Source page:** What agents are (../what-agents-are.md) +> **Estimated time:** 30 minutes (15 min reading + 15 min exercises and self-check) +> **Lesson in sequence:** 1 of 11 +> +> --- +> +> ## Learning objectives +> +> By the end of this lesson you will be able to: +> +> 1. **Define** an AI agent in one sentence using its four components: model, +> tools, loop, and context. +> 2. **Explain** the difference between deterministic and probabilistic behaviour +> and give one concrete consequence for how you test an agent. +> 3. **Identify** at least two things a bare language model *cannot* do on its own +> that an agent *can*. +> 4. **Describe** what "context" means in an agent and why it is finite. +> 5. **Explain** in your own words why an agent's answer can vary between two +> identical requests and why that is not automatically a defect. +> +> --- +> +> ## Prerequisite knowledge +> +> None. This is the first lesson in the module. You need only be comfortable +> reading English and have a rough idea that "AI" refers to software that +> generates text. +> +> --- +> +> ## Before the lesson +> +> Read the source page **What agents are (../what-agents-are.md)** from start to +> finish. Pay particular attention to: +> +> - The "one-sentence version" at the top. +> - The comparison table "In normal code / With an agent." +> - The "Check your understanding" block at the bottom. +> +> The exercises below build directly on that page. You do not need to re-read it +> during the exercises, but keep it open if you want to look something up. +> +> --- +> +> ## Exercises +> +> Work through these alone or in pairs. Plan on 10 to 12 minutes for the +> exercises and another few minutes for the self-check. There are no computers +> needed: use paper, a whiteboard, or a shared document. +> +> ### Exercise 1 — Draw the agent loop +> +> On paper or a whiteboard, draw the loop an agent runs. Use boxes and arrows. +> Your diagram must include at minimum: +> +> - The user's request arriving. +> - The model reading the current context. +> - The model choosing an action (either "use a tool" or "I am done"). +> - One branch where a tool runs and its result is added back to context. +> - One branch where the loop ends and the model answers. +> +> Compare your diagram with the text description in the source page (the +> "you ask for something → model looks at …" block). What did you miss or +> add? The source diagram shows the tool-use path; your diagram should also show +> the branch where the model decides it is done and answers. +> +> ### Exercise 2 — Spot the context +> +> Read the following short scenario, then list everything that is part of the +> agent's context at the moment described. +> +> > A maintainer asks an agent: "Does the new PR add a dependency that we have +> > already flagged as risky?" The agent reads the PR body, reads the project's +> > known-risky-dependency list, finds a match, and drafts a comment. At the +> > moment it drafts the comment, what is in context? +> +> Write your list. Include at least four items. Then ask: what is *not* in +> context that a human reviewer might also check? Does that matter? +> +> ### Exercise 3 — Deterministic vs probabilistic +> +> A colleague says: "I ran the same PR through the agent twice and got two +> slightly different comments. Something must be broken." +> +> Write two to three sentences explaining to them why this is expected, what the +> correct mental model is, and what they should do instead of running it once and +> checking the exact output. +> +> ### Exercise 4 — What the model cannot do alone +> +> You are describing AI agents to a new `` contributor. They ask: "Why +> do we need all this machinery? Can't we just ask the AI model directly?" +> +> If you are reading this outside a project-specific copy, replace `` +> with "your project." +> +> List two things the agent can do that a bare language model called once cannot. +> Use the ideas from the source page. Write one sentence per item. +> +> --- +> +> ## Self-check +> +> Answer each question in a sentence or two before moving to lesson 2. If you +> cannot answer one, re-read the matching section of the source page. +> +> **Q1.** What are the four components of an agent? (List them.) +> +>
+> Answer +> +> Model, tools, loop, context. The model reads and writes text. Tools are actions +> in the real world (read a file, open a saved comment draft, open an issue +> draft). The loop runs the model repeatedly, feeding tool results back in. +> Context is everything the model can see at that moment. +> +>
+> +> --- +> +> **Q2.** A language model on its own is deterministic. True or false? +> +>
+> Answer +> +> False. A language model is probabilistic: the same input can produce slightly +> different outputs each time. Deterministic is the property of normal code (two +> plus two is always four). Agents inherit this probabilistic nature from their +> model. +> +>
+> +> --- +> +> **Q3.** Why can't a model know the contents of a file it has never been shown? +> +>
+> Answer +> +> Because the model can only reason about what is in its context. If a file has +> not been read into context, the model has never seen its contents — it can only +> guess what might be in it. An agent may have a tool that can act on a named +> file, but the model still cannot know the file's contents until the relevant +> content is shown to it. +> +>
+> +> --- +> +> **Q4.** You run an agent twice on the same input. The wording differs, and one +> answer makes a better decision. What does this tell you about how you should +> test an agent? +> +>
+> Answer +> +> It tells you two things. Wording variation is expected and is not automatically +> a defect. Quality or decision variation is exactly why testing an agent by +> running it once and checking the exact output is not reliable. Instead you test +> with *evals*: run many examples, look at the results together, and judge the +> behaviour across the range of inputs. +> +>
+> +> --- +> +> **Q5.** Finish this sentence in your own words: "An agent can draft a reply to +> a stale issue, but it cannot …" +> +>
+> Answer +> +> Any reasonable completion that captures the human-oversight theme from the +> source page, for example: "… post it without a maintainer reviewing and +> approving the proposal." The key idea is that the agent proposes; a person +> confirms. +> +>
+> +> --- +> +> ## Summary +> +> An agent is a loop. A language model at its centre reads text and picks one +> action at a time; tools turn those actions into real effects in the world; the +> loop feeds results back so the model can act again; context is everything the +> model can see. Because the model is probabilistic, agents are tested with evals +> over many examples, not by checking one output once. +> +> These ideas recur in every later lesson. Lesson 2 builds on them directly, +> showing how to drive an agent through a conversation and steer it when it +> starts off in the wrong direction. +> +> --- +> +> ## Next +> +> **Lesson 2 — Working with agents (../working-with-agents.md)**. If the packaged +> lesson wrapper is not available in your copy yet, follow the source page +> directly. +> +> --- +> +> ## Licence +> +> Apache License 2.0 (PRINCIPLE 17). Pages written with help from AI carry a +> `Generated-by:` note in their commit message following ASF Generative Tooling +> Guidance. + +### Exercise answer keys + +**Exercise 1 - Draw the agent loop.** A correct answer (drawn or described) must +include: the user request arriving; the model reading current context; the model +choosing an action (tool, or done); a branch where a tool runs and its result is +added back to context; and a branch where the loop ends and the model answers. +Note any of these five the learner missed. + +**Exercise 2 - Spot the context.** Scenario: a maintainer asks whether a new PR +adds a dependency already flagged as risky; the agent reads the PR body, reads +the known-risky-dependency list, finds a match, and drafts a comment. At the +moment it drafts, context includes at least: the maintainer's question, the PR +body, the known-risky-dependency list, and the match it found. Things a human +might also check that are NOT in context: the full code diff beyond the PR body, +the dependency's real-world reputation or CVE history beyond the local list, +prior project discussion, and the contributor's intent. It matters because the +model can only reason about what is in context. + +**Exercise 3 - Deterministic vs probabilistic.** A good answer says: the model +is probabilistic, so identical inputs can yield slightly different outputs; +wording variation is expected and not automatically broken; and instead of +running it once and checking the exact text, you should evaluate quality and +decisions with evals across many examples. + +**Exercise 4 - What the model cannot do alone.** Two valid items, one sentence +each: (1) take real actions through tools, such as reading a file, opening a +saved comment draft, or opening an issue draft; (2) run a loop across several +steps, feeding tool results back in to refine the work; (3) use information it +was never trained on by reading it into context, such as the current PR or the +project's risk list. Accept any two that match these ideas. + +### Self-check answer keys + +**Q1. What are the four components of an agent?** Model, tools, loop, context. +The model reads and writes text; tools are actions in the world, such as reading +a file, opening a saved comment draft, or opening an issue draft; the loop runs +the model repeatedly and feeds tool results back; context is everything the +model can see at that moment. + +**Q2. A language model on its own is deterministic. True or false?** False. A +model is probabilistic: the same input can produce slightly different outputs. +Determinism is a property of normal code. Agents inherit the probabilistic nature +from their model. + +**Q3. Why can't a model know the contents of a file it has never been shown?** +Because the model can only reason about what is in its context. A file not read +into context has never been seen, so the model can only guess at its contents. +An agent may have a tool that can act on a named file, but the model still +cannot know the file's contents until the relevant content is shown to it. + +**Q4. You run an agent twice on the same input. The wording differs, and one +answer makes a better decision. What does this tell you about testing?** Wording +variation is expected and is not automatically a defect. Quality or decision +variation is why checking one run for an exact output is not reliable. You test +with evals: run many examples, look at the results together, and judge behaviour +across the range. + +**Q5. Finish: "An agent can draft a reply to a stale issue, but it cannot ..."** +Any completion carrying the human-oversight theme is correct, for example: "... +post it without a maintainer reviewing and approving it." The key idea: the +agent proposes, a person confirms. + +### Summary (use at close) + +An agent is a loop. A model at its centre reads text and picks one action at a +time; tools turn those actions into real effects; the loop feeds results back so +the model can act again; context is everything the model can see. Because the +model is probabilistic, agents are tested with evals over many examples, not by +checking one output once. Next: Lesson 2 - Working with agents. diff --git a/docs/education/training/README.md b/docs/education/training/README.md new file mode 100644 index 00000000..a8f14ef4 --- /dev/null +++ b/docs/education/training/README.md @@ -0,0 +1,106 @@ + + + + +**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)* + +- [Apache Training module: Building and running AI agents for open-source projects](#apache-training-module-building-and-running-ai-agents-for-open-source-projects) + - [Who this is for](#who-this-is-for) + - [Relationship to the source pages](#relationship-to-the-source-pages) + - [Module map](#module-map) + - [Delivery formats](#delivery-formats) + - [Prerequisites](#prerequisites) + - [Placeholders](#placeholders) + - [Licence](#licence) + + + +# Apache Training module: Building and running AI agents for open-source projects + +This directory packages the [maintainer-education stream](../README.md) as a +reusable, LMS-neutral **Apache Training module**. Any project — ASF or not — +can use it to *teach* the material in a structured course, not just circulate +the pages as reading. It is shaped for upstream contribution to +[Apache Training](https://training.apache.org/) so the module can live there +once it is stable. + +## Who this is for + +Instructors and facilitators running the education stream as a course: +workshops, on-boarding sessions, reading groups, self-paced curricula, or any +environment where learners need per-lesson structure (objectives, exercises, +assessment) in addition to the reference pages. + +Learners using the module self-paced do not need an LMS. Every lesson is a +plain Markdown file: read it, work through the exercises, and answer the +self-check questions before moving on. + +## Relationship to the source pages + +Each lesson in this module is a **wrapper** around one or more +[progression pages](../README.md). The source pages are the canonical +reference; the lessons add LMS-friendly structure on top: + +- **Learning objectives** — what a learner will be able to *do* after the + lesson, written as observable, assessable outcomes. +- **Exercises** — hands-on activities a learner can complete without access + to a live system (paper/whiteboard activities use the page's own examples). +- **Self-check** — short questions (and answers) the learner uses to gate + themselves before moving to the next lesson. + +Nothing in this directory duplicates the reference material; it only frames it. + +## Module map + +| Lesson | Source page | Learning time | +|---|---|---| +| [Lesson 1 — What agents are](lesson-01-what-agents-are.md) | [What agents are](../what-agents-are.md) | ~30 min | +| Lesson 2 — Working with agents | [Working with agents](../working-with-agents.md) | ~30 min | +| Lesson 3 — Choosing models | [Choosing models](../choosing-models.md) | ~30 min | +| Lesson 4 — Your first skill | [Your first skill](../your-first-skill.md) | ~60 min | +| Lesson 5 — Writing safe skills | [Writing safe skills](../writing-safe-skills.md) | ~45 min | +| Lesson 6 — Debugging a skill | [Debugging a skill](../debugging-skills.md) | ~45 min | +| Lesson 7 — Writing portable skills | [Writing portable skills](../portable-skills.md) | ~30 min | +| Lesson 8 — Eval-driven development | [Eval-driven development](../eval-driven-development.md) | ~60 min | +| Lesson 9 — Agentic and autonomous work | [Agentic and autonomous work](../agentic-work.md) | ~45 min | +| Lesson 10 — English as a programming language | [English as a programming language](../english-as-code.md) | ~30 min | +| Lesson 11 — How to contribute | [How to contribute](../contributing.md) | ~30 min | +| Hands-on lab | [Tutorials](../tutorials.md) | ~90 min | + +> Lessons 2–11 and the lab follow the same format as lesson 1. They are +> added per-sub-item; this file tracks them as placeholders until each one +> lands. + +## Delivery formats + +**Self-paced.** Learners read the source page, then work through the lesson +wrapper (objectives, exercises, self-check) on their own. No instructor or LMS +needed. + +**Instructor-led.** An instructor presents the key ideas from the source page, +assigns the exercises to pairs or small groups, and uses the self-check +questions for a brief group debrief before moving on. The facilitator guide +(planned, not yet shipped) covers room setup, timing, and group discussion +prompts. + +**LMS upload.** Each lesson is a Markdown file that can be converted to SCORM, +xAPI, or any other format a specific LMS supports. The module does not assume +any particular LMS. Learning time estimates above are rough guides for LMS +credit-hour tagging. + +## Prerequisites + +No prior AI experience. Learners should be comfortable reading and writing +plain text, and familiar with the idea of a software project that uses version +control. Specific technical prerequisites are stated in each lesson. + +## Placeholders + +Exercises use `` wherever a real project name would appear. +Substitute your own project name when working through the activities. + +## Licence + +Apache License 2.0 (PRINCIPLE 17). Contributions carry a `Generated-by:` note +in their commit message following ASF Generative Tooling Guidance. diff --git a/docs/education/training/lesson-01-what-agents-are.md b/docs/education/training/lesson-01-what-agents-are.md new file mode 100644 index 00000000..32f54bfe --- /dev/null +++ b/docs/education/training/lesson-01-what-agents-are.md @@ -0,0 +1,233 @@ + + + + +**Table of Contents** *generated with [DocToc](https://github.com/thlorenz/doctoc)* + +- [Lesson 1 — What agents are](#lesson-1--what-agents-are) + - [Learning objectives](#learning-objectives) + - [Prerequisite knowledge](#prerequisite-knowledge) + - [Before the lesson](#before-the-lesson) + - [Exercises](#exercises) + - [Exercise 1 — Draw the agent loop](#exercise-1--draw-the-agent-loop) + - [Exercise 2 — Spot the context](#exercise-2--spot-the-context) + - [Exercise 3 — Deterministic vs probabilistic](#exercise-3--deterministic-vs-probabilistic) + - [Exercise 4 — What the model cannot do alone](#exercise-4--what-the-model-cannot-do-alone) + - [Self-check](#self-check) + - [Summary](#summary) + - [Next](#next) + - [Licence](#licence) + + + +# Lesson 1 — What agents are + +**Source page:** [What agents are](../what-agents-are.md) +**Estimated time:** 30 minutes (15 min reading + 15 min exercises and self-check) +**Lesson in sequence:** 1 of 11 + +--- + +## Learning objectives + +By the end of this lesson you will be able to: + +1. **Define** an AI agent in one sentence using its four components: model, + tools, loop, and context. +2. **Explain** the difference between deterministic and probabilistic behaviour + and give one concrete consequence for how you test an agent. +3. **Identify** at least two things a bare language model *cannot* do on its own + that an agent *can*. +4. **Describe** what "context" means in an agent and why it is finite. +5. **Explain** in your own words why an agent's answer can vary between two + identical requests and why that is not automatically a defect. + +--- + +## Prerequisite knowledge + +None. This is the first lesson in the module. You need only be comfortable +reading English and have a rough idea that "AI" refers to software that +generates text. + +--- + +## Before the lesson + +Read the source page **[What agents are](../what-agents-are.md)** from start to +finish. Pay particular attention to: + +- The "one-sentence version" at the top. +- The comparison table "In normal code / With an agent." +- The "Check your understanding" block at the bottom. + +The exercises below build directly on that page. You do not need to re-read it +during the exercises, but keep it open if you want to look something up. + +--- + +## Exercises + +Work through these alone or in pairs. Plan on 10 to 12 minutes for the +exercises and another few minutes for the self-check. There are no computers +needed: use paper, a whiteboard, or a shared document. + +### Exercise 1 — Draw the agent loop + +On paper or a whiteboard, draw the loop an agent runs. Use boxes and arrows. +Your diagram must include at minimum: + +- The user's request arriving. +- The model reading the current context. +- The model choosing an action (either "use a tool" or "I am done"). +- One branch where a tool runs and its result is added back to context. +- One branch where the loop ends and the model answers. + +Compare your diagram with the text description in the source page (the +"you ask for something → model looks at …" block). What did you miss or +add? The source diagram shows the tool-use path; your diagram should also show +the branch where the model decides it is done and answers. + +### Exercise 2 — Spot the context + +Read the following short scenario, then list everything that is part of the +agent's context at the moment described. + +> A maintainer asks an agent: "Does the new PR add a dependency that we have +> already flagged as risky?" The agent reads the PR body, reads the project's +> known-risky-dependency list, finds a match, and drafts a comment. At the +> moment it drafts the comment, what is in context? + +Write your list. Include at least four items. Then ask: what is *not* in +context that a human reviewer might also check? Does that matter? + +### Exercise 3 — Deterministic vs probabilistic + +A colleague says: "I ran the same PR through the agent twice and got two +slightly different comments. Something must be broken." + +Write two to three sentences explaining to them why this is expected, what the +correct mental model is, and what they should do instead of running it once and +checking the exact output. + +### Exercise 4 — What the model cannot do alone + +You are describing AI agents to a new `` contributor. They ask: "Why +do we need all this machinery? Can't we just ask the AI model directly?" + +If you are reading this outside a project-specific copy, replace `` +with "your project." + +List two things the agent can do that a bare language model called once cannot. +Use the ideas from the source page. Write one sentence per item. + +--- + +## Self-check + +Answer each question in a sentence or two before moving to lesson 2. If you +cannot answer one, re-read the matching section of the source page. + +**Q1.** What are the four components of an agent? (List them.) + +
+Answer + +Model, tools, loop, context. The model reads and writes text. Tools are actions +in the real world (read a file, open a saved comment draft, open an issue +draft). The loop runs the model repeatedly, feeding tool results back in. +Context is everything the model can see at that moment. + +
+ +--- + +**Q2.** A language model on its own is deterministic. True or false? + +
+Answer + +False. A language model is probabilistic: the same input can produce slightly +different outputs each time. Deterministic is the property of normal code (two +plus two is always four). Agents inherit this probabilistic nature from their +model. + +
+ +--- + +**Q3.** Why can't a model know the contents of a file it has never been shown? + +
+Answer + +Because the model can only reason about what is in its context. If a file has +not been read into context, the model has never seen its contents — it can only +guess what might be in it. An agent may have a tool that can act on a named +file, but the model still cannot know the file's contents until the relevant +content is shown to it. + +
+ +--- + +**Q4.** You run an agent twice on the same input. The wording differs, and one +answer makes a better decision. What does this tell you about how you should +test an agent? + +
+Answer + +It tells you two things. Wording variation is expected and is not automatically +a defect. Quality or decision variation is exactly why testing an agent by +running it once and checking the exact output is not reliable. Instead you test +with *evals*: run many examples, look at the results together, and judge the +behaviour across the range of inputs. + +
+ +--- + +**Q5.** Finish this sentence in your own words: "An agent can draft a reply to +a stale issue, but it cannot …" + +
+Answer + +Any reasonable completion that captures the human-oversight theme from the +source page, for example: "… post it without a maintainer reviewing and +approving the proposal." The key idea is that the agent proposes; a person +confirms. + +
+ +--- + +## Summary + +An agent is a loop. A language model at its centre reads text and picks one +action at a time; tools turn those actions into real effects in the world; the +loop feeds results back so the model can act again; context is everything the +model can see. Because the model is probabilistic, agents are tested with evals +over many examples, not by checking one output once. + +These ideas recur in every later lesson. Lesson 2 builds on them directly, +showing how to drive an agent through a conversation and steer it when it +starts off in the wrong direction. + +--- + +## Next + +**[Lesson 2 — Working with agents](../working-with-agents.md)**. If the packaged +lesson wrapper is not available in your copy yet, follow the source page +directly. + +--- + +## Licence + +Apache License 2.0 (PRINCIPLE 17). Pages written with help from AI carry a +`Generated-by:` note in their commit message following ASF Generative Tooling +Guidance. diff --git a/docs/education/what-agents-are.md b/docs/education/what-agents-are.md index 2bc639a9..4b713906 100644 --- a/docs/education/what-agents-are.md +++ b/docs/education/what-agents-are.md @@ -76,7 +76,8 @@ An **agent** wraps the model in two things the model does not have on its own: 1. **Tools**, which are concrete actions in the real world. "Read this file." "Search the code for this word." "Run this command and give me the output." - "Post this comment." Each tool does one job and reports back. + "Open a saved comment draft." "Open an issue draft." Each tool does one job + and reports back. 2. **A loop**, which is the machinery that runs the model again and again. Each time round the loop, the model sees everything that has happened so far and @@ -97,9 +98,12 @@ you ask for something The model is still only reading and writing text. But now some of the text it writes is *"use this tool with these inputs"*, and the loop turns that text into -a real action and feeds the result back. That feedback loop of act, observe, and -act again is what makes an agent more than a chatbot. It can look things up, -check its own work, and correct course when a result surprises it. +a real action and feeds the result back. Which actions are allowed depends on +the tools and permissions you give the agent: a safe teaching setup might only +open a saved comment draft, while a more privileged workflow might post after a +maintainer approves. That feedback loop of act, observe, and act again is what +makes an agent more than a chatbot. It can look things up, check its own work, +and correct course when a result surprises it. ## What the agent can "see": context @@ -128,8 +132,9 @@ almost every tool you have used is built on it. An agent is **probabilistic**. The model does not compute one guaranteed answer. It produces a *likely* one, and "likely" leaves room for variation. Ask the same question twice and you may get two wordings, two orderings, and occasionally two -different decisions. Neither is a bug in the usual sense. It is the nature of the -tool. +different decisions. Wording variation is not automatically a bug. Decision or +quality variation is a signal to test the agent across many examples, because it +may or may not be acceptable for the workflow. Three consequences follow, and they shape everything in this progression: @@ -141,9 +146,9 @@ Three consequences follow, and they shape everything in this progression: | The program is the code | The program is the code *and the words you give it* | You do not need to master the right-hand column yet. The point for now is only -that "it changed its answer" is expected, not broken, and that testing an agent -means running many examples and looking at the results together, not checking -one answer once. +that "it changed its wording" is expected, while "it changed its decision" is +something to evaluate. Testing an agent means running many examples and looking +at the results together, not checking one answer once. ## Why this matters for a maintainer @@ -166,7 +171,7 @@ Before moving on, can you answer these in a sentence each? - What are the two things an agent adds to a bare language model? - What does it mean that the agent can only reason about its *context*? - Why can the same request give a slightly different answer twice, and why is - that not a bug? + that not automatically a bug? If any of those is fuzzy, re-read the matching section. The next page assumes these three ideas.