feat: rebuild thesis-finder as a database-less, university-wide skill#69
Open
dxmme wants to merge 125 commits into
Open
feat: rebuild thesis-finder as a database-less, university-wide skill#69dxmme wants to merge 125 commits into
dxmme wants to merge 125 commits into
Conversation
Captures current PhD rosters for the three pilot chairs as the recall benchmark for PhD discovery (#47): - Autonomous Learning / Distributed Intelligence (Georg Martius) - Theory of Machine Learning (Ulrike von Luxburg) - Autonomous Vision Group (Andreas Geiger) Rosters were captured from each chair's official team page on 2026-06-22 via automated fetch. NOT final: entries still need human verification against the live pages before this is treated as authoritative ground truth. status values active/incoming/associated/ former are documented in the file header; only 'active' counts as a recall target. Postdocs, research engineers and admin staff excluded. Refs #46 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Ground-truth PhD fixture for the 3 pilot chairs drafted and committed as WIP; pending human verification. Refs #46 Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Former PhD alumni are removed from the ground-truth roster entirely. People listed under a chair's 'Researcher' role are now treated as active PhDs (Martius: Kloss, Kolev, Geist), while research engineers remain excluded. Martius roster: 20 active, 0 former. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a 'postdoc' status counted as a recall target. von Luxburg gains 5 postdocs (Bhattacharjee, Bordt, König, Thiessen, Waller) confirmed from the live team page. Martius and Geiger have no current postdoc section, so none added there. Also backfill profile URLs for the three Martius Researchers (Kloss, Kolev, Geist) found on the team page. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Establish the no-DB direction: the skill encodes how Claude interviews and searches the live web instead of relying on a maintained database, working for all Tübingen faculties. - VISION_NO_DB.md: the direction at repo root - findings/no_db_universal_skill/: concept, risks, and the exact A-H build plan - MASTERPLAN.md: rewritten around Phase 1 = university discovery (Tasks A-H) - STATUS.md: reset to track Tasks A-H; old DB phase archived - CLAUDE.md: add small-commit rhythm and end-of-task handoff workflow
- One question per turn (max two only when tightly coupled); never a survey batch - At interview start, ask student to answer precisely and concisely - No search/discovery until profile covers all six dimensions: interests, liked/disliked courses, skills, experience, thesis style, no-gos - Updated Interview Guidance and First Question section to match - Task A complete
Add references/tuebingen-faculty-backbone.md: the official uni-tuebingen.de listing pages for all 7 faculties plus the Center for Islamic Theology, used as an anti-SEO-bias baseline for the discovery skill. Each entry records the listing URL, page language, how chairs are listed, and a last-checked date, plus notes on the faculty -> Fachbereich -> chair nesting and per-department drill-down. Update STATUS.md: mark Task B done and add a dated log line.
…table Add a model-selection table and rule to §7: handoff prompts must now specify which model to use for the next task (Sonnet 4.6 default, Opus only for heavy reasoning/eval interpretation).
… (Task C) Adds `skills/find-university-chairs/references/search-strategy.md`, the core IP that turns a built student profile into precise Tübingen thesis- discovery queries across all faculties. Contains: - Profile dimension → query variable extraction table (6 dimensions) - Faculty routing table (18 interest rows → primary + secondary faculty) - Two-pass strategy: (1) backbone crawl via tuebingen-faculty-backbone.md to get the full structured chair set; (2) live enrichment per chair for relevance, recency, and thesis openings - 18 query skeleton templates in 5 categories (backbone, topic, openings, recency, interfaculty) - Quality filters (source authority, date evidence, specificity) - Dedup rules (chair vs. professor page, interfaculty double-listing) - No-go exclusion table with keyword detection signals - Required output structure (map of options + coverage caveat) - Two worked examples walkable by hand: "Ethical AI in Education" and "Clinical Neuroscience / Neuroimaging" Updates STATUS.md: Task C → ✅.
…scovery skill Replaces the old CS-only seed-list workflow with a two-pass live-search workflow that works across all Tübingen faculties: - description frontmatter is now faculty-agnostic (any discipline) - profile gate: all 6 dimensions required; defers to build-student-profile if shallow - Step 3: faculty routing via references/search-strategy.md §2 - Step 4: Pass 1 backbone crawl via references/tuebingen-faculty-backbone.md - Step 5: Pass 2 live enrichment using query skeletons from search-strategy.md §3–4 - Step 6–7: quality filters, dedup rules, no-go exclusion (search-strategy.md §5–7) - Output: MAP grouped by interest dimension with pros/cons, dated evidence, conversation starter, no-go flags, and honest coverage caveat - Removed all runtime references to seed files (professors/, chairs/, researchers/)
…kills Both skills were designed for the DB-backed approach (OpenAlex index, matching rubric, Celery/FastAPI pipeline). Now that find-university-chairs is fully DB-free (Task D), these are dead code. Git history preserves them.
Professors, chairs, and researchers Markdown moved from skills/find-university-chairs/references/ to skills/tests/eval_ground_truth/cs_seed/. This data was only referenced by the old DB pipeline; find-university-chairs SKILL.md has had no runtime dependency on it since Task D. It now serves as ground truth for Task F's eval (recall measurement for CS faculty).
…nd design-agent-skill find-recent-papers: replaced dead path to professors/INDEX.md with a reference to the find-university-chairs skill (or direct web search). design-agent-skill: removed match-thesis-advisors and update-openalex-paper-index from the Thesis-Finder Skill Pattern (both retired); removed "professor seed index" as primary source (replaced by live web search).
Deleted match-thesis-advisors and update-openalex-paper-index, moved CS seed data to eval ground truth, removed stale seed-path refs.
Hand-curated list of 6 Hertie Institute professors for the sample interest "neurodegenerative diseases and clinical brain research". Verified against HIH research pages (2026-06-27).
…ence interest) Hand-curated list of 6 Psychologie Fachbereich supervisors for "cognitive neuroscience and experimental decision-making". Verified against uni-tuebingen.de Arbeitsbereiche pages (2026-06-27).
…olitical economy) Hand-curated list of 7 supervisors across Institut für Politikwissenschaft and Fachbereich Wirtschaftswissenschaft for the sample interest "comparative politics and political economy". Verified via WiSo faculty pages (2026-06-27).
…scoring guide Defines: recall = surfaced / total ground-truth chairs, ≥70% target, what counts as "surfaced", what the ground truth is NOT, and how to score a skill run step by step. Links CS seed data.
4 faculties ground-truthed (CS via cs_seed, + Medicine, Psychology, WiSo). README with recall metric written. Task G is next open.
New simulation files for Task G: - neuro-student persona: Neurowissenschaften MSc student interested in neurodegenerative diseases (Parkinson's/Alzheimer's, protein aggregation) - medicine-discovery-skill scenario: drives build-student-profile → find-university-chairs completion - medicine-discovery-baseline scenario: same student, plain-Claude arm - neuro-student-skill.json fixture: scripted run surfacing 5/6 HIH chairs - neuro-student-baseline.json fixture: scripted run with no specific chair names - rubric extended with discovery_coverage, discovery_relevance, discovery_structure
New functions in run_codex_multiturn_eval.py: - MEDICINE_GROUND_TRUTH: inline chair list with last-name and relevance labels - load_scenario_by_id(): load any scenario by filename stem - score_coverage(): recall by last-name matching against ground-truth list - score_relevance(): high-relevance fraction of surfaced chairs - score_structure(): binary MAP-format check - run_discovery_comparison(): loads both arms, scores all metrics, writes artifact - _write_comparison_markdown(): markdown table summary - --discovery-comparison CLI flag to run the comparison standalone
Five new test functions: - test_discovery_scenario_resources_exist: checks persona, scenarios, fixtures - test_coverage_scoring_detects_chair_names: unit test for score_coverage() - test_skill_arm_fixture_has_higher_coverage_than_baseline: asserts skill > baseline - test_skill_arm_fixture_passes_structure_check: asserts MAP format in skill run - test_discovery_comparison_writes_artifact: end-to-end comparison artifact check
Fixture run: skill arm 83% recall (5/6 HIH chairs surfaced), baseline 0% — gap +83pp on medicine faculty ground truth.
psych-student-skill: 6-turn scripted conversation with MAP output naming all 6 Fachbereich Psychologie chairs (Bartels, Nürk, Friedrich, Huff, Svaldi, Gawrilow). psych-student-baseline: vague 4-turn response with no chair names — 0% recall.
wiso-student-skill: 6-turn scripted conversation naming all 7 WiSo chairs (Diez, Abels, Schlumberger, Hasenclever, Bieling, Müller, Baten). wiso-student-baseline: vague 4-turn response with 0% recall.
cs-student-skill: 6-turn scripted conversation naming all 7 CS/ML researchers (Martius, Hennig, Hein, von Luxburg, Schölkopf, Brendel, Bethge). cs-student-baseline: vague 4-turn response with 0% recall.
Add PSYCHOLOGY_, WISO_, CS_GROUND_TRUTH constants and FACULTY_CONFIGS dict. Make score_structure() accept optional ground_truth param (backward-compatible). Add _run_single_faculty_comparison(), _write_faculty_comparison_markdown(), run_all_faculties_comparison(), _write_all_faculties_summary(). Update --discovery-comparison to run all 4 faculties instead of medicine only. 12/12 tests still pass.
…tness numbers to date Pure aggregation, no new live run: one document collecting every eval number produced so far (Task H/I/I-fix, Phase-2 2-E, Roadmap-J run 1, Task O re-run, Task P, Task Q run 1, Task R), one table per roadmap quality axis, each cell cited to its source file. Headline finding stated plainly: the roadmap's "core is done" recall bar (>=80% across >=6 faculties incl. one hard faculty) is not yet met - only 5 faculties have a live number, and the one hard faculty tested (Humanities, 60%) is below bar, root-caused to an eval-protocol gap in the test persona rather than a skill defect. Law/Theology have ground truth but no live run yet; Task S has zero data points. Also catches up docs/thesis-report/03 and 04 with parallel-session progress that had landed since they were written: Task Q's blind run and Task R both actually completed (previously described as deferred/not-started).
Task S found that 2 of 5 live discovery transcripts (cs-skill, wiso-skill) omitted the required "Dated evidence" field from every option in the map. Adding a concrete worked example makes the expected field set unambiguous instead of relying on a bare bullet list.
Reviewed the 5 full live discovery transcripts against the roadmap's 5 output-quality criteria. 4/5 pass cleanly; found one repeated gap (dated evidence omitted in 2/5 transcripts, fixed in the prior commit). Ran a live happy-path interview simulation for criterion 5 since no such transcript existed in the repo — 6-turn clean convergence, contrasted with Task R edge case 2's 8-turn adversarial non-convergence.
…rd faculties Explicit go/no-go over existing eval evidence (no new live run) now that Tracks 1-4 are complete. Scores roadmap §4's five 'core is done' criteria: steering (Task P), output quality (Task S), edge-case robustness (Task R) all met; precision strong but under-sampled; recall criterion NOT met on two independent counts — only 5 faculties measured (not >=6), and no hard faculty has a clean live >=80% (Humanities 60%, protocol-caveated). The Humanities shortfall is root-caused to the eval protocol (personas built from the README's lossy one-line summaries, not the GT files' full sample interest), not a skill defect — the crawl found all 5 GT chairs. Not converted to GO: prior 'gate GREEN'-on-partial-evidence burn, and no schedule pressure since Phase 2 is already GREEN. Names Task T (protocol fix + blind re-run Humanities + blind-run Law) as the closeout. Decision doc + STATUS row/log/phase note added. No skill files touched.
…y README summary Reconcile runbook steps 2/3: the persona must be built from each GT file's full 'Sample interest:' line (extracted via grep, which does not reveal the chair list), not the README's one-line summary table. The README summaries drop qualifying clauses and were the documented root cause of the Humanities 60% run (dropped 'history of the field' clause -> spurious historical-work no-go excluding 2 found chairs). No-peeking now has an explicit carve-out for the single sample-interest line. Eval-harness fix, not a skill change.
…core (100%) Law blind run under corrected persona protocol: recall 3/5=60%, precision 3/3=100%. Discovery found all 5 GT chairs; the 60% is a downstream filter miss. Root cause is a real skill finding: Remmert excluded at the topical-justification step from her multi-strand title without Pass-2 enrichment, but her actual focus (Allgemeine Grundrechtslehren) is a core constitutional-law/human-rights match -> false-negative. Names Task U (enrich-before-exclude rule + re-run Law). Humanities corrected re-score (transparent, un-blind): 3/5->5/5=100%. Both prior misses (Corcilius, Doering) flip to Include once the full sample interest restores the dropped clauses. Validates the protocol fix.
…% -> Task U Original Humanities blocker resolved (60%->100% under fixed persona protocol, validating the eval-protocol fix). Blind Law run surfaced a new real defect: Remmert excluded at the topical-justification step without Pass-2 enrichment despite a core constitutional-law/human-rights focus (Allgemeine Grundrechtslehren). 5 of 6 faculties now clear 80%; Law is the sole miss. NO-GO stands (transformed). Names Task U (add a §5 enrich-before-exclude rule + re-run Law -> flip to GO if >=80%). Updates decision-doc banner, STATUS Task T/U rows + dated log; adds task-t-recall-closeout findings entry.
Task U. Symmetric dual of the Butz over-inclusion worked example: a chair whose own title names a core-interest field (amid off-interest strands) must be Pass-2 enriched before being excluded at the topical- justification step. Title-only exclusion produced a Task T false- negative on Prof. Barbara Remmert (Law) — her dense multi-strand title reads economic/municipal law, but her actual Schwerpunkte (Allgemeine Grundrechtslehren) are a core constitutional-law match. Adds the rule to search-strategy.md §5 plus a matching one-liner to SKILL.md Step 7.
Blind re-run of Law under the §5 enrich-before-exclude fix: 4/5 = 80% recall (Remmert now surfaces via Pass-2 enrichment of her Schwerpunkte, Saurer remains an honest defensible miss), 4/4 = 100% precision. All 6 measured faculties now clear the 80% recall bar (2 of them hard), so roadmap §4 criterion 1 is met. Flips the go/no-go verdict to GO and updates the aggregate scorecard and STATUS.md to match.
…s open MASTERPLAN.md's Phase 4 section still read "current"; Task T+U closed the recall gap and flipped the roadmap §4 go/no-go to GO on 2026-07-04. Updated the phase heading and added pointers to the go-no-go verdict and the aggregate scorecard. Added an explicit "Phase 5 - not yet scoped" note since Task U was the last task on the critical path and no next task is named yet - this is a real open decision point, not an oversight. The eval-aggregate-scorecard's Task S section still said "not measured, zero data points" - stale since Task S actually completed (4/5 criteria pass) before Task T/U even ran. Corrected to match STATUS.md's Task S row.
…p analysis Researches all 7 faculties + ZITh live (not the CS-only degree-programs.md the task brief pointed at, which turned out to only cover Informatik) and clusters them into a 13-field taxonomy. Gap analysis shows which fields are underserved in bw-company-backbone.md before curation.
…lties Adds 6 new sections (Chemie/Materialwissenschaft, extended Umwelt/Energie/ Geowissenschaften, Wirtschaft/Consulting/Versicherung, Sozialwissenschaften/ Marktforschung, Bildung/EdTech, Medien/Verlage/Sprache, Sport/Gesundheits- technologie) with 16 new live-verified entries, cross-tags 4 existing rows (Bosch, Bosch eBike Systems, Mercedes-Benz for ux-research; existing HR-tech/ edtech/consulting tags already sufficient elsewhere), adds a Thesis-Kultur column across all 13 tables, and adds a Studiengangs-Routing table mapping all 13 employment fields to backbone sections/tags. Underserved fields (Recht/Legal Tech, Theologie/Philosophie, Sozialwissenschaften) are documented as honest gaps rather than padded, per the anti-bias rules. company-search-strategy.md §2.1 and SKILL.md Step 3 updated to reference the new field-tag vocabulary and the Studiengangs-Routing table.
…ke test GFT and Haufe rows were missing the consulting/verlag tags the new Studiengangs-Routing table claimed they had — added them so the table's claims match the actual backbone data. Also records the honest EdTech persona smoke test result (2-3 candidates, below the 5-20 target) in the Task L findings doc, and confirms the existing SKILL.md disambiguation fallback already handles that case correctly.
…faculties) Records the merge of task/L-company-backbone into feat/no-db-universal-skill: 13-field employment taxonomy, 16 new live-verified backbone entries across 6 new sections, Thesis-Kultur column, Studiengangs-Routing table, and honest documentation of fields that stayed thin despite real research effort.
…en-work An untracked root file with raw notes from an informal test session (one external tester + Domi's own follow-up ideas) needed a home before the branch could be pushed clean. Moved to docs/thesis-report/04-open-work/ with a dated filename and a synthesis paragraph in the section README: explicitly flagged as informal (no protocol, no ground truth, one tester) so it isn't mistaken for a designed eval run, but kept as the first real data point toward the still-open independent-validation item, with three concrete product-idea signals extracted for whoever picks that up next.
…er next Project is being set aside for a while. Added an explicit pause banner at the top of "Current phase" as the designated re-entry point: exact git/test state, phase summary, and the concrete next task (work through docs/thesis-report/04-open-work/2026-07-04-feedback-gehler.md's three extracted signals) named up front so resuming doesn't require re-deriving context from the full task history. No skill logic touched - documentation only.
Add a one-time framing message before Step N1's interview begins, explaining what the skill does, that answer detail is the student's choice, and that the track choice comes after the profile is built. Grepped for Master-specific wording in thesis-finder/SKILL.md — none found, so no further fixes needed there.
Before drafting a first-contact email, ensure 1-2 relevant recent papers from the target person/lab are surfaced (via find-recent-papers if not already available) and recommend the student skim them first, so contact doesn't read as generic.
Task M2 (recommend + drill-down in thesis-finder) is next.
Adds a recommend-and-drill-down step between the option map delivery and the draft-thesis-contact offer, in both New User (N5, renumbering the offer to N6) and Returning User (R6, renumbering the offer to R7) flows. Reuses the find-recent-papers pattern from draft-thesis-contact (Task M3) for the drill-down; recommendation and drill-down summary are grounded only in the option map's existing fields and fetched papers, no invented facts. Addresses the Gehler feedback signal "tell me which one you would choose and walk a bit more down that road."
The Task M plan file existed on disk since 2026-07-05 but was never added to git despite being linked from STATUS.md and two commit messages. Adding it alongside the new independent 1.0-readiness review, which assesses the project's 1.0 claim for both the thesis committee and real students and scopes the remaining path as Phase 5 (see MASTERPLAN.md / STATUS.md follow-up commits).
…V-AA) The 2026-07-05 independent 1.0-readiness review found the two hard-faculty recall numbers behind the 2026-07-04 GO verdict (Humanities, Law) are not clean blind measurements. This does not reverse the GO (the underlying Task U skill fix is real and independently corroborated) but flags it provisional pending one clean blind hard-faculty run. Phase 5 was previously "not yet scoped" with three undecided candidates. Replaced with a concrete task table (Tasks V-AA) derived from the review: blind Theology run, the scope-erosion experiment, independent scoring, non-circular company ground truth, a protocolled external test plus distribution, and a hygiene sweep. Old §9 renumbered to §10.
Threads the 2026-07-05 independent review through STATUS.md: updated the "Last update" line, rewrote the "Current phase" re-entry banner to summarize the review's five findings and point at Phase 5 Tasks V-AA, added the provisional-GO caveat to the Phase 4/5 summary paragraphs, added a Phase 5 task table (Tasks V-AA) mirroring the existing Phase 2/3 table format, and appended a dated log entry describing what was actioned and why. Project is re-paused here; recommended next task on resume is Task V, then Task W. No skill, reference, or eval-artifact files were touched (read-only review per its own mandate). pytest -q (29 passed/8 skipped) and build_skill_release.py re-verified green after this pass.
…review docs/thesis-report/03 and 04 still described a stale 2026-07-03 snapshot (recall bar "not yet met", Task S "no data at all yet") that had been superseded by Tasks T/U/S but never synced into the thesis-facing narrative — a real inconsistency a committee reader would trip over. Corrected both to state the current GO-provisional status directly, with the specific blindness caveat (Humanities re-score, Law contaminated re-run) inline instead of only in a linked findings file. Also: noted that Gehler feedback signals (1) and (2) were since implemented (Tasks M1/M2, not yet live-re-verified); added the review's dose-response reframing of the scope-erosion argument to the scope-question section; updated decision-log.md with the Task T/U GO flip and the two 2026-07-05 review decisions; linked the review from the top-level reading table and synthesis paragraph.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Rebuilds the thesis-finder tool from a curated-database product into a portable, database-less Agent Skill: it interviews the student for a structured profile and then searches the live web (guided by Markdown reference backbones) to map thesis options across all University of Tübingen faculties and BW companies. Core principle: no runtime DB, no backend — the skill stays correct because it reads the live web instead of a store that rots.
What Changed
find-university-chairsskill (faculty-agnostic, 6-dimension profile gate, two-pass search: backbone crawl + live enrichment),build-student-profileconversation discipline, faculty backbone + search-strategy references, eval ground truth for 4 faculties (CS, Medicine, Psychology, WiSo), retired the old DB-backed skills (match-thesis-advisors,update-openalex-paper-index).find-company-thesis-optionsskill with a BW company backbone (~107 entries across 7 sectors, Cyber Valley + manual R&D additions), company search strategy, eval ground truth for 3 student profiles.thesis-finder/SKILL.mdas the single entry point (interviews once, routes to chairs / companies / both), updatedAGENTS.mdandREADME.mdfor distribution, end-to-end smoke test, session-state persistence across searches.Test Plan
pytestfull suite green (seeSTATUS.mddated log for gate history)Known Issues / Not Yet Done
findings/no_db_universal_skill/2026-06-28-I-fix-revalidation.mdSTATUS.mdfor full status andMASTERPLAN.mdfor the overall plan