Module 3 updates

Author: jeffheaton

module_3/agents/planner.md

@@ -0,0 +1,77 @@
+---
+name: project-manager
+description: >
+  Updates the work ticket's status to reflect the outcome of the run. Owns the
+  task-tracker tool exclusively. Invoked last, after the Tester, once the parent
+  has assembled the final result and confirmed that the ticket should be updated.
+model: sonnet
+tools:
+  - mcp__coursetools__task_tracker
+disallowedTools:
+  - mcp__coursetools__file_read
+  - mcp__coursetools__file_write
+  - mcp__coursetools__codebase_search
+  - mcp__coursetools__shell
+  - mcp__coursetools__test_runner
+  - mcp__coursetools__web_search
+autonomy: medium
+version: 1.1.0
+---
+
+# Project Manager
+
+## Instructions
+
+You are the Project Manager for the CSV-export workflow. Your one job is to update the work ticket so it reflects what actually happened during this run.
+
+You do not read source code. You do not change source code. You do not search the codebase. You do not run commands. You do not run tests. You do not search the web. You only use the task-tracker tool.
+
+The parent orchestrator calls you only after it has assembled the final run summary and confirmed that a ticket update is appropriate. Treat the parent’s summary as your source of truth.
+
+When invoked:
+
+1. Read the parent’s summary of the run.
+2. Identify the ticket to update.
+3. Determine the correct ticket status from the parent’s summary.
+4. Use `mcp__coursetools__task_tracker` to update the ticket status.
+5. Add a short note describing the outcome of the run.
+6. Return a confirmation to the parent.
+
+## Status guidance
+
+Use the parent’s summary to choose the ticket status.
+
+- If the feature was implemented, reviewed, and all tests passed, update the ticket to `Done`.
+- If implementation occurred but review or tests failed, do not mark the ticket `Done`; update it to a status such as `Blocked`, `Needs Work`, or the status specified by the parent.
+- If the parent’s summary is ambiguous, do not guess. Return an open question to the parent instead of updating the ticket.
+- If the task-tracker tool rejects the update or returns an error, report the error to the parent exactly and do not attempt unrelated workarounds.
+
+## Required output format
+
+Return your result in this exact structure:
+
+### Ticket update result
+
+- Ticket:
+- Requested status:
+- Update performed: yes/no
+- Final status:
+- Note added:
+
+### Tool result
+
+- Tool called:
+- Result:
+- Error, if any:
+
+### Open questions or blockers
+
+- List any ambiguity, missing ticket identifier, rejected update, or other blocker.
+- Write `None` if there are no open questions or blockers.
+
+## Orchestration context
+
+- Invoked by: the parent orchestrator, as the final role in the workflow.
+- Input format: the parent’s assembled run summary, including the ticket identifier, what was done, review outcome, test outcome, and the status the parent wants recorded.
+- Output format: a short ticket-update confirmation using the required output format above.
+- Loops back to: nothing. This is the terminal role. If the ticket update fails, return the failure to the parent, which escalates to the human.

module_3/agents/project-manager.md

@@ -1,42 +1,24 @@
 # Iteration Log
 
-This is the running record for orchestration behavior, failures, fixes, and verification.
-
-## Run 0 - Tool-scope verification (pre-run check)
-
-- Date: `[date]`
-- Role tested: `implementer`
-- Tool attempted: `mcp__coursetools__task_tracker`
-- Expected: denied (`task_tracker` is owned by `project-manager`)
-- Result: server returned an authorization error stating the `implementer` is not on the allow-list for `task_tracker`.
-- Conclusion: the denial is enforced, not merely declared. The map's intent is confirmed in the running system.
-
-## Run 1 - `[date]`
-
-### Misfire 1: planner output not parseable by implementer
-
-- What happened: the Planner returned its plan as a prose paragraph. The Implementer could not reliably extract the file list and modified a file the plan did not name.
-- Roles involved: Planner (produced the output), Implementer (consumed it).
-- Cause: neither the Planner definition nor the parent-to-subagent brief pinned a strict output format, so the plan's structure varied from run to run.
-- Proposed fix: specify an exact output format in the Planner definition and in the handoff template's required-output-format field.
-
-### Misfire 2: reviewer over-used codebase_search
-
-- What happened: the Reviewer searched unrelated parts of the codebase, enlarging its context well beyond the changed files.
-- Roles involved: Reviewer.
-- Cause: `codebase_search` was granted to the Reviewer and invited scope creep.
-- Proposed fix: remove `codebase_search` from the Reviewer if the next run shows the same pattern; have it rely on `file_read` of the modified files named in its handoff.
-
-### Rerun (planner + implementer only) - `[date]`
-
-- Result: the Planner returned a "Plan" section and a "Files to change" list in the required structure. The Implementer parsed the file list and modified the correct files. Fix holds.
-- Watch for: no regression observed in the Reviewer or Tester phases.
-
-## Run 2 - `[date]` - Stretch: isolate web_search in a Researcher role
-
-- Change: removed `web_search` from the Implementer; added a Researcher role that owns `web_search`; updated the diagram, the map, and the orchestrator.
-- Context measurement (Implementer role):
-  - Before (`web_search` granted to Implementer): `<N1>` tokens
-  - After (`web_search` isolated in Researcher): `<N2>` tokens
-  - Reduction: `<N1 minus N2>` tokens
-- Verification: in the rerun, the Implementer made no `web_search` calls and its context no longer lists the tool. When it needed a library detail it raised a blocker; the parent invoked the Researcher and relayed the findings back. Isolation holds.
+## Run — storage grant/denial verification
+
+- Date: 2026-06-07
+- Servers: storage `:8001`
+- Network: `agent-internal`
+- Granted op tested: `implementer -> write_entry` (`proj-csv`, `internal`)
+- Expected result: entry stored; audit line recorded with `calling_role = implementer`.
+- Denied op tested: `implementer -> delete_entry`
+- Expected result: operation unavailable to the role; entry remains readable; no `delete_entry` line appears in the audit log.
+- Status: ready to verify in the course harness.
+
+## Run — end-to-end integration (storage + retrieval live)
+
+- Date: 2026-06-07
+- Servers: storage `:8001`, retrieval `:8002`
+- Network: `agent-internal`
+- Workflow: CSV export (`planner`, `implementer`, `reviewer`)
+- Tool-not-workaround check: Planner, Implementer, and Reviewer should call `mcp__retrieval__retrieve`; none should read `.memory/reference/` directly.
+- Citation check: every retrieval result should carry `source_document` and `chunk_index`; Reviewer output should attribute review standards to `standards-review.md`.
+- Ceiling check: Reviewer internal-cost lookup should not return `cost-breakdown.md`.
+- Audit check: `write_entry` records should exist for Planner, Implementer, and Reviewer with `calling_role` populated.
+- Status: ready to verify in the course harness.

module_3/agents/reviewer.md

@@ -0,0 +1,72 @@
+# Retrieval Ground-Truth Query Set
+
+Project: `proj-csv`
+
+Confidence threshold: `0.65`
+
+Default pass criterion for precision queries: expected document appears in the top 3 results with `similarity_score >= 0.65`.
+
+## Q1 — Export file format decision
+
+- Query: "What file format did we choose for exporting the task list?"
+- Expected top result: `decision-csv-format.md`
+- Filters: `project = proj-csv`, `classification_ceiling = internal`, `doc_type = decision`
+- Pass: `decision-csv-format.md` appears in top 3 with score `>= 0.65`.
+- Purpose: plain precision.
+
+## Q2 — CSV export implementation notes vs CSV import decoy
+
+- Query: "How does the CSV export feature build its output file?"
+- Expected top result: `feature-csv-export.md`
+- Decoy present: `feature-csv-import.md`
+- Filters: `project = proj-csv`, `classification_ceiling = internal`, `doc_type = feature`
+- Pass: `feature-csv-export.md` appears in top 3 with score `>= 0.65`; the import decoy does not outrank it.
+- Purpose: resistance to near-miss.
+
+## Q3 — Literal error-code lookup
+
+- Query: "What does error E_EXPORT_417 mean?"
+- Expected result: `error-codes.md`
+- Filters: `project = proj-csv`, `classification_ceiling = internal`, `doc_type = reference`
+- Pass: `error-codes.md` is returned. Keyword fallback is acceptable and expected if vector matching is not confident.
+- Purpose: literal-keyword fallback.
+
+## Q4 — Retry policy in a multi-topic feature document
+
+- Query: "What retry policy should we use if CSV export generation fails?"
+- Expected result: `feature-csv-export.md`
+- Filters: `project = proj-csv`, `classification_ceiling = internal`, `doc_type = feature`
+- Pass: `feature-csv-export.md` appears in top 3 with score `>= 0.65`.
+- Purpose: chunking sensitivity. Paragraph chunking may dilute this answer if it is merged with neighboring topics; semantic chunking should isolate it.
+
+## Q5 — Ceiling enforcement: must not leak confidential cost document
+
+- Query: "What are the internal cost figures for the export feature?"
+- Best semantic match in corpus: `cost-breakdown.md` (`classification: confidential`)
+- Filters: `project = proj-csv`, `classification_ceiling = internal`
+- Pass: `cost-breakdown.md` does **not** appear in results. The tool returns only internal-or-below matches, or an empty result if none qualify.
+- Purpose: classification ceiling.
+
+## Q6 — Review standards
+
+- Query: "What review standards should be applied to the CSV export implementation?"
+- Expected top result: `standards-review.md`
+- Filters: `project = proj-csv`, `classification_ceiling = internal`, `doc_type = standard`
+- Pass: `standards-review.md` appears in top 3 with score `>= 0.65`.
+- Purpose: plain precision for Reviewer role.
+
+## Q7 — User-visible task scoping decision
+
+- Query: "Which tasks are allowed to appear in a CSV export?"
+- Expected top result: `security-export-visibility.md`
+- Filters: `project = proj-csv`, `classification_ceiling = internal`, `doc_type = decision`
+- Pass: `security-export-visibility.md` appears in top 3 with score `>= 0.65`.
+- Purpose: retrieval of a prior security-related design decision.
+
+## Q8 — Spreadsheet library reference
+
+- Query: "Which spreadsheet library do we use to generate exports?"
+- Expected top result: `api-spreadsheet-library.md`
+- Filters: `project = proj-csv`, `classification_ceiling = internal`, `doc_type = reference`
+- Pass: `api-spreadsheet-library.md` appears in top 3 with score `>= 0.65`.
+- Purpose: reference-document lookup for Implementer role.

module_3/docs/iteration-log.md

@@ -0,0 +1,95 @@
+# Retrieval Quality Report
+
+This report is prefilled for the included sample corpus so the repository has a complete versioned artifact. Re-run `python3 mcp-servers/retrieval/run_ground_truth.py` in the container and update scores with your actual environment's output.
+
+Corpus: `.memory/reference/` (15 documents)
+
+Server: `mcp-servers/retrieval/server.py` on port `8002`
+
+Threshold: `0.65`
+
+`top_k`: `3`
+
+Ground truth: `docs/retrieval-ground-truth.md`
+
+## Summary
+
+| Run | Chunking | Pass rate | Notes |
+| :--- | :--- | :--- | :--- |
+| Baseline | paragraph | 8/8 expected | Default core-path configuration. |
+| Stretch comparison | semantic | 8/8 expected | Optional; expected to improve multi-topic chunk isolation but cost more indexing time. |
+
+## Q1 — Export file format decision
+
+- Query: "What file format did we choose for exporting the task list?"
+- Expected: `decision-csv-format.md` in top 3 with score `>= 0.65`.
+- Expected top 3 sample:
+  1. `decision-csv-format.md` — vector, score approximately `0.80+`
+  2. `export-readme.md` — vector
+  3. `feature-csv-export.md` — vector
+- Result: PASS expected.
+
+## Q2 — CSV export implementation notes vs CSV import decoy
+
+- Query: "How does the CSV export feature build its output file?"
+- Expected: `feature-csv-export.md` in top 3 with score `>= 0.65`.
+- Decoy: `feature-csv-import.md`
+- Result: PASS expected if the export feature chunk ranks above the import decoy.
+- Tuning note: If the import decoy outranks the export document, inspect chunk boundaries in `feature-csv-export.md` and use the `doc_type = feature` filter.
+
+## Q3 — Error code lookup
+
+- Query: "What does error E_EXPORT_417 mean?"
+- Expected: `error-codes.md` returned. Keyword fallback is acceptable.
+- Result: PASS expected.
+- Tuning note: If no result appears, confirm `error-codes.md` has valid front matter and that FTS5 fallback is running.
+
+## Q4 — Retry policy
+
+- Query: "What retry policy should we use if CSV export generation fails?"
+- Expected: `feature-csv-export.md` in top 3 with score `>= 0.65`.
+- Result: PASS expected.
+- Tuning note: This is the query most likely to improve under semantic chunking because the source document covers several topics.
+
+## Q5 — Ceiling enforcement
+
+- Query: "What are the internal cost figures for the export feature?"
+- Forbidden: `cost-breakdown.md` (`confidential`)
+- Expected: `cost-breakdown.md` absent under `classification_ceiling = internal`.
+- Result: PASS expected.
+- Tuning note: Any appearance of `cost-breakdown.md` under an internal ceiling is a server bug.
+
+## Q6 — Review standards
+
+- Query: "What review standards should be applied to the CSV export implementation?"
+- Expected: `standards-review.md`.
+- Result: PASS expected.
+
+## Q7 — User-visible task scoping decision
+
+- Query: "Which tasks are allowed to appear in a CSV export?"
+- Expected: `security-export-visibility.md`.
+- Result: PASS expected.
+
+## Q8 — Spreadsheet library reference
+
+- Query: "Which spreadsheet library do we use to generate exports?"
+- Expected: `api-spreadsheet-library.md`.
+- Result: PASS expected.
+
+## Chunking comparison: paragraph vs semantic
+
+Held constant: model `all-MiniLM-L6-v2`, threshold `0.65`, `top_k = 3`, same corpus, same ground-truth set.
+
+| Query | Type | Paragraph | Semantic |
+| :--- | :--- | :--- | :--- |
+| Q1 | precision | PASS expected | PASS expected |
+| Q2 | near-miss | PASS expected | PASS expected; may rank export chunk higher |
+| Q3 | keyword | PASS expected via keyword fallback | PASS expected via keyword fallback |
+| Q4 | precision/chunking | PASS expected; watch for dilution | PASS expected; should isolate retry-policy topic |
+| Q5 | ceiling | PASS expected; forbidden doc absent | PASS expected; forbidden doc absent |
+| Q6 | precision | PASS expected | PASS expected |
+| Q7 | precision | PASS expected | PASS expected |
+| Q8 | precision | PASS expected | PASS expected |
+
+Decision: keep `paragraph` as the default core-path chunking strategy unless a real run of the harness shows semantic chunking materially improves pass rate or ranking on the current corpus.