Capture Claude's text narration from transcript file

Adds the "film room" missing piece: Claude's decisions and framing
between tool calls, read from the Claude Code transcript JSONL at every
Stop / SubagentStop / SessionEnd.

- migrations/003_messages.py creates `messages` table (PK: message_id +
  block_index, idempotent ingest)
- src/cc_logger/transcripts.py reads transcript JSONL and extracts only
  `text` blocks. Extended `thinking` blocks are encrypted by Anthropic
  (signature only, no plaintext) — that's an API-level choice we can't
  work around.
- New Stop handler + transcript ingestion called from Stop, SubagentStop,
  and SessionEnd for live + reconciliation capture
- install-hooks.py now installs a Stop hook too
- inspect.py interleaves text blocks (· prefix) with tool calls so you
  can see "what Claude was thinking out loud" inline with "what it did"
- docs/SCHEMA.md + docs/HOOKS.md updated; 8 new unit tests for transcript
  parsing (skips thinking blocks, handles malformed lines, etc.)

Author: kkrlstrm

README.md

@@ -19,23 +19,29 @@ SESSION 94b8ee2b-b51f-4125-a116-82adaf4066af
            and propose which personas are reachable in each week of June-August.'
 
   [root  completed]
+    · 'I'll start by pulling the campaign performance and matching it against
+       district fiscal data, then fan out to verify against state sources.'
     Bash       'glab api "groups/12345/projects?search=accounts"'              3.0s  ok
     Bash       'psql ... -c "SELECT campaign, replies FROM eb_campaigns ..."'  2.0s  ok
     Bash       'ls /Users/me/Downloads/k12-district-fiscal-sustainability'      0ms  ok
+    · 'Got the baseline. Now I'll spawn three sub-agents in parallel: one for
+       NY state data, one for Ohio, one for the cross-state academic calendar.'
     Agent      'Map district fiscal sustainability data'                       38s   ok
       [general-purpose ab72bd109071caf  completed]
+        · 'Searching state Comptroller / Auditor databases for fiscal-stress designations.'
         WebSearch  'K-12 district fiscal stress New York Comptroller 2024 2025'  5.9s  ok
         WebFetch   'https://www.osc.ny.gov/state-agencies/audits/fiscal-stress'  8.2s  ok
         WebSearch  'Ohio Auditor school district fiscal distress 2024'          6.1s  ok
         WebFetch   'https://ohioauditor.gov/auditsearch/Reports/2024'           64s   FAIL
+        · 'Ohio Auditor blocked the fetch. Falling back to Comptroller summary.'
         ... 47 more tool calls
         → 'Found 31 districts in NY designated fiscal stress, 12 in OH...'
     Agent      'Cross-reference academic calendars by state'                  5m 55s ok
       [general-purpose ab9dcc9453675d9  completed]
         ... 64 tool calls
     ... 7 more sub-agents
 
-  10 invocations (9 sub-agents), 556 tool calls (24 failed, 0 pending)
+  10 invocations (9 sub-agents), 556 tool calls (24 failed, 0 pending), 142 text blocks
 ```
 
 `cc-logger insights` adds the cross-session view — power-law distribution of where your time goes, top failure domains, sub-agent fan-out patterns, hourly activity.
@@ -74,6 +80,7 @@ python scripts/install-hooks.py     # wires the Claude Code hooks
 - Every sub-agent invocation (root + children, with linkage to the spawning `Agent` tool call)
 - Every tool call in the capture allowlist (Agent, Bash, Edit, Write, WebFetch, WebSearch, and `mcp__.*`)
 - Tool input + tool response payloads as JSONB; anything >50KB spills to a separate `artifacts` table
+- **Claude's text narration between tool calls** — read from the Claude Code transcript file at every `Stop` / `SubagentStop`, stored in the `messages` table. (Extended `thinking` blocks are encrypted by Anthropic — only `text` blocks are capturable.)
 - Optional regex redaction of common secret patterns before write (on by default)
 
 ## Privacy

docs/HOOKS.md

@@ -14,8 +14,9 @@ Official Claude Code hooks documentation: https://code.claude.com/docs/en/hooks.
 | `PostToolUse` | After a tool succeeds. **Filtered**. | Updates the matching `tool_calls` row with the response, `status='success'`, and duration. Spills payloads >50KB to `artifacts`. |
 | `PostToolUseFailure` | When a tool fails. **Filtered**. | Updates the matching `tool_calls` row with `error`, `status='failure'`, duration. |
 | `SubagentStart` | Sub-agent spawned. | Inserts an `agent_invocations` row. Resolves the parent `Agent` tool_call by `subagent_type` match. |
-| `SubagentStop` | Sub-agent finishes. | Updates the matching `agent_invocations` row with `last_message`, `ended_at`, `status='completed'`. |
-| `SessionEnd` | Session ends (exit, logout, kill, etc.). | Updates `sessions.ended_at` + `end_reason`. Sweeps any still-pending `tool_calls` and `agent_invocations` for this session to `orphaned`. |
+| `SubagentStop` | Sub-agent finishes. | Updates the matching `agent_invocations` row with `last_message`, `ended_at`, `status='completed'`. **Also reads the sub-agent's transcript file and ingests `text` blocks into the `messages` table.** |
+| `Stop` | Root agent finishes a turn (one response to one user message). | Reads the root transcript file at `transcript_path`, extracts assistant `text` blocks, INSERTs them into `messages` (idempotent on `message_id` + `block_index`). This is where Claude's mid-process narration lands. |
+| `SessionEnd` | Session ends (exit, logout, kill, etc.). | Updates `sessions.ended_at` + `end_reason`. Sweeps any still-pending `tool_calls` and `agent_invocations` for this session to `orphaned`. Also does a final transcript ingestion pass to catch anything `Stop` missed. |
 
 ## Tool capture allowlist
 
@@ -47,6 +48,17 @@ cc-logger resolves the link by matching on `subagent_type`:
 
 In practice, Claude Code emits hook events sequentially even when sub-agents execute in parallel, so the multi-candidate case is rare.
 
+## Transcript-based message capture
+
+Hooks don't include Claude's narration text in their payloads — they only fire at action boundaries. To capture the *decisions* Claude is making mid-process (e.g., "I'll start by exploring the project structure..."), cc-logger reads the Claude Code JSONL transcript file at `transcript_path` (a field present in every hook event).
+
+- On `Stop` / `SubagentStop`: incremental ingest, near-realtime
+- On `SessionEnd`: final reconciliation pass
+
+Only `text` blocks are extracted. Claude's `thinking` (extended thinking) blocks are encrypted in the transcript by Anthropic — only a `signature` is present, no plaintext reasoning. This is an API-level choice and not something cc-logger can work around.
+
+Insertion is idempotent (`ON CONFLICT (message_id, block_index) DO NOTHING`), so repeated reads of the same transcript are safe.
+
 ## Hook payload notes
 
 Every hook event includes a common envelope:

docs/SCHEMA.md

@@ -53,6 +53,24 @@ cc-logger writes to 4 tables. The full DDL is in [`migrations/001_initial_schema
 | `started_at` | TIMESTAMPTZ | When PreToolUse fired. |
 | `received_at` | TIMESTAMPTZ | When the worker actually processed the event. Difference shows async queue lag. |
 
+## `messages` — assistant text blocks (Claude's narration)
+
+Populated by reading the Claude Code transcript JSONL at `Stop` / `SubagentStop` / `SessionEnd`. Only `text` blocks are captured; Claude's `thinking` blocks are encrypted in the transcript (signature only, no plaintext) and we can't extract them.
+
+| column | type | notes |
+|---|---|---|
+| `message_id` | TEXT | Anthropic message UUID from the transcript. |
+| `block_index` | INTEGER | Position of the text block within the message's `content` array. PK is composite (`message_id`, `block_index`). |
+| `session_id` | TEXT FK | References `sessions(session_id)`. |
+| `invocation_id` | TEXT FK | The agent that produced the message (root or sub-agent). |
+| `role` | TEXT | `assistant` (we only capture assistant text). |
+| `block_type` | TEXT | `text` (we only capture text blocks). |
+| `text` | TEXT | The text Claude said, after redaction. |
+| `position` | INTEGER | Line number in the source JSONL — gives a stable in-transcript ordering. |
+| `created_at` | TIMESTAMPTZ | When the row was inserted. |
+
+Indexed on `(session_id, position)` and `(invocation_id)`.
+
 ## `artifacts` — overflow for any field >50KB
 
 | column | type | notes |

examples/settings-hooks.json

@@ -71,6 +71,13 @@
         ]
       }
     ],
+    "Stop": [
+      {
+        "hooks": [
+          { "type": "http", "url": "http://127.0.0.1:8787/hook", "timeout": 5 }
+        ]
+      }
+    ],
     "SessionEnd": [
       {
         "hooks": [

migrations/003_messages.py

@@ -0,0 +1,82 @@
+#!/usr/bin/env python3
+"""Migration 003: Assistant message capture.
+
+Adds a `messages` table that stores Claude's text/narration blocks
+extracted from the Claude Code transcript JSONL files. This is the
+"what Claude was thinking out loud" layer — the decisions and framing
+between tool calls.
+
+Note: Claude's extended `thinking` blocks are encrypted in the transcript
+(only a `signature` is present, no plaintext). We capture only `text`
+blocks. See docs/PRIVACY.md.
+
+Usage:
+    python3 migrations/003_messages.py            # dry-run
+    python3 migrations/003_messages.py --apply
+"""
+import argparse
+import os
+import sys
+from pathlib import Path
+
+from dotenv import load_dotenv
+import psycopg
+
+load_dotenv(Path(__file__).resolve().parent.parent / ".env")
+
+
+DDL_STATEMENTS = [
+    """
+    CREATE TABLE IF NOT EXISTS messages (
+        message_id      TEXT NOT NULL,
+        block_index     INTEGER NOT NULL,
+        session_id      TEXT NOT NULL REFERENCES sessions(session_id) ON DELETE CASCADE,
+        invocation_id   TEXT REFERENCES agent_invocations(invocation_id) ON DELETE SET NULL,
+        role            TEXT NOT NULL DEFAULT 'assistant',
+        block_type      TEXT NOT NULL DEFAULT 'text',
+        text            TEXT NOT NULL,
+        position        INTEGER NOT NULL,
+        created_at      TIMESTAMPTZ NOT NULL DEFAULT now(),
+        PRIMARY KEY (message_id, block_index)
+    )
+    """,
+    "CREATE INDEX IF NOT EXISTS idx_messages_session_position ON messages (session_id, position)",
+    "CREATE INDEX IF NOT EXISTS idx_messages_invocation ON messages (invocation_id)",
+    """
+    CREATE OR REPLACE VIEW vw_session_messages AS
+    SELECT m.session_id, m.invocation_id, m.position, m.text, m.created_at
+    FROM messages m
+    ORDER BY m.session_id, m.position
+    """,
+]
+
+
+def get_dsn() -> str:
+    dsn = os.getenv("DATABASE_URL") or os.getenv("NEON_CC_LOGGER_URL")
+    if not dsn:
+        sys.exit("DATABASE_URL not set in environment (.env)")
+    return dsn
+
+
+def main() -> None:
+    ap = argparse.ArgumentParser(description=__doc__)
+    ap.add_argument("--apply", action="store_true")
+    args = ap.parse_args()
+    dsn = get_dsn()
+
+    if not args.apply:
+        print("DRY RUN — would execute the following statements:\n")
+        for stmt in DDL_STATEMENTS:
+            print(stmt.strip())
+            print("---")
+        print(f"\n{len(DDL_STATEMENTS)} DDL statements. Re-run with --apply.")
+        return
+
+    with psycopg.connect(dsn, autocommit=True) as conn, conn.cursor() as cur:
+        for stmt in DDL_STATEMENTS:
+            cur.execute(stmt)
+    print(f"Applied {len(DDL_STATEMENTS)} DDL statements.")
+
+
+if __name__ == "__main__":
+    main()

scripts/install-hooks.py

@@ -30,6 +30,7 @@
     "UserPromptSubmit",
     "SubagentStart",
     "SubagentStop",
+    "Stop",
     "SessionEnd",
 )
 TOOL_MATCHERS = ("Agent|Bash|Edit|Write|WebFetch|WebSearch", "mcp__.*")

src/cc_logger/handlers.py

@@ -12,7 +12,7 @@
 from psycopg import AsyncConnection
 from psycopg.types.json import Json
 
-from . import models
+from . import models, transcripts
 from .artifacts import truncate
 from .filters import should_capture
 from .linking import resolve_parent
@@ -247,6 +247,17 @@ async def handle_subagent_stop(conn: AsyncConnection, ev: models.SubagentStop) -
             """,
             (ev.last_message, _now(), ev.agent_id),
         )
+    # Ingest the sub-agent's transcript so its text blocks land in `messages`.
+    await transcripts.ingest(conn, ev.session_id, ev.transcript_path, ev.agent_id)
+
+
+async def handle_stop(conn: AsyncConnection, ev: models.Stop) -> None:
+    """Root agent finished a turn. Ingest the transcript incrementally so
+    Claude's narration / decisions land in `messages` near-realtime.
+    """
+    # Root invocation gets the message rows; root_id is synthesized from session_id.
+    root_id = f"root::{ev.session_id}"
+    await transcripts.ingest(conn, ev.session_id, ev.transcript_path, root_id)
 
 
 async def handle_session_end(conn: AsyncConnection, ev: models.SessionEnd) -> None:
@@ -278,6 +289,9 @@ async def handle_session_end(conn: AsyncConnection, ev: models.SessionEnd) -> No
             """,
             (_now(), ev.session_id),
         )
+    # Final reconciliation pass on the transcript: catch anything Stop missed.
+    root_id = f"root::{ev.session_id}"
+    await transcripts.ingest(conn, ev.session_id, ev.transcript_path, root_id)
 
 
 HANDLERS = {
@@ -288,6 +302,7 @@ async def handle_session_end(conn: AsyncConnection, ev: models.SessionEnd) -> No
     "PostToolUseFailure": handle_post_tool_use_failure,
     "SubagentStart": handle_subagent_start,
     "SubagentStop": handle_subagent_stop,
+    "Stop": handle_stop,
     "SessionEnd": handle_session_end,
 }
 

src/cc_logger/inspect.py

@@ -101,6 +101,22 @@ async def render(session_id: str, file=None) -> None:
             tc_cols = [c.name for c in cur.description]
             tool_calls = [dict(zip(tc_cols, r)) for r in await cur.fetchall()]
 
+            # Pull assistant text blocks (may be empty if Stop hook hasn't run yet)
+            try:
+                await cur.execute(
+                    """
+                    SELECT invocation_id, text, position, created_at
+                    FROM messages WHERE session_id = %s
+                    ORDER BY position
+                    """,
+                    (session_id,),
+                )
+                msg_cols = [c.name for c in cur.description]
+                messages = [dict(zip(msg_cols, r)) for r in await cur.fetchall()]
+            except Exception:
+                # `messages` table may not exist on older installs
+                messages = []
+
     # Header
     print("", file=out)
     print(f"SESSION {session['session_id']}", file=out)
@@ -125,9 +141,29 @@ async def render(session_id: str, file=None) -> None:
     children: dict[str | None, list[dict]] = {}
     for i in invocations:
         children.setdefault(i["parent_invocation_id"], []).append(i)
-    calls_by_inv: dict[str | None, list[dict]] = {}
+
+    # Interleave tool_calls and messages per invocation by timestamp.
+    # tool_calls have started_at; messages have created_at + position (line
+    # number in transcript). We use created_at as the sort key for messages.
+    timeline_by_inv: dict[str | None, list[tuple]] = {}
     for tc in tool_calls:
-        calls_by_inv.setdefault(tc["invocation_id"], []).append(tc)
+        timeline_by_inv.setdefault(tc["invocation_id"], []).append(
+            (tc["started_at"], "tool", tc)
+        )
+    for m in messages:
+        timeline_by_inv.setdefault(m["invocation_id"], []).append(
+            (m["created_at"], "msg", m)
+        )
+    for inv_id in timeline_by_inv:
+        timeline_by_inv[inv_id].sort(key=lambda x: x[0])
+
+    def _fmt_msg(msg: dict, indent: str) -> str:
+        text = msg["text"].strip()
+        # Wrap long lines onto continuation lines for readability
+        first = text[:120].replace("\n", " ")
+        if len(text) > 120:
+            first += "..."
+        return f'{indent}  · {first!r}'
 
     def render_inv(inv: dict, depth: int) -> None:
         indent = "  " * depth
@@ -136,8 +172,11 @@ def render_inv(inv: dict, depth: int) -> None:
             head += f" {inv['agent_id'][:18]}"
         head += f"  {_STATUS_GLYPH.get(inv['status'], '?')}]"
         print(head, file=out)
-        for tc in calls_by_inv.get(inv["invocation_id"], []):
-            print(f"{indent}  {_fmt_tool_summary(tc)}", file=out)
+        for _ts, kind, item in timeline_by_inv.get(inv["invocation_id"], []):
+            if kind == "tool":
+                print(f"{indent}  {_fmt_tool_summary(item)}", file=out)
+            else:
+                print(_fmt_msg(item, indent), file=out)
         for child in children.get(inv["invocation_id"], []):
             render_inv(child, depth + 1)
         if inv.get("last_message"):
@@ -151,9 +190,10 @@ def render_inv(inv: dict, depth: int) -> None:
     n_fail = sum(1 for t in tool_calls if t["status"] == "failure")
     n_pend = sum(1 for t in tool_calls if t["status"] == "pending")
     n_subs = sum(1 for i in invocations if i["parent_invocation_id"])
+    n_msgs = len(messages)
     print("", file=out)
     print(f"  {len(invocations)} invocations ({n_subs} sub-agents), {n_tools} tool calls "
-          f"({n_fail} failed, {n_pend} pending)", file=out)
+          f"({n_fail} failed, {n_pend} pending), {n_msgs} text blocks", file=out)
 
 
 def run(session_id: str) -> None:

src/cc_logger/models.py

@@ -77,6 +77,13 @@ class SubagentStop(HookEnvelope):
     last_message: str | None = None
 
 
+class Stop(HookEnvelope):
+    """Fires when the root agent finishes a turn. We use it to incrementally
+    ingest assistant text blocks from the transcript file."""
+    hook_event_name: Literal["Stop"]
+    stop_hook_active: bool | None = None
+
+
 class SessionEnd(HookEnvelope):
     hook_event_name: Literal["SessionEnd"]
     reason: str | None = None
@@ -91,6 +98,7 @@ class SessionEnd(HookEnvelope):
     "PostToolUseFailure": PostToolUseFailure,
     "SubagentStart": SubagentStart,
     "SubagentStop": SubagentStop,
+    "Stop": Stop,
     "SessionEnd": SessionEnd,
 }