Skip to content

SKILL.md

Latest commit

 

History

History
89 lines (74 loc) · 4.43 KB

SKILL.md

File metadata and controls

89 lines (74 loc) · 4.43 KB
Raw
name interp
description Use this skill whenever the user asks WHY a language model produced a particular output, wants to find which attention heads / MLPs / features are responsible for a behavior, or asks to run mechanistic-interpretability analyses (logit lens, direct logit attribution, activation patching, induction heads, SAE features, activation steering) on a transformer. Triggers include "why did the model predict X", "which heads cause Y", "what circuit does this", "interpret this feature", "is this an induction head", "steer the model toward".

interp — mechanistic interpretability for coding agents

You can answer why a transformer made a prediction, with real mech-interp techniques, and back every claim with a number you computed. Do not speculate about model internals when you can measure them with this skill.

Setup (once)

All paths are relative to this skill's directory — the directory containing this SKILL.md (e.g. .claude/skills/interp/, or the repo root in a standalone clone). Call it $SKILL below.

$SKILL/run.sh setup    # creates $SKILL/.venv, installs deps (CPU-friendly)

run.sh cd's to its own directory, so it works from any CWD. Run every command through it: $SKILL/run.sh <subcommand> ... (equivalently $SKILL/.venv/bin/python -m interp <subcommand> ...). Add --json to any command to get a structured result you can parse; omit it for a human summary. Default model is gpt2; pass --model <name> for any TransformerLens-supported model. Device defaults to CPU (slow but trustworthy); MPS is opt-in via --device mps — TransformerLens warns it may be silently incorrect.

Choosing the right tool

The user wants to know… Command
How the prediction forms across layers logit-lens PROMPT --target " TOK"
Which heads/MLPs write the answer (correlational) dla PROMPT " ANSWER" --baseline " OTHER"
What the model causally depends on patch CLEAN CORRUPT " ANSWER" " OTHER"
Which heads are causally necessary patch-heads CLEAN CORRUPT " ANSWER" " OTHER"
Whether a head is an induction head induction
Where one head attends attention PROMPT LAYER HEAD
Which interpretable SAE features fire on text sae "TEXT"
Whether a direction changes behavior steer PROMPT --pos ... --neg ...
Everything at once → a written report explain PROMPT " ANSWER" --baseline " OTHER" --corrupt CORRUPT

How to answer a "why did it predict X" question

  1. Run explain — it writes out/explain.md + figures under $SKILL/out/ (gitignored; override with --outdir) and returns structured findings (logit lens, DLA, patching, SAE features).
  2. Read the findings and write the prose yourself. The tool gives you measurements; you turn them into an explanation. Cite specific components, e.g. "head L9H9 contributes +1.66 to the Mary−John logit difference".
  3. For a causal claim, you MUST have run patching — DLA alone is correlational. Construct a minimal pair: the corrupt prompt differs from the clean one only in the token(s) that flip the answer, and must tokenize to equal length.
  4. Show the figure paths and the report path to the user.

Key conventions (read before first use)

  • Tokens carry leading spaces. GPT-2 tokenizes " Mary" (with space), not "Mary". Always pass answer/baseline/target tokens with the leading space.
  • Minimal pairs must be equal length or patch raises a clear error.
  • Recovery score from patching: 1.0 = this activation fully restores clean behavior, 0.0 = no causal effect.
  • DLA is correlational, patching is causal. Never state a causal conclusion from DLA alone.
  • Honesty: a single-vector steer is coarse and often degrades fluency — report when output becomes repetitive rather than claiming a clean success.

Worked example (IOI)

$SKILL/run.sh explain \
  "When John and Mary went to the store, John gave a drink to" " Mary" \
  --baseline " John" \
  --corrupt "When John and Mary went to the store, Mary gave a drink to"

Expected: logit lens shows " Mary" becoming the top guess at layer 10 (final P ≈ 0.48); DLA surfaces the name-mover heads L9H9/L9H6/L10H0; head patching surfaces S-inhibition heads L8H6/L8H10 and negative name mover L10H7 as the causal carriers. (These match Wang et al., 2022.) Note explain runs ~330 forward passes for the patching scans — expect minutes on CPU, not seconds.