Skip to content

ioi.md

Latest commit

 

History

History
90 lines (67 loc) · 3.53 KB

ioi.md

File metadata and controls

90 lines (67 loc) · 3.53 KB
Raw

Walkthrough: reproducing the IOI circuit

The Indirect Object Identification (IOI) task, from Wang et al., 2022, is the canonical worked example in mechanistic interpretability. GPT-2 small reliably completes

"When John and Mary went to the store, John gave a drink to ___"

with " Mary" — the indirect object, not the repeated subject " John". This example uses interp to reproduce the circuit, one technique at a time. All commands assume source .venv/bin/activate (or use ./run.sh <args>).

1. Confirm the behavior and watch it form — logit lens

interp logit-lens "When John and Mary went to the store, John gave a drink to" \
  --target " Mary"

The final prediction is " Mary" with P ≈ 0.48 (next candidate " them" at 0.21). Tracking its rank layer by layer shows it is undecidable early (rank ≈ 43,000 at the embeddings, still ≈ 265 after layer 8), collapses to rank 14 at layer 9, and becomes the top token at layer 10 — telling us the deciding computation happens in the late layers.

2. Which components write the answer — DLA

interp dla "When John and Mary went to the store, John gave a drink to" \
  " Mary" --baseline " John"

Attribution to logit( Mary) − logit( John) is dominated by L9H9 (+1.66), L9H6 (+1.32), L10H0 (+0.88) — the name-mover heads. Negative contributors (L10H7, L11H10) are the negative name movers. This matches Wang et al.

DLA is correlational: it shows what writes to the logit, not what the computation depends on. For that we need patching.

3. What is causally necessary — activation patching

Build a minimal pair: a corrupt prompt identical except the names are swapped, so the answer flips. It must tokenize to the same length.

interp patch \
  "When John and Mary went to the store, John gave a drink to" \
  "When John and Mary went to the store, Mary gave a drink to" \
  " Mary" " John"

clean logit-diff ≈ +3.2, corrupt ≈ −3.5. Patching the clean activation at the swapped subject-name token restores the clean behavior (recovery ≈ 1.0). Treat this as a sanity check, not a finding: the swapped name is the only place the prompts differ, so patching it back must restore the behavior — it validates the minimal pair and the patching machinery.

The genuinely causal, circuit-level result comes from patching per head:

interp patch-heads <clean> <corrupt> " Mary" " John"

This surfaces the S-inhibition heads L8H6 (+0.15) and L8H10 (+0.20), the negative name mover L10H7 (−0.29), and name mover L9H9 (+0.12) as the heads carrying the causal signal — the internal components Wang et al. 2022 identified, recovered here by intervention rather than correlation.

4. What concepts are present — SAE features

interp sae "When John and Mary went to the store, John gave a drink to"

Among the top features: "names of people" / "proper nouns" features peaking on John (Neuronpedia, GPT-2 small res-jb SAEs) — independent confirmation that the model represents the names as name-type entities.

5. One command for all of it

interp explain \
  "When John and Mary went to the store, John gave a drink to" " Mary" \
  --baseline " John" \
  --corrupt "When John and Mary went to the store, Mary gave a drink to"

writes out/explain.md with every result above plus figures in out/figures/ (gitignored; use --outdir to write elsewhere). A committed sample of this report lives at reports/explain.md.