Round 2 · Compression attempts
Round 1 found that attention patterns are spectrally concentrated (~5% of coefficients for 80% of energy). Before asking whether this is exploitable, we needed to know why it happens. Two candidates: (1) the causal mask forces a lower-triangular structure that is inherently low-frequency, or (2) the model has learned to attend in structured, low-bandwidth ways.
These two causes have very different implications. If it's just the mask geometry, the spectral structure contains no semantic information. If it's learned head behaviour, there might be something meaningful to exploit.
Imagine you're analysing the patterns on a chess board mid-game. The board has a fixed geometric constraint (alternating black and white squares) and learned strategic patterns (piece positions). Before drawing conclusions about chess strategy from the pattern you see, you need to separate the constraint effect from the strategic effect. This experiment is that separation.
We measured attention pattern compressibility in three conditions:
- Normal GPT-2 on real sentences — the baseline
- Shuffled tokens — real sentences with word order randomised, disrupting semantic structure while keeping the same causal mask
- Causal baseline — uniform attention over visible context (pure mask geometry, no learned content)
| Condition | Coefficients for 80% energy |
|---|---|
| Normal GPT-2 | 5.23% |
| Shuffled tokens | 5.12% |
| Causal uniform baseline | 9.25% |
The shuffled-token result matches normal GPT-2 almost exactly (5.12% vs 5.23%). The causal baseline is less compressible (9.25%), not more.
Learned head inductive biases, not the causal mask, drive attention's spectral concentration. The mask alone actually makes attention less compressible than the learned patterns. When you shuffle the tokens, breaking semantic content while keeping the same mask, the compressibility is unchanged — confirming the structure comes from what GPT-2 has learned, not from the mask's geometry.
What has it learned? GPT-2's attention heads are known to specialise: some attend to recent tokens (local heads), some attend to a fixed "sink" position, some follow diagonal patterns. These behavioural specialisations produce globally smooth, low-frequency patterns — hence the spectral concentration. The pattern is real, but it encodes routing style, not content routing.
This is a critical negative result for the compression direction: the spectral structure reveals head behaviour, not the semantic signal you'd want to preserve in a compressed KV-cache. Compressing it discards the token-specific routing decisions. (exp-attn-fft-perplexity confirms this directly.)
- Could you train GPT-2 to have more spectrally concentrated attention (even lower-frequency heads) without losing performance? This would be a route to genuinely compressible KV-caches.