Round 2 · Compression attempts
Given that attention patterns are spectrally concentrated, can you actually compress them — keep only the low-frequency FFT coefficients, discard the rest — without the model falling apart? This is the direct practical test of the compression idea.
A JPEG image compressed at 90% quality looks almost identical to the original. Compressed at 10% quality it becomes blocky and blurred but is still recognisable. The question here is whether attention patterns behave like images — with a graceful degradation curve where you can trade quality for compression — or like encryption keys, where removing even a tiny fraction destroys everything.
KV-cache compression is a real problem in deployed LLMs. At long context lengths, storing all the key-value pairs from every attention head consumes enormous memory. If you could keep only 10% of the FFT coefficients per attention matrix, you'd get a 10× reduction in KV-cache memory with acceptable quality loss. This experiment tests whether that tradeoff exists.
We passed GPT-2 through its normal forward pass, but at each layer intercepted the attention score matrices, applied FFT compression at varying retention ratios (keeping 50%, 20%, 10%, 5%, 2%, 1% of coefficients), and measured the resulting perplexity on a held-out text corpus. Baseline (uncompressed): perplexity ~48.
| Retention ratio | Resulting perplexity |
|---|---|
| 100% (baseline) | 48 |
| 50% | ~850 |
| 20% | ~900 |
| 10% | ~920 |
| 5% | ~920 |
Every compression ratio tested produced catastrophic perplexity collapse. There is no "graceful degradation" curve — the model breaks immediately, and the damage does not grow with further compression. The failure happens at 50% and doesn't get meaningfully worse at 1%.
The spectral structure of attention patterns is an illusion of exploitability. Yes, 95% of the energy lives in 5% of the coefficients — but the remaining 5% of energy, spread across the discarded high-frequency components, is not noise. It carries the token-specific routing decisions: which exact token this position should attend to, among all the tokens in context. That information is high-frequency (it changes sharply from token to token) and cannot be discarded.
Think of it this way: the low-frequency envelope of an attention pattern encodes "this head generally attends to recent tokens." The high-frequency detail encodes "at this specific position, it should attend to token 7 rather than token 6." The second piece of information is what makes the model work. Discarding it for compression breaks the model immediately, regardless of how little spectral energy it carries.
The compression hypothesis is falsified. Attention patterns cannot be compressed in the frequency domain without destroying the model.
- Could a trained compression scheme — where the model learns to reconstruct high-frequency attention from compressed low-frequency information — succeed where post-hoc FFT truncation fails?
- What is the minimum information needed per attention head per token to preserve model quality?