Round 1 · Spectral diagnostics
This was the founding question of the whole research programme: can GPT-2's weight matrices be compressed by exploiting spectral structure? The intuition came from an analogy with radio signals — if weights encode information at specific "frequencies," keeping only those frequencies might preserve most of the model's capability.
The specific test: do weight matrices have a low-rank structure in either the FFT basis (frequency domain) or the SVD basis?
Imagine a painting stored as a JPEG. JPEGs work by throwing away high-frequency detail (sharp edges, fine texture) that the human eye barely notices, keeping only the low-frequency structure (broad shapes, colours). This lossy compression works because natural images are low-frequency — most of the information lives in a small fraction of the frequency coefficients.
If GPT-2 weight matrices were like natural images, you could do the same thing: keep the low-frequency components, discard the rest, and recover most of the model's capability. This experiment tests whether that premise is true.
If weight matrices had low-rank spectral structure, compression would be straightforward and lossless. LoRA (Low-Rank Adaptation) already exploits the fact that fine-tuning deltas are low-rank. The question here is whether the pretrained weights themselves have this property.
We computed the FFT of every weight matrix in GPT-2 (attention and MLP weights across all 12 layers) and measured how many coefficients are needed to capture 80% of the spectral energy. We did the same with SVD, measuring the rank needed to capture 80% of the matrix's variance.
Baseline: for a truly random matrix, you'd expect ~50% of coefficients to be needed in either basis.
| Basis | Fraction of coefficients for 80% energy |
|---|---|
| FFT | ~44% |
| SVD (rank_80) | ~42% |
| Random matrix (expected) | ~50% |
Both bases give nearly identical results: ~44% of coefficients are needed for 80% of energy. This is indistinguishable from a random matrix. The two independent bases agree, ruling out basis-specific artefacts.
GPT-2's weight matrices are spectrally flat. There is no low-frequency concentration, no natural compression structure. The radio analogy does not hold for static weights.
This directly falsifies the starting hypothesis. Unlike fine-tuning deltas (which LoRA correctly identifies as low-rank), the pretrained weights are distributed across their full spectral range. Whatever structure GPT-2 has learned is encoded holographically — no small subset of frequencies captures it.
The exception (discovered in exp-attn-spectrum): dynamic attention patterns are structured, but for causal-mask geometric reasons rather than learned semantic ones.
- Weight compression via FFT truncation: fails immediately (perplexity collapses)
- The "radio signal" model of how LLMs store knowledge
- Any compression scheme that assumes low-frequency concentration in weight space
- LoRA exploits that fine-tuning deltas are low-rank. Could intermediate representations (residual stream changes between layers) be low-rank even if the weights themselves aren't?