Round 7 · FNO's actual domain
After finding that FNO fails on language-style tasks (hierarchical retrieval, 2-hop recall through attention), we wanted to characterise its positive use case. FNO was designed for operator learning — learning a function that maps one function to another, evaluated on a fixed spatial grid. The canonical example: given the initial temperature distribution along a rod, predict its distribution after some time (the 1D heat equation). Does FNO actually win here?
A violin bow is useless for hammering nails — that doesn't make it a bad tool. It's the right tool for making music. These FNO experiments are about finding what the bow is actually good at, after establishing that it's a bad hammer.
Three models — FNO, Transformer, and FNet (fixed FFT, no learnable weights) — were trained on the 1D heat equation: given u(x, t=0), predict u(x, t=1). 1,000 training samples, 200 test samples. Performance measured as relative L2 error (lower is better; 0 = perfect, 1 = as bad as predicting zero everywhere).
| Model | Relative L2 error |
|---|---|
| FNO | 0.0078 |
| Transformer | 0.0108 |
| FNet (fixed FFT) | 0.674 |
FNO wins clearly. The Transformer is competitive but ~40% higher error. FNet fails completely — showing that the advantage specifically comes from learnable frequency weights, not from FFT processing alone.
FNO's inductive bias — learnable complex weights applied to the Fourier modes of the input — is exactly right for this task. The heat equation applies a fixed spatial operator repeatedly. FNO directly parametrises this operation in frequency space: heat diffusion suppresses high-frequency components faster than low-frequency ones, which is exactly what learnable frequency-domain weights capture.
The FNet result is a key control: processing with a fixed FFT (no learned frequency weights) doesn't help at all. The Fourier basis is necessary but not sufficient — you need to learn the right frequency weights for the specific operator. FNO has architecture that makes this easy; standard transformers have to discover it more indirectly.
Above ~500K parameters, FNO is also more parameter-efficient than Transformers on this class of task (see exp-fno-modes). It does more with fewer parameters when the problem genuinely has frequency-domain structure.
The summary: FNO is a precision instrument for physics simulation and operator learning on fixed spatial grids. It is parameter-efficient, accurate, and theoretically well-motivated for PDEs. It has no business in a language model.
- Does FNO's advantage scale to 2D/3D PDEs and more complex operators?
- Could a hybrid that uses FNO layers for positional/structural processing and attention for content routing outperform either alone?