Round 4 · p-adic geometry
Even if Euclidean space can represent tree distances faithfully, perhaps attention computed using p-adic distances — which directly encodes tree structure — would learn faster or more reliably than standard cosine attention. We tested this with an "oracle" p-adic attention: an attention mechanism given direct access to the ground-truth tree distances.
Suppose you're teaching someone to navigate a city. You could let them learn the layout through experience (standard attention), or you could give them a map that directly encodes every distance (oracle p-adic). Does having the map help? Or is the experience-based learner just as good?
At tree depth 4 (16 leaves, 64-dimensional embeddings — well within the d > N regime where standard attention succeeds), we compared three variants:
- Standard attention with random embedding initialisation
- Oracle p-adic attention (uses true ultrametric tree distances)
- Standard attention with hierarchical path-encoding initialisation
At depth 4 (16 leaves, d=64):
- No performance gap exists. Standard attention already achieves ~100% accuracy. Oracle p-adic has nothing to improve upon.
- Hierarchical path-encoding initialisation: 4× slower convergence than random init, despite having "correct" geometric structure built in.
At scales where standard attention works (d > N), providing explicit geometric knowledge about the tree structure provides no benefit — and forcing geometric priors on the initialisation actively hurts training.
Two conclusions. First, the geometry problem at depth 4 is already solved by standard attention. There is no performance ceiling to lift. Second, and more surprising: giving the model pre-computed geometric knowledge constrains the gradient-based optimiser's search in ways that make learning harder, not easier. The model finds a better solution by starting from a random configuration and discovering the embedding structure itself.
This is a general lesson that extends beyond this specific experiment: carefully designed geometric priors injected before training often slow things down, because gradient descent on a high-dimensional loss landscape can find solutions that violate the prior's constraints. Don't constrain the optimiser's starting point based on what the solution "should" look like.
The interesting depth is 6 — where standard attention fails and the question of what can fix it becomes urgent. See exp-depth-sweep.
- Does oracle p-adic help at depth 6 and above — the regime where standard attention breaks?
- Can p-adic-inspired training objectives (rather than initialisation) be used to guide learning without constraining it from the start?