Round 5 · Grassmannian attention — identifying a limitation
Grassmannian attention was faster on hierarchical retrieval. Does it also help on the 2-hop recall task — the task where standard attention performs poorly (19.5% after 10K steps)?
A good GPS system is great at finding the nearest gas station (1-hop: query → nearest match). A relay race requires multiple GPS handoffs — find the destination of the first step, then use that as the start of the second search. The question is whether Grassmannian's richer similarity computation helps with this multi-step coordination problem.
Flat 2-hop associative recall: given key A → find value B; use B as next query → find value C. Compared Grassmannian k=2 against standard dot-product attention over 10,000 steps.
| Architecture | Accuracy at 10K steps |
|---|---|
| Standard attention | 20.1% |
| Grassmannian (k=2) | 17.9% |
Grassmannian is marginally worse on 2-hop recall.
This is an important negative result that characterises Grassmannian's limitation. The subspace overlap similarity computes ||Q^T K||_F^2, which is always non-negative (a squared Frobenius norm is ≥ 0). This means Grassmannian attention logits can never go sharply negative — they can favour one key, but they can't strongly suppress another.
For 2-hop chaining, sharp suppression matters. To successfully chain A → B → C, the model must at the second step attend strongly to the "B" token and actively suppress all other tokens in context. A non-negative logit makes this harder: background tokens get too much residual attention weight, contaminating the second-hop retrieval.
This identifies a clean architectural tradeoff:
- Grassmannian is better for selection tasks — finding which item in a set is most similar to a query
- Standard attention is better for chaining tasks — following exact key-value chains across layers, requiring sharp suppression of non-matching items
The fix — subtracting a per-query baseline from the Grassmannian logit to partially restore signed discriminability — was not tested but remains an open direction.
- Would a "centered" Grassmannian logit (logit − mean_logit_per_query) restore 2-hop performance while preserving the selection advantage?
- Is there a unified primitive that handles both regimes? Or is the right approach mixed-head architectures with different primitives per head?