Fix Falcon ALiBi bias when using the KV cache - #2977
Open
GargiGupta-io wants to merge 1 commit into
Open
Conversation
Contributor
There was a problem hiding this comment.
Code Review
This pull request fixes an issue in the Falcon transformer decoder where the ALiBi bias was incorrectly built from the query window length instead of the attention cache length during cached decoding. This caused broadcasting failures and disabled ALiBi during single-token decoding. The fix ensures the mask spans the entire cache length when attention_cache is present. Additionally, a unit test test_cached_decoding_matches_full_forward has been added to FalconCausalLMTest to verify that cached decoding matches a full forward pass. No review comments were provided, so there is no feedback to address.
The ALiBi bias was built from the query window instead of the cache, so it was shorter than the attention scores whenever a cache was in use: - With more than one token per step it fails to broadcast, raising an error. - With a single token it broadcasts silently and evaluates to all zeros, because `cumsum(ones(1)) - 1 == 0`. Since `generate()` decodes one token at a time, `FalconCausalLM.generate()` has been running with ALiBi effectively disabled at every decode step. Build the bias over the cache length when a cache is present. Unused cache positions are already masked out by the causal attention mask, so an all-ones mask yields the correct absolute positions. Verified: cached decoding now matches a full forward pass (max abs diff 2.98e-07), and the no-cache path is bit-identical before and after, both with and without a padding mask.
GargiGupta-io
force-pushed
the
fix-falcon-alibi-kv-cache
branch
from
August 23, 2026 09:05
0aa321a to
5b26201
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of the change
_build_alibi_tensorbuilds the bias from the query window, so with a cache it returns(batch, heads, 1, q_len)against attention scores of(batch, heads, q_len, kv_len):InvalidArgumentError. This is the reported bug.(batch, heads, 1, 1), and evaluate to all zeros sincecumsum(ones(1)) - 1 == 0.generate_stepdecodes one token at a time, soFalconCausalLM.generate()has been running with ALiBi disabled at every decode step. Not mentioned in the issue, and the worse of the two.Building the bias over the cache length fixes both. Unused cache slots are already masked out by the causal mask, which
_compute_attention_maskalready accounts for.Verified in the Colab: cached decoding matches a full forward pass (max abs diff 2.98e-07), and the no-cache path is bit-identical before and after, with and without a padding mask.
keras_hub/src/models/falcon/passes (18 passed, 11 skipped), and reverting the fix while keeping the new test makes it fail.Reference
Fixes #2861. Same fix as #2862 by @pctablet505, which was closed unmerged by its author.
Colab Notebook
https://colab.research.google.com/drive/1Hg1h55GT7wfn0lvz-R92Y5brx5I6jkJz
Checklist