Skip to content

Fix Falcon ALiBi bias when using the KV cache - #2977

Open
GargiGupta-io wants to merge 1 commit into
keras-team:masterfrom
GargiGupta-io:fix-falcon-alibi-kv-cache
Open

Fix Falcon ALiBi bias when using the KV cache#2977
GargiGupta-io wants to merge 1 commit into
keras-team:masterfrom
GargiGupta-io:fix-falcon-alibi-kv-cache

Conversation

@GargiGupta-io

@GargiGupta-io GargiGupta-io commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Description of the change

_build_alibi_tensor builds the bias from the query window, so with a cache it returns (batch, heads, 1, q_len) against attention scores of (batch, heads, q_len, kv_len):

  • Multi-token steps don't broadcast → InvalidArgumentError. This is the reported bug.
  • Single-token steps do broadcast, to (batch, heads, 1, 1), and evaluate to all zeros since cumsum(ones(1)) - 1 == 0. generate_step decodes one token at a time, so FalconCausalLM.generate() has been running with ALiBi disabled at every decode step. Not mentioned in the issue, and the worse of the two.

Building the bias over the cache length fixes both. Unused cache slots are already masked out by the causal mask, which _compute_attention_mask already accounts for.

Verified in the Colab: cached decoding matches a full forward pass (max abs diff 2.98e-07), and the no-cache path is bit-identical before and after, with and without a padding mask. keras_hub/src/models/falcon/ passes (18 passed, 11 skipped), and reverting the fix while keeping the new test makes it fail.

Reference

Fixes #2861. Same fix as #2862 by @pctablet505, which was closed unmerged by its author.

Colab Notebook

https://colab.research.google.com/drive/1Hg1h55GT7wfn0lvz-R92Y5brx5I6jkJz

Checklist

  • I have added all the necessary unit tests for my change.
  • I have verified that my change does not break existing code and works with all backends (TensorFlow, JAX, and PyTorch).
  • My PR is based on the latest changes of the main branch (if unsure, rebase the code).
  • I have followed the Keras Hub Model contribution guidelines in making these changes.
  • I have followed the Keras Hub API design guidelines in making these changes.
  • I have signed the Contributor License Agreement.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request fixes an issue in the Falcon transformer decoder where the ALiBi bias was incorrectly built from the query window length instead of the attention cache length during cached decoding. This caused broadcasting failures and disabled ALiBi during single-token decoding. The fix ensures the mask spans the entire cache length when attention_cache is present. Additionally, a unit test test_cached_decoding_matches_full_forward has been added to FalconCausalLMTest to verify that cached decoding matches a full forward pass. No review comments were provided, so there is no feedback to address.

The ALiBi bias was built from the query window instead of the cache, so
it was shorter than the attention scores whenever a cache was in use:

- With more than one token per step it fails to broadcast, raising an
  error.
- With a single token it broadcasts silently and evaluates to all zeros,
  because `cumsum(ones(1)) - 1 == 0`. Since `generate()` decodes one
  token at a time, `FalconCausalLM.generate()` has been running with
  ALiBi effectively disabled at every decode step.

Build the bias over the cache length when a cache is present. Unused
cache positions are already masked out by the causal attention mask, so
an all-ones mask yields the correct absolute positions.

Verified: cached decoding now matches a full forward pass (max abs diff
2.98e-07), and the no-cache path is bit-identical before and after, both
with and without a padding mask.
@GargiGupta-io
GargiGupta-io force-pushed the fix-falcon-alibi-kv-cache branch from 0aa321a to 5b26201 Compare August 23, 2026 09:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Falcon: call_with_cache fails when KV-cache length exceeds query length (ALiBi built over query length)

1 participant