[Fix] Gate decode shortcuts and validate dims across layers - #1203
Open
zhiyuan1i wants to merge 5 commits into
Open
[Fix] Gate decode shortcuts and validate dims across layers#1203zhiyuan1i wants to merge 5 commits into
zhiyuan1i wants to merge 5 commits into
Conversation
Shape-equality decode shortcuts (B*T==N, T==1) misfire on packed varlen batches that contain zero-length or multi-token sequences: short_conv's step() ignores cu_seqlens and misaligns tokens to cache rows, rwkv7's decode branch reads/writes out of bounds, and GSA's inference shortcut treats a packed batch as dense. Gate the shortcuts on every sequence having exactly one token (dense T==1 unchanged).
int(hidden_size * expand) truncates floating-point representations of integers (50 * 0.58 -> 28 instead of 29) and silently uses non-integer dims. Round first and reject true non-integers with math.isclose, and assert divisibility for head dims, matching the convention already used in abc/raven/kda layers. Also fixes raven's false reject of valid expand values (it checked isclose after int-truncation).
A non-divisible n_groups previously failed late with a cryptic reshape error in the decode path.
The GLA recurrent state is [B, H, head_f_dim, head_i_dim], i.e. input_dim * expand_ratio per sequence.
Both max computations (with their device syncs) only feed rotary, so skip them entirely in the default use_rope=False path.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Batch of small, independent safety fixes found by auditing the codebase for the same bug classes as recent community fix PRs:
short_conv.py,ops/rwkv7/fused_recurrent.py,ops/gsa/fused_recurrent.py): shape-equality decode checks (B*T==N,T==1) are wrong when a packed batch has zero-length or multi-token sequences — short_conv misaligns tokens to cache rows, rwkv7 reads/writes out of bounds, GSA treats a packed batch as dense. Gate shortcuts on every sequence having exactly one token.int(hidden_size * expand)truncates fp-representable integers (50*0.58→28) and silently uses non-integer dims. Round first and reject true non-integers withmath.isclose, and assert divisibility for head dims — the convention already used in abc/raven/kda layers. Also fixes raven's false reject of valid expand values (isclose was checked after int-truncation).num_heads % n_groups: fail fast instead of a cryptic late reshape error in decode.state_size: implement asinput_dim * expand_ratio(GLA recurrent state shape).Test plan
tests/layers/test_layer_dim_validation.py(17 constructor-validation cases, CPU),tests/modules/test_conv.py::test_conv_varlen_decode_detection_with_zero_len_seqtests/layers/test_layer_dim_validation.py17/17 pass locally; ruff and compileall cleanBenchmark / NCU (kernel changes only)
Breaking changes
Checklist