[Backport 3.7] [Sparse ANN] Fold sparse vector tokens into the signed-short range - #1929
Closed
chishui wants to merge 1 commit into
Closed
[Backport 3.7] [Sparse ANN] Fold sparse vector tokens into the signed-short range#1929chishui wants to merge 1 commit into
chishui wants to merge 1 commit into
Conversation
…pensearch-project#1926) * [Sparse ANN] Fold sparse vector tokens into the signed-short range Token IDs are folded into a fixed range via `token % MODULUS_FOR_SHORT` and stored in a signed short[] to keep the memory footprint low. The modulus was 65536, so a token folding into [32768, 65535] was cast to a negative short and, when read back, sign-extended to a negative int: - getToken()/iterator() returned a negative token (e.g. 40000 -> -25536) - toDenseVector() threw NegativeArraySizeException on `maxToken + 1` - dotProduct() mis-scored via a negative index / bad bounds check Those values never produced a usable index, so the effective working range was already [0, 32767]. Set MODULUS_FOR_SHORT to 32768 so every folded token is <= Short.MAX_VALUE and round-trips without sign extension. Because 32768 divides 65536, every token that previously worked folds to the same value, so existing indices are unaffected. Adds unit tests covering the constructor/iterator round-trip, toDenseVector(), and dotProduct() for a token folding to Short.MAX_VALUE. Signed-off-by: Liyun Xiu <xiliyun@amazon.com> * update changelog Signed-off-by: Liyun Xiu <xiliyun@amazon.com> --------- Signed-off-by: Liyun Xiu <xiliyun@amazon.com> (cherry picked from commit 936d3c4) Signed-off-by: Liyun Xiu <xiliyun@amazon.com>
chishui
requested review from
VijayanB,
bzhangam,
heemin32,
jmazanec15,
junqiu-lei,
martin-gaievski,
minalsha,
model-collapse,
naveentatikonda,
navneet1v,
owaiskazi19,
sean-zheng-amazon,
vamshin,
vibrantvarun,
yuye-aws,
zane-neo and
zhichao-aws
as code owners
August 4, 2026 08:40
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
Collaborator
|
We don't allow back porting to older version unless it is security related. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport 936d3c4 from #1926.
The automatic backport failed due to a
CHANGELOG.mdconflict only — main's Bug Fixes section carries an unrelated SemanticHighlighter entry (#1906) that isn't on 3.7. The Java changes cherry-picked cleanly; the changelog entry was re-added alongside 3.7's existing Bug Fixes entries.SparseVectorTestspasses locally on this branch.