[BUG] Sparse ANN throw NegativeArraySizeException for token index ranging from 32768-65535

[chishui]

What is the bug?

In Sparse ANN index which the doc count exceeds approximate_threshold, when token index ranges from 32768 to 65535, the sparse ann query will throw NegativeArraySizeException. As the logic hashes token index to short type but uses 65535 which generates negative short value. It should use 32768.

How can one reproduce the bug?

Create index:

{
  "settings": {
      "index": {
          "sparse": true,
          "number_of_shards": 1,
          "number_of_replicas": 0
      }
  },
  "mappings": {
    "properties": {
      "passage_text":{
          "type": "text"
      },
      "passage_embedding":{
          "type": "sparse_vector",
          "method": {
              "name": "seismic",
              "parameters": {
                "n_postings": 4,
                "cluster_ratio": 0.4,
                "summary_prune_ratio": 0.5,
                "approximate_threshold": 4
              }
          }
      }
    }
  }
}

bulk

{ "index": { "_index": "{{index}}"} }
{ "passage_text": "apple tree", "passage_embedding": {"40000": 1.5, "2000": 2.5},"sparse_embedding": {"40000": 1.5, "2000": 2.5}}
{ "index": { "_index": "{{index}}"} }
{ "passage_text": "banana tree", "passage_embedding": {"2000": 0.5, "3000": 1.3}, "sparse_embedding": {"2000": 0.5, "3000": 1.3}}
{ "index": { "_index": "{{index}}"} }
{ "passage_text": "banana tree", "passage_embedding": {"4000": 3.5, "5000": 1.6}, "sparse_embedding": {"4000": 3.5, "5000": 1.6}}
{ "index": { "_index": "{{index}}"} }
{ "passage_text": "duck", "passage_embedding": {"2000": 3.5, "6000": 1.6}, "sparse_embedding": {"2000": 3.5, "6000": 1.6}}

query:

{    
    "query": {
        "neural_sparse": {
            "passage_embedding": {
                "query_tokens": {
                    "65538": 2.0
                }
            }
        }
    },
    "size": 5
}

What is the expected behavior?

It should be able to query token index > 32768

What is your host/environment?

Operating system, version.

Do you have any screenshots?

If applicable, add screenshots to help explain your problem.

Do you have any additional context?

Add any other context about the problem.

[chishui]

Fixed by #1926