Gate hybrid collapse distinct-groups collection behind an opt-in index setting

Add index.neural_search.hybrid_collapse_distinct_groups_enabled
(dynamic, default false). The default keeps the existing behavior of
collecting the top-size documents per sub-query, preserving score
parity with the same hybrid query without collapse. When enabled,
HybridCollapsingTopGroupsCollector collects the top-size distinct
groups per sub-query instead, so the response contains size groups
whenever that many exist.

Resolves #1947

Signed-off-by: Yasutaka Hisano <yasutennis713@gmail.com>

Author: Ystk-hsn

CHANGELOG.md

@@ -11,7 +11,7 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 - Add `model_selection` (language_option/model_type) parameter to semantic field to resolve the model id from cluster settings ([#1918](https://github.com/opensearch-project/neural-search/issues/1918))
 
 ### Bug Fixes
-* [Hybrid Query] Fix collapse dropping valid groups when one group owns multiple top-scoring documents on a shard ([#1947](https://github.com/opensearch-project/neural-search/issues/1947))
+* [Hybrid Query] Add opt-in index setting `index.neural_search.hybrid_collapse_distinct_groups_enabled` to make collapse return top-`size` distinct groups instead of deduplicated top-`size` documents ([#1947](https://github.com/opensearch-project/neural-search/issues/1947))
 * [Hybrid Query] Fix NoSuchElementException in hybrid query with sort/search_after when a shard returns no results ([#1939](https://github.com/opensearch-project/neural-search/pull/1939))
 * [SemanticHighlighter] Fix SemanticHighlighterExtBuilder.toXContent ([#1906](https://github.com/opensearch-project/neural-search/issues/1906)) (query-insights [#651](https://github.com/opensearch-project/query-insights/issues/651))
 * [Sparse ANN] Fold sparse vector tokens into the signed-short range (modulus 32768) so folded tokens are never sign-extended to a negative value when stored in short[] ([#1926](https://github.com/opensearch-project/neural-search/pull/1926))

src/main/java/org/opensearch/neuralsearch/plugin/NeuralSearch.java

@@ -5,6 +5,7 @@
 package org.opensearch.neuralsearch.plugin;
 
 import static org.opensearch.neuralsearch.highlight.SemanticHighlightingConstants.HIGHLIGHTER_TYPE;
+import static org.opensearch.neuralsearch.settings.NeuralSearchSettings.HYBRID_COLLAPSE_DISTINCT_GROUPS_ENABLED;
 import static org.opensearch.neuralsearch.settings.NeuralSearchSettings.HYBRID_COLLAPSE_DOCS_PER_GROUP_PER_SUBQUERY;
 import static org.opensearch.neuralsearch.settings.NeuralSearchSettings.RERANKER_MAX_DOC_FIELDS;
 import static org.opensearch.neuralsearch.settings.NeuralSearchSettings.NEURAL_STATS_ENABLED;
@@ -376,6 +377,7 @@ public List<Setting<?>> getSettings() {
             NEURAL_STATS_ENABLED,
             SEMANTIC_INGEST_BATCH_SIZE,
             HYBRID_COLLAPSE_DOCS_PER_GROUP_PER_SUBQUERY,
+            HYBRID_COLLAPSE_DISTINCT_GROUPS_ENABLED,
             SparseSettings.IS_SPARSE_INDEX_SETTING,
             NeuralSearchSettings.SPARSE_ALGO_PARAM_INDEX_THREAD_QTY_SETTING,
             NEURAL_CIRCUIT_BREAKER_LIMIT,