Add BWC tests for search_after in hybrid query

[sarthakraghuvanshi]

Description

Adds restart-upgrade and rolling-upgrade BWC tests validating that
sort + search_after (deep pagination) on a hybrid query returns
correctly ordered, correctly paginated results across old, mixed, and
upgraded cluster states. Both suites are gated to skip on versions
below 2.16, where sort/search_after support for hybrid query was
introduced.

Verified end-to-end locally (Linux container, real 3.7.0 → current
snapshot upgrade): both restart-upgrade and rolling-upgrade suites
pass, including the 2/3-upgraded mixed-cluster phase.

Related Issues

Resolves #950

Check List

• New functionality includes testing.
• New functionality has been documented.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

No CHANGELOG entry included — per CONTRIBUTING.md this falls under
"Adding, modifying, or fixing tests," which is explicitly listed as
not requiring one.

By submitting this pull request, I confirm that my contribution is
made under the terms of the Apache 2.0 license.

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🧪 PR contains tests

🔒 No security concerns identified

✅ No TODO sections

🔀 No multiple PR themes

⚡ Recommended focus areas for review Missing Version Gate The PR description states the tests are gated to skip on versions below 2.16, but no version check (e.g., assumeTrue on cluster version) is visible in this test. If the old cluster is < 2.16, the test will fail because sort/search_after with hybrid query is not supported. Verify that a version gate is applied, either in this class or in the abstract base class. public void testHybridSearchWithSearchAfter_E2EFlow() throws Exception { waitForClusterHealthGreen(NODES_BWC_CLUSTER); if (isRunningAgainstOldCluster()) { modelId = uploadTextEmbeddingModel(); createPipelineProcessor(modelId, PIPELINE_NAME); createIndexWithConfiguration( getIndexNameForTest(), Files.readString(Path.of(classLoader.getResource("processor/IndexMappingSingleShard.json").toURI())), PIPELINE_NAME ); // docs 0..4 with stock values 10, 20, 30, 40, 50 for (int docId = 0; docId < 5; docId++) { addDocumentWithSortField(getIndexNameForTest(), String.valueOf(docId), TEXTS.get(docId), (docId + 1) * 10); } createSearchPipeline( SEARCH_PIPELINE_NAME, DEFAULT_NORMALIZATION_METHOD, DEFAULT_COMBINATION_METHOD, Map.of(PARAM_NAME_WEIGHTS, Arrays.toString(new float[] { 0.3f, 0.7f })) ); } else { try { modelId = getModelId(getIngestionPipeline(PIPELINE_NAME), TEXT_EMBEDDING_PROCESSOR); loadAndWaitForModelToBeReady(modelId); // doc 5 with stock value 60 and doc 6 with stock value 70 addDocumentWithSortField(getIndexNameForTest(), "5", TEXTS.get(5), 60); addDocumentWithSortField(getIndexNameForTest(), "6", TEXTS.get(6), 70); validateSearchAfterQuery(7, 65, List.of(60, 50, 40, 30, 20, 10)); validateSearchAfterQuery(7, 35, List.of(30, 20, 10)); } finally { wipeOfTestResources(getIndexNameForTest(), PIPELINE_NAME, modelId, SEARCH_PIPELINE_NAME); } } } Missing Version Gate Same concern as the restart-upgrade test: the PR description mentions a 2.16 version gate, but no explicit assumeTrue/version check is present in the visible code. Without it, the test will fail against pre-2.16 old clusters. public void testHybridSearchWithSearchAfter_E2EFlow() throws Exception { waitForClusterHealthGreen(NODES_BWC_CLUSTER); switch (getClusterType()) { case OLD: modelId = uploadTextEmbeddingModel(); createPipelineProcessor(modelId, PIPELINE_NAME); createIndexWithConfiguration( getIndexNameForTest(), Files.readString(Path.of(classLoader.getResource("processor/IndexMappings.json").toURI())), PIPELINE_NAME ); // docs 0..4 with stock values 10, 20, 30, 40, 50 for (int docId = 0; docId < 5; docId++) { addDocumentWithSortField(getIndexNameForTest(), String.valueOf(docId), TEXTS.get(docId), (docId + 1) * 10); } createSearchPipeline( SEARCH_PIPELINE_NAME, DEFAULT_NORMALIZATION_METHOD, DEFAULT_COMBINATION_METHOD, Map.of(PARAM_NAME_WEIGHTS, Arrays.toString(new float[] { 0.3f, 0.7f })) ); break; case MIXED: modelId = getModelId(getIngestionPipeline(PIPELINE_NAME), TEXT_EMBEDDING_PROCESSOR); loadAndWaitForModelToBeReady(modelId); if (isFirstMixedRound()) { // page after stock 45 out of [10, 20, 30, 40, 50] validateSearchAfterQuery(5, 45, List.of(40, 30, 20, 10)); // doc 5 with stock value 60 addDocumentWithSortField(getIndexNameForTest(), "5", TEXTS.get(5), 60); } else { // stock 60 is before the cursor and must be excluded from the page validateSearchAfterQuery(6, 45, List.of(40, 30, 20, 10)); validateSearchAfterQuery(6, 65, List.of(60, 50, 40, 30, 20, 10)); } break; case UPGRADED: try { modelId = getModelId(getIngestionPipeline(PIPELINE_NAME), TEXT_EMBEDDING_PROCESSOR); loadAndWaitForModelToBeReady(modelId); // doc 6 with stock value 70 addDocumentWithSortField(getIndexNameForTest(), "6", TEXTS.get(6), 70); validateSearchAfterQuery(7, 65, List.of(60, 50, 40, 30, 20, 10)); validateSearchAfterQuery(7, 35, List.of(30, 20, 10)); } finally { wipeOfTestResources(getIndexNameForTest(), PIPELINE_NAME, modelId, SEARCH_PIPELINE_NAME); } break; default: throw new IllegalStateException("Unexpected value: " + getClusterType()); } }

[github-actions]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
General	Avoid static mutable test state Using a static mutable field for modelId across test instances can cause state leakage if the test class is instantiated multiple times or run in parallel with other suites. Prefer an instance field, since the value is fetched from the ingestion pipeline in the upgraded branch anyway. qa/restart-upgrade/src/test/java/org/opensearch/neuralsearch/bwc/restart/HybridSearchWithSearchAfterIT.java [50] -private static String modelId = ""; +private String modelId = ""; Suggestion importance[1-10]: 4 __ Why: Valid minor code quality suggestion about avoiding static mutable state, but the pattern is consistent with other BWC tests in the codebase and has minimal actual impact.	Low
	Make mixed-round doc count check more resilient In the MIXED first-round branch, validateSearchAfterQuery is called before adding doc 5, so expectedDocCount is 5. However, in the non-first mixed round, doc 5 was added in the previous round making count 6, but the ordering of assertions calls validateSearchAfterQuery before any add. This is correct only if the second mixed round runs after the first completed. Consider making the doc count check more resilient (e.g., assert >= expected) or explicitly add doc 5 in the else-branch guard to avoid flakiness if rounds execute out of order or repeat. qa/rolling-upgrade/src/test/java/org/opensearch/neuralsearch/bwc/rolling/HybridSearchWithSearchAfterIT.java [81-94] -case MIXED: - modelId = getModelId(getIngestionPipeline(PIPELINE_NAME), TEXT_EMBEDDING_PROCESSOR); - loadAndWaitForModelToBeReady(modelId); - if (isFirstMixedRound()) { - // page after stock 45 out of [10, 20, 30, 40, 50] - validateSearchAfterQuery(5, 45, List.of(40, 30, 20, 10)); - // doc 5 with stock value 60 - addDocumentWithSortField(getIndexNameForTest(), "5", TEXTS.get(5), 60); - } else { - // stock 60 is before the cursor and must be excluded from the page - validateSearchAfterQuery(6, 45, List.of(40, 30, 20, 10)); - validateSearchAfterQuery(6, 65, List.of(60, 50, 40, 30, 20, 10)); - } - break; +// ensure doc 5 exists in case first-round add did not persist to this node view yet +validateSearchAfterQuery(6, 45, List.of(40, 30, 20, 10)); +validateSearchAfterQuery(6, 65, List.of(60, 50, 40, 30, 20, 10)); Suggestion importance[1-10]: 3 __ Why: The suggestion identifies a potential ordering concern but the improved_code is essentially identical to existing code in the else-branch, offering no real change. The concern is speculative for BWC test ordering.	Low

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 83.42%. Comparing base (11b2e13) to head (228aa77).

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #1903      +/-   ##
============================================
- Coverage     83.45%   83.42%   -0.03%     
+ Complexity     3898     3896       -2     
============================================
  Files           291      291              
  Lines         13844    13844              
  Branches       2304     2304              
============================================
- Hits          11553    11550       -3     
- Misses         1456     1457       +1     
- Partials        835      837       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.