Deduplicate ingest inference inputs within sub-batches

[linjiaye929]

Description

Deduplicates equal text inference inputs within a single ingest sub-batch, while preserving the existing document and field order. Each unique input is inferred once and its result is scattered to every original position with independent mutable result objects.

This applies only to the dense text_embedding and sparse sparse_encoding ingest batching paths. Sparse WORD and TOKEN_ID requests remain separate inference boundaries. Existing skip_existing filtering and batch error propagation are unchanged.

This does not add a search cache, in-flight search coalescing, cross-request cache, node-local TTL cache, or cluster-wide cache.

Related Issues

Refs #1558

Test evidence

• gradle.bat test -PskipSpotlessEclipse -x spotlessApply -x buildJniLib -x cmakeJniLib --tests "org.opensearch.neuralsearch.processor.InferenceProcessorTests" --tests "org.opensearch.neuralsearch.processor.TextEmbeddingProcessorTests" --tests "org.opensearch.neuralsearch.processor.SparseEncodingProcessorTests" --no-daemon --max-workers=1 — passed: 112 tests, 0 failures (6 shared, 69 dense, 37 sparse).
• Regression tests failed before the production change with dense input [tiny, duplicate, duplicate, lengthy input] instead of [tiny, duplicate, lengthy input], and sparse format inputs [value1, value1] instead of [value1].

The standard spotlessJavaCheck could not configure locally because the Eclipse JDT formatter P2 mirror timed out; no formatter configuration change is included.

Check List

• New functionality includes testing.
• New functionality has been documented in the changelog.
• API changes companion pull request is not applicable; there is no API change.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR is not applicable; this changes no public API or configuration.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🧪 PR contains tests

🔒 No security concerns identified

✅ No TODO sections

🔀 No multiple PR themes

⚡ Recommended focus areas for review Shallow copy of nested results copyInferenceResult performs a shallow copy: for List results it creates a new ArrayList wrapping the same element references, and for Map results a new LinkedHashMap sharing values. When the same inference input appears in multiple field positions of different documents, downstream code that mutates nested elements (e.g., inner maps of token weights, or nested lists of floats) will observe cross-document mutation because those inner objects are still shared. The tests only assert reference inequality of the outer container, not of nested contents, so this may go undetected. Consider deep-copying nested mutable structures or documenting that consumers must not mutate values in place. private Object copyInferenceResult(Object result) { if (result instanceof List<?> listResult) { return new ArrayList<>(listResult); } if (result instanceof Map<?, ?> mapResult) { return new LinkedHashMap<>(mapResult); } return result; }

[github-actions]

PR Code Suggestions ✨

Explore these optional code suggestions:

Category	Suggestion	Impact
Possible issue	Deep-copy scattered inference results copyInferenceResult only performs a shallow copy of the outer List or Map. Since duplicate results are scattered to multiple documents that may then mutate nested elements (e.g., pruning token maps in-place), sharing nested references can cause cross-document corruption. Consider deep-copying nested collections, or ensure downstream consumers never mutate the shared inner structures. src/main/java/org/opensearch/neuralsearch/processor/InferenceProcessor.java [280-288] private Object copyInferenceResult(Object result) { if (result instanceof List<?> listResult) { - return new ArrayList<>(listResult); + List<Object> copy = new ArrayList<>(listResult.size()); + for (Object element : listResult) { + copy.add(copyInferenceResult(element)); + } + return copy; } if (result instanceof Map<?, ?> mapResult) { - return new LinkedHashMap<>(mapResult); + Map<Object, Object> copy = new LinkedHashMap<>(); + for (Map.Entry<?, ?> entry : mapResult.entrySet()) { + copy.put(entry.getKey(), copyInferenceResult(entry.getValue())); + } + return copy; } return result; } Suggestion importance[1-10]: 6 __ Why: The suggestion correctly identifies a potential issue where shallow copies of List or Map inference results could lead to cross-document corruption if downstream consumers mutate nested elements. However, whether this is an actual bug depends on downstream mutation behavior, which is not fully evident from the diff.	Low