Answer exact-term description lookups from a map instead of a Lucene index

[dionmcm]

Branch: perf/exact-term-lookup-without-lucene · Module: snomed-drools-rf2-validator

Problem

DroolsDescriptionIndex has an in-memory Lucene index, but nothing about the question it answers needs a search engine:

• the term was a StringField — a single token, never analysed,
• the query was a TermQuery — exact,
• both clauses were Occur.FILTER — nothing scored,
• the only value read back was a stored field holding the description id.

This is effectively exact-match implemented with an inverted index, and reading the stored id forces LZ4 decompression of a stored-fields block on every call.

Evidence

1,960 worker-thread samples from a full-Snapshot run, attributing each sample to its first frame in library code:

69.6%  DroolsDescriptionIndex.findMatchedDescriptionTerm
24.8%  RuleExecutor worker (actual rule evaluation)
 1.1%  DescriptionHelper.getTag
 0.6%  DroolsConceptService.findStatedAncestorsOfConcept

The leaf frames under it are entirely Lucene mechanics, none of which contribute to the answer:

11.9%  ByteBuffersDataInput.slice
 5.0%  ByteBuffersDataInput.readBytes
 4.8%  LZ4.decompress
 3.7%  HeapByteBufferR.duplicate
 2.8%  ByteArrayDataInput.reset
 2.4%  ByteBuffersDataInput.ensureAssumptions
 2.2%  SegmentTermsEnum.getFrame
 2.1%  DataInput.readVInt
 1.8%  FieldReader.iterator

Change

Two HashMaps — active and inactive — from term to description ids. Same public method, same semantics, immutable result set.

Values are String[] rather than Set<String> deliberately. There is roughly one distinct term per description (2.26M on this edition) and a single-element HashSet costs ~150 bytes of overhead against ~24 for a single-element array;
using sets would have made this structure larger than the Lucene index it replaces, so this keeps heap pressure down. Terms shared by several descriptions are rare, so growing by one on collision is cheaper than carrying a
set everywhere.

Description ids are unique keys of descriptionMap, so the ids held against one term cannot repeat and Set.of is safe.

This was the only Lucene user in the module, so the four now-unused org.apache.lucene dependencies are removed. Removing them also drops the Java 23 or later ... please update Apache Lucene warning from every run.

Tests

DroolsDescriptionIndexTest covers exact match, active/inactive separation of a shared term, several descriptions sharing a term, and — importantly — that lookups stay exact: "Heart", "heart structure" and "Heart structures" must all miss "Heart structure". A tokenising or case-folding replacement would pass the happy path and fail these.

Measurement

Rule execution 208s -> 54s (3.9x).
Total 307s -> 165s. Process CPU 1,832s -> 735s, so this is less work done, not better parallelism.

Building the structure also dropped the gap between Components loaded and Running tests from ~15s to ~4s.

---

About this series

One of five independent performance changes to the Drools validation path, measured against a real 722,404-concept AU edition Snapshot. Each is a single commit on master, builds and passes the full suite alone, and leaves the findings byte-identical: 10,949 before and after, with matching report TSVs.

Together they take a full-Snapshot validation from 636s to 118s.