perf(nlp): run the spaCy pipeline once per distinct text

[r0h1tb]

Problem

_process() is the only expensive call in unstructured/nlp/tokenize.py and the only one without a cache. word_tokenize(), pos_tag() and sent_tokenize() each memoize their own extracted view, which hides the repetition from their callers but not from spaCy.

is_possible_narrative_text() reaches all three while classifying a single element:

_tokenize_for_cache <- sent_tokenize <- sentence_count <- exceeds_cap_ratio <- is_possible_narrative_text
word_tokenize <- exceeds_cap_ratio <- is_possible_narrative_text
pos_tag <- is_possible_narrative_text

Three identical Docs per element. On a 400-paragraph document that is 88% of partition_html() (is_possible_narrative_text 1.60s of 1.81s in cProfile).

Fix

Memoize the Doc so the three extractors share one pipeline run.

Texts over 8 KiB bypass the cache. A Doc is much heavier than the token lists the existing caches hold (~8 KB serialised for a 68-char paragraph) and _process() accepts input up to spaCy's 1M-char limit, so one oversized element could otherwise pin an outsized object for the life of the process.

All three consumers only read from the Doc and each returns a fresh list/tuple, so sharing is safe. _process() has no callers outside this module.

Numbers

	before	after
partition_html, 400 <p>	1127 ms	395 ms
partition_text, 400 paragraphs	1129 ms	403 ms

Same shared classification path, so this is not HTML-specific. Scaling was already linear and stays linear — this is a constant-factor change. The test suite itself drops 38.2s to 33.0s.

Tests

Four tests in test_unstructured/nlp/test_tokenize.py, counting entries into spaCy below every cache in the module rather than timing anything. With the source change reverted on this branch:

FAILED test_the_spacy_pipeline_runs_once_per_distinct_text - assert 3 == 1
FAILED test_repeated_text_does_not_re_run_the_pipeline     - assert 2 == 1
FAILED test_oversized_text_bypasses_the_doc_cache
FAILED test_text_at_the_threshold_is_still_cached

Suite before 146 failed / 2531 passed, after 146 failed / 2533 passed — the same 146 IDs, all pre-existing here (pandoc and the ML extras aren't installed locally). 14 modules that can't be collected without those extras were excluded from both runs.

[cubic-dev-ai]

1 issue found across 4 files

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="unstructured/nlp/tokenize.py">

<violation number="1" location="unstructured/nlp/tokenize.py:153">
P2: When identical text arrives concurrently, `@lru_cache` lets each simultaneous miss run `_run_pipeline`, losing the one-pipeline-per-text optimization. Add per-key single-flight synchronization for threaded callers.</violation>
</file>

<sub>Shadow auto-approve: would not auto-approve because issues were found.

Re-trigger cubic</sub>

[cubic-dev-ai]

P2: When identical text arrives concurrently, @lru_cache lets each simultaneous miss run _run_pipeline, losing the one-pipeline-per-text optimization. Add per-key single-flight synchronization for threaded callers.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At unstructured/nlp/tokenize.py, line 153:

<comment>When identical text arrives concurrently, `@lru_cache` lets each simultaneous miss run `_run_pipeline`, losing the one-pipeline-per-text optimization. Add per-key single-flight synchronization for threaded callers.</comment>

<file context>
@@ -148,10 +150,31 @@ def _get_nlp() -> spacy.language.Language:
     return _load_spacy_model()
 
 
+@lru_cache(maxsize=CACHE_MAX_SIZE)
+def _process_cached(text: str) -> spacy.tokens.Doc:
+    """Memoized `_run_pipeline`, keyed on the text.
</file context>

[r0h1tb]

Both addressed or answered in 468ec3b.

P2, tokenize.py:171 — characters vs bytes. Correct, and the comment was the thing that was wrong. len(text) counts characters, so calling the bound "8 KiB" understates it for non-ASCII — 8,192 CJK characters is roughly 24 KB of UTF-8. Characters are the right unit to bound on, though: Doc size tracks token count, not UTF-8 width, so switching the comparison to encoded length would bound the wrong quantity. Reworded the comment and the changelog to say 8,192 characters; the constant was already named MAX_CACHEABLE_CHARS.

P2, tokenize.py:153 — concurrent misses. Real but left alone. lru_cache isn't single-flight, so two threads asking for the same uncached string can both run the pipeline. The cost is one redundant run that would have happened anyway before this change — not a correctness problem, and the cache converges immediately after. The three existing caches in this module (word_tokenize, pos_tag, _tokenize_for_cache) have the same property. Adding per-key locking here would introduce a lock-ordering surface for a transient duplicate, and only in this one of the four.

[cubic-dev-ai]

0 issues found across 2 files (changes from recent commits).

<sub>Shadow auto-approve: would not auto-approve. Auto-approval blocked by 1 unresolved issue from previous reviews.

Re-trigger cubic</sub>