PAPER_EN.tex

\documentclass[11pt]{article}

\usepackage[margin=1.15in]{geometry}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{amsmath,amssymb,amstext}
\usepackage{newpxtext}
\usepackage[vvarbb]{newpxmath}
\usepackage{microtype}
\usepackage{graphicx}
\usepackage{booktabs}
\usepackage{enumitem}
\usepackage{xcolor}
\usepackage{array}
\usepackage{caption}
\usepackage{titlesec}
\usepackage{fancyhdr}
\usepackage[normalem]{ulem}
\usepackage{hyperref}

\definecolor{headingcolor}{HTML}{1F3864}
\definecolor{linkcolor}{HTML}{2A5DB0}

\hypersetup{
  colorlinks=true,
  linkcolor=linkcolor,
  citecolor=linkcolor,
  urlcolor=linkcolor,
  pdftitle={Synchronous Online Verification Gating in Semantic Caches},
  pdfauthor={Chengyou Xin}
}

% Section headings: sans, bold, small-caps-ish color accent to set them
% apart from the serif body text without introducing a second display face.
\titleformat{\section}
  {\normalfont\sffamily\bfseries\large\color{headingcolor}}
  {\thesection}{0.6em}{}
\titleformat{\subsection}
  {\normalfont\sffamily\bfseries\color{headingcolor}}
  {\thesubsection}{0.6em}{}
\titlespacing*{\section}{0pt}{1.6em}{0.7em}
\titlespacing*{\subsection}{0pt}{1.2em}{0.4em}

\captionsetup{font=small,labelfont={bf,sf},labelsep=colon,margin=8pt,skip=8pt}

\renewcommand{\arraystretch}{1.2}
\setlist[itemize]{itemsep=3pt,topsep=4pt}
\setlist[enumerate]{itemsep=3pt,topsep=4pt}

\pagestyle{fancy}
\fancyhf{}
\fancyhead[L]{\small\sffamily\textsc{Synchronous Verification Gating in Semantic Caches}}
\fancyhead[R]{\small\sffamily\thepage}
\renewcommand{\headrulewidth}{0.4pt}
\fancypagestyle{plain}{
  \fancyhf{}
  \fancyfoot[C]{\small\sffamily\thepage}
  \renewcommand{\headrulewidth}{0pt}
}

\newcommand{\taulow}{\ensuremath{\tau_{\text{low}}}}
\newcommand{\tauhigh}{\ensuremath{\tau_{\text{high}}}}

\title{\textbf{Synchronous Online Verification Gating in Semantic Caches}\\[6pt]
  {\Large\normalfont An Empirical Study}}
\author{Chengyou Xin \\[2pt] \small\textsc{LoopDot AI Research}}
\date{\small 2026-07-26}

\begin{document}

\maketitle
\thispagestyle{plain}

\begin{center}
\small\textbf{Code and full experimental artifacts:} this repository, \texttt{cacheverifier/}, \texttt{results/}, \texttt{configs/}
\end{center}
\vspace{4pt}

\begin{quote}
\noindent\textbf{Erratum (added 2026-08-09, after the paper's initial release / Zenodo DOI 10.5281/zenodo.21703365 and sharing on HN/Reddit/Twitter)}

\noindent The SearchQueries dataset's Group D/E results had a data defect: the upstream vCache HuggingFace dataset (\texttt{vCache/\allowbreak SemBenchmarkSearchQueries})'s \texttt{response\_\allowbreak llama\_\allowbreak 3\_\allowbreak 8b} column was never actually populated for this split --- all 150,000 records' \texttt{answer} field was the identical constant placeholder string (``Not required for the benchmark because of the id\_set''), not the ``Llama-3-8B-generated answers'' this paper originally described. This meant the Group D/E cross-encoder verifier's second scoring input (the candidate answer text) was identical across every candidate and carried no discriminative information --- sufficient on its own to explain the previously reported ``near-chance'' verifier performance on SearchQueries, with no need for a more complex semantic explanation (this paper's working ``axis problem'' hypothesis, see \texttt{RESEARCH\_\allowbreak PROPOSAL.md} \S10 direction 5, has also been ruled out as the primary cause).

\noindent Real answers have been regenerated with the DeepSeek API (\texttt{deepseek-chat}; not Llama-3-8B, since no API access to that specific model was available) for the 103,188 affected records (the candidates actually read by the verifier in the gray zone, 68.8\% of the dataset), and the full Group D/E results --- including all downstream ablations (Section~\ref{sec:deployment}) --- were rerun using the identical match trace and a threshold grid recalibrated to the corrected model's own score distribution (rather than reusing the old grid --- exactly the lesson this paper's own Section~\ref{sec:groupd} already emphasized), on the same machine that produced the original Group E results. Every SearchQueries figure in the body below has been replaced with the corrected version; the original (defective) result files remain archived in \texttt{results/\allowbreak search\_\allowbreak queries\_\allowbreak groupD\_\allowbreak cross\_\allowbreak encoder.json} / \texttt{search\_\allowbreak queries\_\allowbreak groupE\_\allowbreak finetuned.json} for diffing. \textbf{The correction is not uniformly ``better.''} The fine-tuned Group E result is now stronger than originally reported, but the off-the-shelf Group D result is now worse --- the original data made it look merely ``close to ineffective''; the corrected data reveals it is genuinely worse than doing nothing at most operating points (details in Sections~\ref{sec:groupd}/\ref{sec:groupe}). The three downstream ablations (label-noise robustness, cold-start data requirements, drift decay) each moved in the same direction: the placeholder data had systematically \emph{understated} these deployment risks. This defect does not affect any LmArena, Quora, or Twitter customer-support results --- none of them ever depended on the missing column.
\end{quote}

\begin{abstract}
\noindent
Semantic caches replace exact matching with vector similarity to reuse an LLM's past answers, but similarity and answer correctness are not the same quantity: static similarity thresholds have been reported in production to produce false-positive rates in the double digits. A recent line of work, vCache, learns per-embedding adaptive thresholds online in exchange for a formal error-rate bound; another, Krites, introduces an LLM judge in the similarity gray zone, but that verification is designed to run \textbf{asynchronously}, off the serving path, and its own evaluation always substitutes a ground-truth oracle for the real judge --- the paper itself discusses ``blocking, synchronous verification'' as an alternative but neither implements nor measures it. This paper fills that specific gap: under a single-tier cache architecture, using a \textbf{real} (non-oracle) lightweight verifier, gating cache hits \textbf{synchronously} and online, and running a formal Pareto-frontier comparison against static-threshold and adaptive-threshold baselines on two public benchmarks (SemCacheLMArena and SemCacheSearchQueries, roughly 210{,}000 real requests combined).

The headline finding is \textbf{conditional, not unqualified}: with an oracle verifier, synchronous gating raises hit rate by 20--28 percentage points at matched error rate on both datasets, proving the mechanism has substantial theoretical headroom. Swapping in an off-the-shelf real verifier (an ms-marco cross-encoder) cashes in only a small slice of that headroom, and only on the conversational dataset (LmArena) --- under the fairest possible comparison (interpolated against the static-threshold frontier), the best reproducible net gain is about +1.9 percentage points of hit rate (statistically significant but modest), while on the short-query dataset (SearchQueries) this off-the-shelf verifier \textbf{causes more net harm than benefit} (36 tested operating points: 1 win, 12 ties, 23 losses to the static-threshold frontier; held-out AUC 0.5984, above chance but still not enough to reliably beat an already-strong similarity baseline) --- \textbf{[2026-08-15 update]} but this ``net harmful'' verdict itself used a post-hoc grid-searched threshold; retested with honest calibration (chronological calibration/test split, Youden's J threshold selection --- see the end of Section~\ref{sec:groupd}), the same untuned verifier instead beats the static-threshold frontier at every tested point. A substantial share of the original verdict was itself a product of the grid search, not solely a reflection of the verifier lacking discriminative power --- a qualification discovered after this paper's initial Go/No-Go evaluation, and one that should be weighed alongside the ``weak Go'' verdict discussed below. Accordingly, this paper's Go/No-Go verdict is a \textbf{weak Go}: the mechanism is not falsified, but the naive expectation that ``any off-the-shelf verifier will meaningfully improve the cache'' is refuted by the evidence --- how well the verifier matches the data domain is the variable that actually determines whether this mechanism has practical value. A follow-up experiment (Section~\ref{sec:groupe}) tests this diagnosis directly: fine-tuning the same cross-encoder on each dataset's own gray-zone labels raises held-out AUC from 0.72 to 0.88 on LmArena and, more importantly, from 0.5984 to 0.7120 on SearchQueries --- turning that net-harmful result into a verifier that beats the static-threshold frontier at 53 of 54 tested operating points, 1 tie, zero losses (best net lead +4.50 percentage points). A third, independently-sourced dataset (Quora Question Pairs, outside vCache's own benchmarks) replicates the same pattern at a smaller magnitude proportional to its own lower headroom: across all three datasets tested, fine-tuning never produced a worse result than the untuned verifier. \textbf{[2026-08-16 update]} This grid-search methodology could itself carry the same optimism bias; retested with the identical honest calibration used above (chronological calibration/test split, Youden's J), the zero-loss verdict holds unchanged, LmArena/SearchQueries's net lead actually increases, and Quora's narrows without reversing direction --- see the end of Section~\ref{sec:groupd}. Three further ablations (Section~\ref{sec:deployment}) test whether this survives realistic deployment conditions: fine-tuning tolerates label noise up to roughly 30\% before turning harmful, keeps improving with more cold-start data with no observed saturation point, and --- contrary to expectation --- shows no continued decay in benefit as production traffic drifts away from its fine-tuning window, all three findings holding consistently across datasets. A fourth check (Section~\ref{sec:twitter}) repeats the recipe on real production customer-support traffic (two brands from a public Twitter support-ticket corpus, spanning up to 3.5 years): one brand (e-commerce) replicates every prior finding, but the other (telecom, the longest and only multi-year traffic history tested) is a genuine counter-example --- fine-tuning turns harmful regardless of label noise, more cold-start data makes it worse rather than better, and its benefit decays continuously rather than plateauing --- all three traced to a single cause, a non-stationary gray-zone positive rate across the traffic's history, which revises Section~\ref{sec:deployment}'s re-tuning-cadence conclusion from unconditional to conditional on monitoring that stability. A prototype monitor for exactly that instability (Section~\ref{sec:monitor}) --- two classical change-point tests run on gray-zone labels alone, no extra inference required --- flags the shift partway through the affected brand's held-out traffic while never false-alarming on the unaffected one, turning this from an identified risk into a demonstrated, low-cost detection capability. \textbf{[2026-08-17 update]} Two methodological gaps previously flagged as limitations have now been closed (Sections~\ref{sec:latency}, \ref{sec:tauhigh}): the oracle's latency was an unmeasured 70ms modeling assumption, and a real API measurement shows that figure was low by roughly 24x (measured mean 1687.8ms), directly weakening the claim that synchronous verification's latency cost is negligible; \tauhigh\ was previously fixed at a single anchor, and sweeping it reveals its optimal value is dataset-dependent --- widening \tauhigh\ more than triples the net lead on LmArena, while the same change flips SearchQueries (corrected) to a net loss. A third correction targets Group B (Section~\ref{sec:groupb}): a line-by-line comparison against the official vCache implementation found that the official code seeds every cache entry with two synthetic bootstrap observations this paper's port lacked, making this paper's original reproduction harder to exit cold start than the official algorithm; after the fix, Group B's hit rate rises 4.4x--29.1x across all three datasets while error rate stays below target $\delta$ throughout (the formal guarantee is unaffected) --- the originally reported hit-rate ranges in Section~\ref{sec:groupb} should be read as this paper's prior port, not the official vCache algorithm's true ceiling. \textbf{[2026-08-19 update]} Section~\ref{sec:reranker} tested whether a larger or more broadly-trained off-the-shelf reranker closes the SearchQueries gap: \texttt{ms-marco-MiniLM-L12-v2} (same distribution, pure capacity increase) and \texttt{BAAI/\allowbreak bge-reranker-base} (broader training distribution, also larger) produced nearly identical gains (+0.0226 vs.\ +0.0214) on the one dataset that actually separates the two variables (SearchQueries) --- neither ``the model is too small'' nor ``the training distribution is wrong'' explains the gap; off-the-shelf model selection, in either direction, fails to meaningfully close it within the scope tested, and the in-domain fine-tuning from Section~\ref{sec:groupe} remains the only verified remedy. \textbf{[2026-08-19 update]} Section~\ref{sec:crc} upgrades gray-zone threshold calibration from a point estimate to finite-sample risk control: Conformal Risk Control (CRC) gives a formal guarantee --- ``reusing this hit keeps the error rate under a user-specified budget $\alpha$'' --- under a random-split assumption, and across three datasets and four alpha levels (5\%/2\%/1\%/0.5\%), the certified risk's efficiency loss relative to a full-distribution oracle stays within 3\% of 1.0 throughout; the finite-sample cost is essentially negligible at the sample sizes tested. The same framework surfaces two new findings: a verifier's AUC ranking does not equal its risk-constrained reuse-utility ranking (SearchQueries, the lowest-AUC dataset, has higher reuse throughout than higher-AUC Quora); and re-running the same calibration under a chronological (rather than random) split leaves LmArena safe --- its drift direction happens to favor conservatism --- but produces a genuine, systematic guarantee violation on Quora (exceeding target at all four alpha levels, up to 80\% relative overshoot) --- this paper's first measurable exchangeability failure, showing that calibrating on historical traffic and deploying to future traffic cannot be assumed safe by default. A direct numeric comparison against vCache was judged infeasible after a methodology audit (three structural mismatches: decision granularity, synchronous vs.\ asynchronous verification, and guarantee type), replaced with a structural comparison table; see Section~\ref{sec:crc} for details. \textbf{[2026-08-20 update]} Section~\ref{sec:rewrite} is a negative ablation: it tests whether the TweakLLM paradigm (``rewrite a rejected candidate's answer with a lightweight LLM instead of calling it a miss'') beats the existing binary gate. Across three datasets, the rewritten-then-judged error rate (22\%--61\%, dataset-dependent) is far higher than the binary gate's status quo of ``reject means miss, zero error risk,'' and the rewrite call's own latency (mean 2.7--6.2 seconds) is not clearly lower than a fresh generation's would be either. A counter-intuitive headline finding: the false-reject stratum, where the candidate should have been accepted, shows almost no difference in judged-incorrect rate from the true-reject stratum, where the candidate genuinely was a different question (95\% confidence intervals overlap on all three datasets) --- whether the rewrite succeeds appears almost unrelated to whether the verifier's original rejection was itself justified. \textbf{[2026-08-22 update]} Section~\ref{sec:topk} tests a design choice made everywhere else in this paper: retrieving a single nearest neighbor only. An external, independent re-derivation of this project flagged that \texttt{VectorCacheStore} hardcodes \texttt{k=1}, so if that one candidate falls in the gray zone and is rejected, the request is judged a miss even when the semantic second-nearest neighbor would have been correct and was never seen. A prior monotonicity argument (HNSW returns neighbors in descending similarity order, so once rank-1 is below \taulow\ no later candidate can be better) narrows the test to exactly one branch: rank-1 in the gray zone and rejected. A pure ground-truth ceiling (no verifier calls) shows non-trivial headroom on all three datasets: 0.75\% of all requests on LmArena, 1.62\% on Quora, 5.66\% on SearchQueries (corrected) have a correct candidate that was never surfaced. Plugging in a real verifier (off-the-shelf and the same fine-tuned checkpoint as Section~\ref{sec:groupe}) to test a K=2 cascade again produces three different stories: on LmArena, hit rate rises significantly while error rate never moves significantly at any grid point --- close to a free lunch; on Quora, the effect is essentially zero, consistent with its small ceiling; on SearchQueries there is a real trade-off --- 43\%--53\% of 36--54 grid points show both hit rate and error rate rising significantly, and fine-tuning narrows that share from 53\% to 43\% without eliminating it. K and fine-tuning are complementary, not substitute, levers: at a fixed error-rate ceiling, using both always yields the highest hit rate. \textbf{[2026-08-28 update]} Section~\ref{sec:selfselect} tests an open question left by Section~\ref{sec:crc}'s CRC work: Protocol R/T both test static splits of a fixed historical trace, not a genuine online closed loop --- if the CRC gate's own reuse/reject decisions feed back into future cache state, that could induce a self-selection feedback loop. A genuine online closed-loop simulation (built without modifying the experiment framework the rest of this paper depends on) compares ``insert regardless of hit/miss'' (this paper's existing convention elsewhere) against ``insert only on miss'' (closer to real production semantics). The result is a precise monotonic dose-response relationship: the higher the direct-hit rate (Quora 3.7\% $<$ SearchQueries 28.1\% $<$ LmArena 72.8\%), the more self-selection sparsifies the cache and the larger the resulting harm --- no detectable effect on Quora, a moderate significant harm on SearchQueries, and on LmArena realized risk more than triples (strongly significant). A case-by-case audit of 1{,}098 false-accepts on LmArena confirms 68.9\% match a specific mechanism: a correct candidate that never earns an independent cache write because it keeps being successfully reused, forcing later equivalent queries to match an incorrect substitute --- not a statistical coincidence. Online recalibration fully compensates for this harm on Quora and SearchQueries, but not on LmArena, leaving an unlocated threshold between the two as an open question. \textbf{[2026-08-29 update]} A cleaner single-variable manipulation locates this threshold: generalizing the binary ``insert on hit'' switch into a continuous write probability $p$ (rather than approximating it indirectly via \tauhigh\ or synthetic duplicate data, both of which would introduce additional confounds), swept over $p \in \{0.1, 0.25, 0.5, 0.75\}$ on LmArena. The harm does decrease monotonically with $p$, but slowly --- at $p=0.75$, residual harm is still only about 35\% lower than at $p=0$, and the effect never leaves the significant range anywhere across the tested interval. The qualitative audit was also extended to all three datasets: the crowding-out mechanism explains 68.9\% of LmArena's false-accepts but only 12.4\% of SearchQueries' explainable cases --- SearchQueries' self-selection harm is mostly driven by a different, still-unidentified mechanism, so the mechanism verified on LmArena cannot simply be assumed to generalize to other datasets. \textbf{[2026-08-30 update]} Section~\ref{sec:costsensitive} replaces the hit-rate/error-rate frontier with an explicit cost-ratio analysis: $\mathrm{cost}(r) = r \times \mathrm{error\_rate} + (1 - \mathrm{hit\_rate})$, where $r$ is the cost of an error relative to the cost of a miss. After closing a grid-coverage gap in D/E's honest calibration (extended to $\taulow \in \{0.80,...,0.99\}$ to match Group A, via 6 new GPU-computed operating points --- the original narrower grid had produced a spurious ``static threshold wins again at high $r$'' artifact on LmArena that fully disappears once closed), all three datasets show the same qualitative result: when the cost of an error clearly exceeds the cost of a miss (typically $r \gtrsim 1$--9, dataset-dependent), synchronous verification is economically superior to a static threshold almost universally; only when misses and errors cost about the same, or misses cost more, does a static threshold aimed at maximum hit rate win. Section~\ref{sec:redteam} uses an LLM (DeepSeek) to generate adversarial samples covering five known failure axes (negation, action-verb swap, direction reversal, named-entity swap, quantity swap) and tests the verifier's worst-case behavior on deliberately constructed hard cases rather than natural benchmark data: on 306 samples clearing the gray-zone threshold, the off-the-shelf verifier's false-accept rate is 84.0\% (95\% CI [79.7\%,87.9\%]), far exceeding anything measured on natural data elsewhere in this paper; more importantly, the in-domain fine-tuning shown in Section~\ref{sec:groupe} to substantially repair natural-data discriminative power provides \textbf{no protection whatsoever} on these adversarial samples (87.6\%, CI nearly fully overlapping the untuned result) --- fine-tuning learns this dataset's natural noise distribution, not a general robustness to deliberately constructed attacks, and the two are different capabilities. \textbf{[2026-08-30 update]} Section~\ref{sec:advtrain} directly tests Section~\ref{sec:redteam}'s open question (d): can training specifically targeted at these five failure axes close the gap? Mixing a fresh, non-overlapping batch of 223 adversarial training triples (only 3.8\% of the combined training set) into Section~\ref{sec:groupe}'s original natural training data and re-running the identical fine-tuning procedure leaves natural-data AUC essentially unchanged (0.8749 vs.\ the original 0.88) but cuts the adversarial held-out false-accept rate from 84--88\% to 53.6\% (95\% CI [48.0\%,59.2\%], non-overlapping with either baseline) --- a real, significant improvement proving this capability gap is not a fundamental limitation of the fine-tuning mechanism, just a missing slice of training data. But 53.6\% still means more than half of adversarial samples fool the verifier, and one category (named-entity swap) actually got worse (48.7\%$\to$61.5\%) --- substantially mitigated, far from solved.
\end{abstract}

\section{Introduction}
\label{sec:intro}

Semantic caching --- replacing exact matching with vector similarity --- is a standard technique for cutting the cost and latency of LLM applications today. AWS reports that semantic caching can cut cost by up to 86\% and improve latency by up to 88\%. But similarity and answer correctness are different quantities: a poorly configured semantic cache can have a false-positive rate as high as 99\%, and engineering teams have had to ship emergency hotfixes after a cache served an incorrect refund policy.

Existing cache-hit logic is essentially $\text{sim}(q, h) \geq \tau \rightarrow \text{hit}$, where $\text{sim}$ measures \emph{Question $\approx$ Question$'$}, while the question that actually matters is \emph{does Question $\rightarrow$ Answer still hold} --- an answer-relevance / entailment problem, not a similarity problem. ``Can my dog eat honey'' and ``what's the deal with dogs and honey'' can score highly similar, but ``how do I pause my subscription'' versus ``how do I cancel my subscription'' --- a pair with high similarity and completely different correct answers --- is exactly what a static threshold cannot distinguish.

Three published works each address part of this problem (Section~\ref{sec:related}), but none has directly measured: \emph{under a single-tier architecture with no offline curation, and with a real model rather than ground truth standing in for the verifier, is it actually worth putting verification on the serving path?} That is this paper's question:

\begin{quote}
\itshape
Under a single-tier cache architecture, can a real, lightweight verifier doing synchronous online gating achieve a higher hit rate than existing methods (static threshold, adaptive threshold) at a comparable error rate --- or a lower error rate at a comparable hit rate?
\end{quote}

This paper's contribution is purely empirical:

\begin{enumerate}[leftmargin=*]
  \item A reproducible evaluation pipeline (HNSW approximate nearest neighbor + a single-pass cached trace + cheap grid replay) that makes a full four-group sweep over roughly 210{,}000 real requests tractable. The nearest-neighbor search that this sweep would otherwise repeat once per grid point is the dominant cost under brute force --- extrapolating to about 19 hours for Groups A and B's threshold/delta grids alone --- and collapses to a single, minutes-long pass under HNSW; verifier inference (Group D) and bootstrap confidence intervals remain separate, non-negligible costs that this optimization does not eliminate (see Section~\ref{sec:groupd} and Section~\ref{sec:latency}).
  \item A line-by-line port of vCache's \texttt{VerifiedDecisionPolicy} (logistic-regression threshold estimation, delta-method variance, randomized exploration probability), not a simplified approximation.
  \item Two measurements missing from the current literature: (a) the real (non-oracle) false-approve/false-reject rate of a verifier in a semantic-caching setting; (b) the actual latency cost of synchronous versus asynchronous verification.
  \item An honest, interpolation-corrected Go/No-Go verdict --- the mechanism works but is highly verifier-dependent, rather than a blanket ``verification helps'' or ``verification doesn't help'' conclusion.
\end{enumerate}

\section{Related Work}
\label{sec:related}

\begin{table}[h]
\centering
\small
\begin{tabular}{@{}p{2.1cm}p{3.6cm}p{2.2cm}p{2.2cm}p{3.6cm}@{}}
\toprule
\textbf{Work} & \textbf{Core method} & \textbf{Introduces a verifier?} & \textbf{Sync/async} & \textbf{Architectural precondition} \\
\midrule
GPTCache (Bang, 2023) & Global fixed similarity threshold & No & --- & Single-tier \\
\addlinespace
vCache (Schroeder et al., 2025) & Online per-embedding threshold learning with an error-rate convergence guarantee & No & --- & Single-tier \\
\addlinespace
Krites (2026) & Static threshold unchanged; gray-zone matches trigger an \textbf{asynchronous} LLM judge that, on approval, promotes the answer into a dynamic tier & Yes, but only for promotion & \textbf{Asynchronous, off the serving path} & Two-tier: offline-curated static tier + online dynamic tier \\
\addlinespace
Calibration Gap (Baral et al., 2026) & Calibrates the single similarity score already produced by the retrieval/reranking stage (P-CHR AUC / CRR); adds no separate stage & No --- calibrates the one existing signal, not a decoupled second-stage verifier & --- & Single-tier, one-signal architecture (better selection/calibration, still ``score $\geq$ threshold $\rightarrow$ hit'') \\
\addlinespace
TweakLLM (2025) & On a match, skips accept/reject and uses a lightweight LLM to \textbf{dynamically rewrite} the cached answer to fit the new query & Partial --- substitutes generative rewriting for a binary decision & Synchronous, on the serving path & Single-tier, but reframes ``verification'' as ``editing'' \\
\bottomrule
\end{tabular}
\caption{Positioning relative to prior semantic-caching work.}
\end{table}

The last two rows are not Krites-style direct competitors but represent two adjacent, different lines of work --- ``calibrate the existing signal better'' and ``sidestep verification by editing the content instead'' --- whose relationship to this paper's positioning is discussed via the external cross-validation in Section~\ref{sec:golo}; they are not analyzed against the same three points below.

Three points fix this paper's position relative to Krites specifically:

\begin{enumerate}[leftmargin=*]
  \item Krites explicitly discusses putting the judge directly on the serving path for blocking verification, and predicts that ``such a policy adds an extra model call to many requests, increasing both average and tail latency, eroding the cache's own benefit'' --- but \textbf{neither implements nor measures it}.
  \item Krites' evaluation ``does not run the LLM judge in simulation\ldots\ but instantiates $J$ directly from the benchmark's ground-truth equivalence-class relation,'' and its Discussion concedes that ``in production, an LLM-based verifier will have a non-zero false-reject and false-approve rate,'' but only offers an analytical upper bound --- \textbf{no empirical measurement}.
  \item Krites' architectural precondition is a two-tier cache (an offline-curated static tier plus an online dynamic tier); its contribution is ``promoting a verified static answer into the dynamic tier,'' not ``judging in real time whether this hit can be trusted.'' The mechanism does not apply to a single-tier dynamic cache deployment with no offline curation pipeline.
\end{enumerate}

This paper's four experimental groups map directly onto those three gaps: Groups A and B reproduce existing baselines; Group C keeps Krites' own oracle verifier-fidelity assumption but makes verification synchronous, isolating the cost of ``going synchronous'' as a single variable; Group D swaps in a real verifier, isolating the cost of ``oracle $\rightarrow$ real'' as a single variable.

\section{Method}

\subsection{Four experimental groups}

\begin{table}[h]
\centering
\small
\begin{tabular}{@{}lp{2.6cm}p{9cm}@{}}
\toprule
\textbf{Group} & \textbf{Method} & \textbf{Description} \\
\midrule
A & Static threshold & $\text{sim} \geq \tau \rightarrow \text{hit}$, reproducing GPTCache; $\tau$ taken from the \texttt{STATIC\_\allowbreak THRESHOLDS} grid in vCache's own \texttt{benchmarks/\allowbreak benchmark.py} \\
\addlinespace
B & Adaptive threshold & A line-by-line port of vCache's \texttt{VerifiedDecisionPolicy} (Section~\ref{sec:groupb}), not a simplified approximation \\
\addlinespace
C & Sync + oracle verifier & The gray zone synchronously calls a ``perfect'' verifier (judging via ground-truth equivalence classes), but assigns it a non-zero modeled latency standing in for a real LLM judge (70ms default) \\
\addlinespace
D & Sync + real verifier & The gray zone synchronously calls a real, non-oracle model \\
\bottomrule
\end{tabular}
\caption{The four experimental groups.}
\end{table}

C and D share the same decision mechanism (\texttt{SynchronousVerifiedPolicy}):

\begin{quote}
\begin{verbatim}
sim(q, h) >= tau_high        -> serve directly (high confidence, no verifier call)
tau_low <= sim(q, h) < tau_high -> call the verifier synchronously;
                                    serve if approved, else fall through
sim(q, h) < tau_low          -> fall through directly (low confidence, no verifier call)
\end{verbatim}
\end{quote}

The only difference between C and D is the verifier implementation --- that is the point of the design: C isolates ``synchronous vs.\ asynchronous'' (under Krites' own verifier-fidelity assumption), and D isolates ``oracle vs.\ real model.''

\subsection{A faithful port of Group B}
\label{sec:groupb}

In vCache's official code (\texttt{vcache-project/\allowbreak vCache}, \texttt{vcache/\allowbreak vcache\_\allowbreak policy/\allowbreak strategies/\allowbreak verified.py}), \texttt{VerifiedDecisionPolicy}'s core logic is:

\begin{enumerate}[leftmargin=*]
  \item Each cache entry independently maintains its own observation history $(\text{similarity}, \text{is\_correct})$, with \textbf{no global fallback across entries} --- with fewer than 6 observations, the policy always declines to exploit (EXPLORE).
  \item Once an entry reaches 6 observations, a 1-D logistic regression (similarity $\rightarrow$ correctness) is fit on its history, yielding a threshold estimate $\hat{t}$ and a slope $\gamma$.
  \item The variance $\text{var}_t$ of $\hat{t}$ is estimated via the delta method (or, under perfect separation, looked up in an empirical variance table shipped with the paper's own code).
  \item Over a 50-point epsilon grid, a set of candidate confidence upper bounds $t'$ is computed and inverted into an ``explore probability'' $\tau_{\text{explore}}$, such that the expected error rate stays below the target $\delta$.
  \item A random draw $u \sim \text{Uniform}(0,1)$ decides the action: explore (miss, verify, and update the observation history) if $u \leq \tau_{\text{explore}}$, otherwise exploit (hit, no verification).
\end{enumerate}

This paper's \texttt{AdaptiveThresholdPolicy} ports all five steps line by line, including the exact empirical variance-table values from the paper's own code, rather than substituting a simplified confidence-interval approximation. This distinction matters: an earlier, simpler Wilson-score-upper-bound approximation (used in an early version of this codebase) introduced a global fallback mechanism absent from the original paper, which systematically overstated hit rate.

\subsection{Computational efficiency: one pass, cheap replay}

Because every request --- hit or miss --- is unconditionally inserted into the cache (matching vCache's own harness), the sequence of ``which historical entry does this request match'' for Groups A/C/D is \textbf{entirely independent of the threshold parameters}. The evaluation pipeline exploits this by splitting the computation into three steps:

\begin{enumerate}[leftmargin=*]
  \item \texttt{build\_\allowbreak match\_\allowbreak trace}: one pass over the whole stream using HNSW (rather than brute force), recording each request's nearest-neighbor match --- computed exactly once.
  \item \texttt{score\_\allowbreak gray\_\allowbreak zone}: the verifier scores, exactly once, every candidate whose similarity falls within the union of the entire parameter sweep.
  \item \texttt{replay}: for each $(\taulow, \tauhigh, \text{threshold})$ combination, hit/miss decisions are cheaply re-derived from the cached results of the two steps above --- without touching the ANN index or the verifier model again.
\end{enumerate}

An empirical extrapolation of brute-force nearest-neighbor search (20{,}000 records, 1024 dimensions, 56 seconds per pass) shows that, without this optimization, repeating that search once per grid point across Groups A and B's threshold/delta grids on roughly 210{,}000 real records would take about 19 hours; with HNSW, the single shared pass collapses to minutes. This does not make the whole pipeline minutes-long end to end --- Group D's real verifier inference and the 200-resample bootstrap confidence intervals are separate costs unaffected by the ANN backend (Sections~\ref{sec:groupd} and~\ref{sec:latency}) --- but it removes what would otherwise be the dominant, and otherwise-prohibitive, bottleneck. This fast path was cross-checked record-by-record against the slow path (\texttt{ExperimentRunner} + \texttt{SynchronousVerifiedPolicy}) run one grid point at a time, under identical parameters, and produced identical results.

\subsection{Ground truth for correctness}

Following vCache's own harness: whether two records' answers are equivalent is judged by the dataset's own equivalence-class label (\texttt{ID\_\allowbreak Set} / \texttt{id\_\allowbreak set}), not by calling a real LLM to compare semantics. A hit's correctness is whether the matched historical entry shares the current request's equivalence class; on a miss, the same check (``would it have been correct had it hit'') is also performed, so every single request yields a full TP/FP/TN/FN entry in the confusion matrix.

\section{Experimental Setup}

\subsection{Datasets}

\begin{table}[h]
\centering
\small
\begin{tabular}{@{}lp{2cm}p{5cm}p{4.2cm}@{}}
\toprule
\textbf{Dataset} & \textbf{Size} & \textbf{Source} & \textbf{Characteristics} \\
\midrule
SemCacheLMArena & 60{,}000 (of 63{,}796) & LM-Arena human-preference logs; GPT-4o-mini-generated, 3{,}500 classes with 1--23 paraphrases each & Conversational, long-form, high lexical diversity \\
\addlinespace
SemCacheSearchQueries & 150{,}000 & ORCAS search logs, Llama-3-8B answers, equivalence classes via union-find + LLM judge & Short queries, keyword-dense \\
\bottomrule
\end{tabular}
\caption{Datasets used.}
\end{table}

Both datasets come directly from HuggingFace \texttt{vCache/\allowbreak SemBenchmarkLmArena} and \texttt{vCache/\allowbreak SemBenchmarkSearchQueries}, using the datasets' own \textbf{precomputed embeddings} (\texttt{emb\_\allowbreak e5\_\allowbreak large\_\allowbreak v2} for LmArena, \texttt{emb\_\allowbreak gte} for SearchQueries) rather than re-encoding with sentence-transformers --- even with a matching model name, re-encoding risks failing to reproduce the paper's numbers due to version or pooling differences.

\subsection{Online evaluation protocol}

Following vCache's official harness: \textbf{no offline history split}. The cache starts empty, and the first $N$ records of the dataset are streamed once, in their original order. Groups A/C/D require no pre-calibration; Group B's threshold learning also happens entirely online, within this same single pass. This differs from this project's originally planned 20\%/80\% history/eval split, which was abandoned specifically so results would be directly comparable to already-published numbers.

\subsection{Hyperparameter grids}
\label{sec:grids}

\begin{itemize}[leftmargin=*]
  \item Group A's $\tau$: $[0.80, 0.83, 0.86, 0.89, 0.92, 0.95, 0.97, 0.98, 0.99]$ (matching vCache's \texttt{STATIC\_\allowbreak THRESHOLDS}).
  \item Group B's target error rate $\delta$: $[0.01, 0.015, 0.02, 0.025, 0.03, 0.035, 0.04, 0.05, 0.06, 0.07]$ (matching vCache's \texttt{DELTAS}).
  \item Group C/D's \tauhigh\ fixed at $0.97$ (one of Group A's own grid points, for comparability); \taulow\ swept over $[0.80, 0.83, 0.86, 0.89, 0.92, 0.95]$.
  \item Group D's verifier raw-score decision threshold is swept as a separate axis: $[-2, -1, 0, 1, 2]$ for LmArena, $[-11.5, -11.0, -10.5, -10.0, -9.0, -8.0]$ for SearchQueries --- the two datasets use different grids for the reason given in Section~\ref{sec:groupd}.
\end{itemize}

\subsection{Verifier implementations}
\label{sec:verifiers}

\begin{itemize}[leftmargin=*]
  \item \textbf{Oracle} (Group C): approves based on ground-truth equivalence class, with a fixed 70ms modeled latency standing in for the API-class LLM judge (GPT-4.1-nano) Krites itself would call --- an explicitly labeled modeling assumption, not a measurement.
  \item \textbf{Cross-Encoder} (Group D): \texttt{cross-encoder/\allowbreak ms-marco-MiniLM-L6-v2}, scoring the relevance of (query, candidate cached answer), with latency measured as actual CPU inference time.
\end{itemize}

\subsection{Metrics}

Hit rate; error rate (global, denominator over all requests); precision/recall (treating ``was it served'' as a prediction of ``was the match correct''); gray-zone-specific false-approve/false-reject rates (computed only over the $\text{verifier\_invoked}=\text{True}$ subset, kept distinct from the global confusion matrix); verifier call rate; mean verifier latency; expected added latency (call rate $\times$ mean latency). Hit rate and error rate are both reported with 95\% bootstrap confidence intervals (200 resamples).

\section{Results}

\subsection{Group A: the classic static-threshold trade-off}
\label{sec:groupa}

\begin{figure}[h]
\centering
\includegraphics[width=0.85\textwidth]{results/lmarena_pareto_full.png}
\caption{Four-group Pareto comparison on SemCacheLMArena.}
\label{fig:lmarena}
\end{figure}

\begin{figure}[h]
\centering
\includegraphics[width=0.85\textwidth]{results/search_queries_pareto_full.png}
\caption{Four-group Pareto comparison on SemCacheSearchQueries.}
\label{fig:searchqueries}
\end{figure}

\begin{table}[h]
\centering
\small
\begin{tabular}{@{}lccc@{}}
\toprule
\textbf{Dataset} & $\tau=0.80$ & $\tau=0.97$ & $\tau=0.99$ \\
\midrule
LmArena & hit 99.7\% / error \textbf{11.4\%} & hit 72.8\% / error 5.16\% & hit 36.1\% / error 2.68\% \\
SearchQueries & hit 85.6\% / error \textbf{34.3\%} & hit 28.1\% / error 4.66\% & hit 10.9\% / error 1.37\% \\
\bottomrule
\end{tabular}
\caption{Group A operating points at three representative thresholds.}
\end{table}

Figures~\ref{fig:lmarena} and~\ref{fig:searchqueries} plot all four groups together; Group A's own curve is discussed first. The error rate at a loose threshold --- 11.4\% on LmArena, a striking 34.3\% on SearchQueries --- is empirical confirmation of this paper's motivation (Section~\ref{sec:intro}): SearchQueries' short, keyword-dense queries produce far more confusable pairs (e.g.\ ``pause'' vs.\ ``cancel'') than LmArena, so the problem with plain threshold tuning is worse on this kind of data.

\subsection{Group B: trading hit rate for a formal error-rate guarantee}

\begin{table}[h]
\centering
\small
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Dataset} & \textbf{Hit-rate range} & \textbf{Error-rate range} \\
\midrule
LmArena & 0.04\%--0.26\% & 0.01\%--0.05\% \\
SearchQueries & 0.16\%--0.69\% & 0.03\%--0.14\% \\
\bottomrule
\end{tabular}
\caption{Group B operating range across the full $\delta$ grid.}
\end{table}

On both datasets, the faithfully ported vCache \texttt{VerifiedDecisionPolicy} drives hit rate to near zero while pushing error rate far below the target $\delta$. This is a structural cost of the algorithm itself: it requires the \textbf{exact same specific cache entry} to be hit as the nearest neighbor by subsequent requests at least 6 times --- all misses --- before it can ever be exploited. When paraphrases within a semantic class scatter their matches across different historical entries of that same class, any single entry rarely accumulates 6 observations, so most entries stay stuck in cold start. This matches vCache's own core selling point --- trading hit rate for a formal guarantee --- rather than indicating a porting error. Because Group B's operating point sits at a completely different hit-rate scale than A/C/D, it is not included directly in the Go/No-Go comparison below.

\textbf{[2026-08-17 update, a real discrepancy found by comparing line-by-line against the official implementation]} Section~\ref{sec:limitations}'s limitations noted that Group B's port had been checked line-by-line against vCache's source but never numerically cross-validated against a run of the official code on the same data. Cloning the official repository (\texttt{github.com/\allowbreak vcache-project/\allowbreak vCache}) and checking the \texttt{\_\allowbreak Algorithm} class's formulas line by line confirmed the design matrix, gamma clipping, $\hat{t}$, the perfect-separation variance table, the delta-method variance, the tau grid search, the EXPLORE/EXPLOIT mapping, and observing only on the miss branch all match this paper's port exactly. But one previously undocumented real difference turned up: the official \texttt{EmbeddingMetadataObj} (a cache entry's metadata object) is constructed with \textbf{two synthetic bootstrap observations pre-seeded} --- $(\text{similarity}=0.0, \text{label}=\text{incorrect})$ and $(\text{similarity}=1.0, \text{label}=\text{correct})$ --- that are never removed; this paper's port (\texttt{\_\allowbreak EmbeddingStats}) started from an empty list. This means the official implementation only needs \textbf{4 real observations} to clear the \texttt{min\_\allowbreak observations=6} cold-start gate, versus \textbf{6 real observations} for this paper's prior port --- and every subsequent fit differs systematically from the official one for lacking those two seed points.

After the fix (\texttt{cacheverifier/\allowbreak cache/\allowbreak adaptive\_\allowbreak threshold.py::\allowbreak \_\allowbreak EmbeddingStats.observations}'s default changed from an empty list to \texttt{[(0.0, 0), (1.0, 1)]}), Group B was rerun at full scale on all three datasets (the $\delta$ grid was reduced from the original 10 points to 4 representative values, $[0.01, 0.03, 0.05, 0.07]$, to control the added compute cost from fitting triggering far more often):

\begin{table}[h]
\centering
\small
\begin{tabular}{@{}lcccccc@{}}
\toprule
\textbf{Dataset} & $\bm{\delta}$ & \textbf{Hit rate before} & \textbf{Hit rate after} & \textbf{Ratio} & \textbf{Error rate after} & \textbf{Still} $\bm{\le \delta}$\textbf{?} \\
\midrule
LmArena & 0.01 & 0.04\% & 1.21\% & 29.1x & 0.02\% & yes \\
LmArena & 0.07 & 0.26\% & 3.57\% & 13.5x & 0.20\% & yes \\
Quora & 0.01 & 0.02\% & 0.12\% & 5.4x & 0.09\% & yes \\
Quora & 0.07 & 0.16\% & 0.91\% & 5.6x & 0.73\% & yes \\
SearchQueries (corrected) & 0.01 & 0.16\% & 0.80\% & 5.1x & 0.12\% & yes \\
SearchQueries (corrected) & 0.07 & 0.69\% & 3.06\% & 4.4x & 0.68\% & yes \\
\bottomrule
\end{tabular}
\caption{Group B hit rate before/after the seed-observation fix, at the two extreme tested $\delta$ values.}
\end{table}

Hit rate rose across all three datasets and all four $\delta$ values, with LmArena's gain (13.5x--29.1x) markedly larger than Quora/SearchQueries' (4.4x--5.6x) --- a plausible explanation is that LmArena's semantic classes scatter their paraphrases more (Section~\ref{sec:datasets}: 3,500 classes, 1--23 paraphrases each), so any single entry already found it harder to accumulate observations, making the ``two fewer real observations needed'' shortcut proportionally more valuable there --- though this explanation was not independently verified. \textbf{Error rate stayed below the target $\delta$ at all 12 tested points --- the formal guarantee was never violated.} This is not ``the mechanism itself has near-zero hit rate'' being overturned, but rather this paper's prior port being needlessly conservative in a way that didn't compromise correctness but did systematically leave hit rate on the table. The ranges reported in the table above should therefore be read as this paper's \emph{prior port's} hit rate, not the true ceiling achievable by the official vCache algorithm; the discussion below (the Go/No-Go criterion only compares A/C/D, never B directly) is unaffected by this correction. One unrelated, pre-existing CLI bug turned up in the same pass: \texttt{run\_\allowbreak baselines.py}'s \texttt{--target-error-rates}/\texttt{--threshold-grid} override flags only take effect on the \texttt{--dataset} code path and are silently ignored when \texttt{--config} is used (an indentation bug puts the override logic inside the \texttt{else} branch) --- worked around here by using \texttt{--dataset} with explicit \texttt{--embedder}/\texttt{--max-samples}, but the bug itself remains unfixed.

\subsection{Group C: the theoretical ceiling of the synchronous mechanism}
\label{sec:groupc}

Following the same approach used for A, C's and D's points are compared against a \textbf{linear interpolation} of Group A's own Pareto frontier (rather than only against A's tested discrete grid points, which --- being coarser --- would understate A's true achievable performance):

\begin{table}[h]
\centering
\small
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Dataset} & \textbf{C's max.\ lead over A's interpolated frontier} & \textbf{All grid points lead?} \\
\midrule
LmArena & \textbf{+20.6 pp} (hit rate, matched error rate) & Yes (6/6) \\
SearchQueries & \textbf{+27.9 pp} & Yes (6/6) \\
\bottomrule
\end{tabular}
\caption{Group C (oracle) versus Group A's interpolated frontier.}
\end{table}

Under the assumption of a perfect verifier, the synchronous gating mechanism has substantial theoretical headroom on both datasets, and more so on SearchQueries --- because that dataset has more candidates that are ``high similarity but wrong,'' giving a verifier more room to recover. This shows the mechanism itself is not falsified: the question is not whether synchronous verification is worth doing in principle, but which verifier to use.

\subsection{Group D: a real verifier cashes in only a sliver of that ceiling}
\label{sec:groupd}

The same interpolation-based comparison, applied to Group D (30 grid points for LmArena, 36 for SearchQueries):

\begin{table}[h]
\centering
\small
\begin{tabular}{@{}lccccc@{}}
\toprule
\textbf{Dataset} & \textbf{Beats A} & \textbf{Tied} & \textbf{Loses} & \textbf{Best net lead} \\
\midrule
LmArena & 14/30 & 5/30 & 11/30 & \textbf{+1.9 pp} ($\taulow{=}0.89$, threshold$\approx 0$, error$\approx 5.99\%$) \\
SearchQueries (corrected, see erratum) & 1/36 & 12/36 & 23/36 & \textbf{+0.30 pp} ($\taulow{=}0.95$, threshold$=9$, error$\approx 5.85\%$) \\
\bottomrule
\end{tabular}
\caption{Group D (cross-encoder) versus Group A's interpolated frontier.}
\end{table}

LmArena's best point is statistically significant: hit rate 89.4\% (95\% CI [89.2\%, 89.7\%]) versus Group A's nearest comparable point ($\tau=0.95$, error rate 5.96\%): hit rate 87.0\% (95\% CI [86.7\%, 87.3\%]) --- the two confidence intervals do not overlap. But this lead is \textbf{less than one-tenth} of the oracle ceiling (+20.6pp), and more than a third of the 30 tested points (11) actually did \emph{worse} than simply tuning the static threshold --- meaning this real verifier's benefit depends heavily on the specific $(\taulow, \text{threshold})$ chosen, not a robust region one can set and forget.

The SearchQueries result (corrected, see erratum above) is \textbf{net harmful}, not ``effectively zero'': only 1 of 36 grid points genuinely beats Group A's interpolated frontier, 12 tie, and \textbf{23 lose} to the static-threshold frontier; the best point's lead is only +0.30 percentage points. The gray zone's false-approve and false-reject rates fluctuate in the 21\%--50\% and 33\%--74\% ranges respectively --- not the ``no signal'' pattern of near-chance rates, but a model that is genuinely using the candidate answer's real content to make a judgment, and is frequently wrong, badly enough that the net effect is worse than no verifier at all. This is also why this paper used different verifier-score threshold grids for the two datasets (Section~\ref{sec:grids}): the grid calibrated on LmArena ($[-2,-1,0,1,2]$) applied to SearchQueries revealed that dataset's real score distribution (corrected) is $[-11.5, 11.6]$, mean $\approx 5.9$ --- shaped quite differently from LmArena's, requiring a grid chosen from SearchQueries' own percentiles ($[1,3,5,7,9,11]$) rather than reused wholesale. This remains a direct empirical demonstration that verifier hyperparameters do not transfer across domains; only the specific numbers have changed with the erratum.

\textbf{[2026-08-15, honest-calibration ablation, directly testing the methodological issue flagged by this section's threshold grid]} The verdict above uses this paper's original approach: sweep a hand-picked, coarse threshold grid, then report the best point in that grid --- exactly the methodological weakness Section~\ref{sec:limitations}'s limitations already flag (``the threshold grid is calibrated post hoc, not fixed in advance on an independent held-out set''). Since the commercial product \texttt{cacheverifier-service} already implements an honest calibration procedure (Youden's J statistic, \texttt{verifier\_\allowbreak core/\allowbreak finetune.py::\allowbreak select\_\allowbreak threshold}), that same procedure was run against this paper's three datasets: at each $\taulow$, the gray-zone stream is split chronologically into a calibration half and a test half; the threshold is picked via Youden's J on the calibration half only, then hit rate/error rate is measured honestly on the test half, which never influenced the threshold, and compared against Group A's interpolated frontier:

\begin{table}[h]
\centering
\small
\begin{tabular}{@{}lp{0.5\textwidth}@{}}
\toprule
\textbf{Dataset} & \textbf{Honest calibration vs.\ Group A frontier (6 tested points)} \\
\midrule
SearchQueries (corrected, full 150,000 records) & \textbf{6/6 beat}, +0.78pp to +3.67pp \\
LmArena (full 60,000 records) & 5/6 beat by a small margin (+0.17pp to +3.04pp), 1/6 ($\taulow{=}0.95$) lost by a small margin (-0.56pp) \\
Quora (full 60,000 records) & 6/6 lost by a small margin, -0.12pp to -1.77pp \\
\bottomrule
\end{tabular}
\caption{Honest-calibration ablation vs.\ Group A, all three datasets.}
\end{table}

\textbf{The SearchQueries verdict reverses.} Under an honestly-calibrated threshold, the same untuned, off-the-shelf verifier beats the static-threshold frontier at all 6 tested points, rather than the ``23 of 36 grid points lost to the static threshold'' reported above. This suggests a substantial share of the original ``net harmful'' verdict was itself an artifact of the grid search --- the hand-picked raw score values ($[1,3,5,7,9,11]$) most likely never covered the operating point Youden's J is able to find, rather than reflecting a genuine absence of usable discriminative power on this dataset. LmArena's honest-calibration result is directionally consistent with, and of similar magnitude to, the originally reported result (+1.9pp at the best point), without the qualitative reversal seen on SearchQueries (an initial run was inflated by a non-random subsample and was corrected with a full-dataset rerun; see \texttt{RESEARCH\_\allowbreak PROPOSAL.md} \S10 direction 1 for the full account). Quora is the only dataset where honest calibration makes things worse --- diagnosis shows this is not a calibration-method problem but the dataset's own structural difficulty: the verifier's overall score AUC is about 0.61 (real signal exists), but the score distributions of correct and incorrect candidates overlap substantially, traceable to the QQP paraphrase-detection task itself mixing annotation noise with deliberately-constructed ``surface-similar, semantically-different'' hard negatives (e.g.\ ``George H.W. Bush'' vs.\ ``George W. Bush'' --- two different presidents, nearly identical text) that cap how separable any single threshold can make this dataset; see \texttt{RESEARCH\_\allowbreak PROPOSAL.md} \S10 direction 1 for details.

Taken together, the three datasets show that honest calibration is \textbf{not} a free correction that reliably helps --- removing the known bias of post-hoc grid search can make results better or worse depending on how genuinely separable the verifier's off-the-shelf scores are on that domain. For SearchQueries, removing the grid-search bias reveals a verifier that had usable signal all along, which the original evaluation simply failed to find; for Quora, removing the bias reveals the verifier's own signal ceiling. \textbf{This result initially covered only the untuned Group D.} Section~\ref{sec:groupe}'s fine-tuned Group E numbers (53/54 wins on SearchQueries) still used the same post-hoc grid-search method and had not yet been re-examined under honest calibration at the time --- \textbf{[2026-08-16, Group E tested, no longer an open question]} the identical procedure (chronological calibration/test split per $\taulow$, Youden's J picked on the calibration half only) was rerun against each dataset's own Group E fine-tuned verifier (script \texttt{scripts/\allowbreak threshold\_\allowbreak calibration\_\allowbreak ablation\_\allowbreak groupE.py}; raw logs/JSON in \texttt{results/\allowbreak logs/\allowbreak } and \texttt{results/\allowbreak *\_\allowbreak groupE\_\allowbreak honest\_\allowbreak calibration.json}):

\begin{table}[h]
\centering
\small
\begin{tabular}{@{}lp{0.32\textwidth}p{0.1\textwidth}p{0.32\textwidth}@{}}
\toprule
\textbf{Dataset} & \textbf{Honest calibration vs.\ Group A frontier (6 tested points)} & \textbf{Best net lead} & \textbf{Original grid search (reported above)} \\
\midrule
LmArena & \textbf{6/6 beat} & +5.66pp & (no exact count reported; text says ``nearly every tested point'') \\
SearchQueries (corrected) & \textbf{6/6 beat} & +7.74pp & 53/54 beat, 1 tie, 0 lost, +4.50pp \\
Quora & 3/6 beat, 3/6 tie, \textbf{0/6 lost} & +0.43pp & 19/48 beat, 29/48 tie, 0/48 lost, +2.03pp \\
\bottomrule
\end{tabular}
\caption{Honest-calibration ablation, Group E (fine-tuned verifier), all three datasets.}
\end{table}

The verdict: \textbf{Group E's core qualitative claim --- the fine-tuned verifier is never a net loss against the static-threshold frontier --- holds fully under honest calibration}, zero losses on all three datasets. But the magnitude splits the same way Group D did: LmArena and SearchQueries's honest-calibration lead does not shrink but actually grows (+5.66pp, +7.74pp, both exceeding the originally reported grid-search numbers), while Quora narrows substantially (+2.03pp $\to$ +0.43pp, the same direction as Group D's honest-calibration narrowing, traceable to the same QQP score-separability ceiling diagnosed earlier in this section rather than a calibration-method artifact). One methodological caveat: honest calibration yields exactly one test point per $\taulow$ (calibration/test split in half each), while the original grid search swept a whole threshold grid per $\taulow$, so ``6/6'' and ``53/54'' have non-comparable denominators --- what is comparable is the \emph{direction} (win/tie/loss proportions) and the \emph{magnitude of the lead when winning}, not a direct ratio-to-ratio comparison. A second caveat concerns data provenance: the LmArena and SearchQueries (corrected) checkpoints used here are the exact same ones that produced this paper's original Section~\ref{sec:groupe} numbers (hosted at Hugging Face \texttt{ChengyouXin/\allowbreak cacheverifier-lmarena}, and the locally-verified \texttt{finetuned\_\allowbreak verifier\_\allowbreak model\_\allowbreak searchqueries\_\allowbreak corrected}, respectively); Quora's original 2026 checkpoint was never persisted anywhere (not committed to the repo, never uploaded to Hugging Face, and long since cleaned off disk), so it was retrained here from the exact original train/test split (\texttt{results/\allowbreak finetune\_\allowbreak verifier\_\allowbreak experiment\_\allowbreak quora.examples.json}, 11,928 train / 5,112 test, matching the original record counts exactly) --- the retrained baseline AUC reproduces the original report bit-for-bit (0.6309, deterministic given a fixed untuned model and test set) and the retrained fine-tuned AUC is close (0.7407 vs.\ the original 0.7393, $\Delta$=0.0014), reasonably a faithful replication of the same experiment rather than an incomparable new one, but it is not literally the same weights, and that distinction is recorded here rather than glossed over.

\subsection{The latency cost of going synchronous}
\label{sec:latency}

At its highest-call-rate operating point (SearchQueries, $\taulow=0.80$: 57.5\% of requests enter the gray zone), the oracle (70ms modeled latency) adds an average of +40.3ms per request; the real cross-encoder's measured average latency at a comparable call rate is about 66--68ms (measured when the SearchQueries run executed alone; the LmArena run's measured latency was inflated to about 288ms because it competed for CPU with three other parallel jobs, and should not be treated as a reliable estimate of the model's own latency). Both figures quantify something Krites' own paper only judged qualitatively (``will increase latency'') without measuring.

\textbf{[2026-08-17 update, a real measurement of the oracle's latency]} The 70ms figure above is a modeling assumption explicitly flagged in Section~\ref{sec:verifiers} (``representing the order of magnitude of the API-class LLM judge Krites would call, GPT-4.1-nano'') that was never empirically validated --- also listed as a gap in Section~\ref{sec:limitations}'s limitations. \texttt{scripts/\allowbreak measure\_\allowbreak oracle\_\allowbreak judge\_\allowbreak latency.py} ran a real measurement: with no access to a GPT-4.1-nano API, DeepSeek (\texttt{deepseek-chat}) was used as a same-class stand-in (small, API-hosted, not locally deployed) --- the same substitution principle already used and documented for the SearchQueries erratum (``no Llama-3-8B access, substituted DeepSeek''). 50 (query, candidate cached answer) pairs were drawn from each of LmArena/Quora/SearchQueries (candidates from neighboring records, mirroring what a real gray-zone judge call sees), sent as an equivalence-judgment prompt structurally similar to Krites' own asynchronous judge (``is the candidate still a correct answer to the query --- YES/NO only''), and called sequentially (not concurrently) to record wall-clock latency, 150 calls total:

\begin{table}[h]
\centering
\small
\begin{tabular}{@{}lc@{}}
\toprule
\textbf{Statistic} & \textbf{Value} \\
\midrule
Paper's modeling assumption & 70ms \\
Measured mean & \textbf{1687.8ms} \\
Measured median & 1651.1ms \\
p95 & 2064.9ms \\
min / max & 1347.9ms / 2765.2ms \\
\bottomrule
\end{tabular}
\caption{Real API-latency measurement (DeepSeek, $n=150$) vs.\ the paper's modeled Oracle latency.}
\end{table}

The measured mean is \textbf{about 24x} the modeling assumption. 70ms looks more like the order of magnitude of network round-trip/queueing overhead alone, without accounting for the time a real LLM actually needs to generate tokens. Substituting this measured mean for the constant 70ms at the operating points reported earlier in this section (this only recomputes the ``expected added latency = call rate $\times$ mean latency'' statistic --- it changes no hit-rate/error-rate number, since the oracle's correctness judgment is independent of the latency value):

\begin{table}[h]
\centering
\small
\begin{tabular}{@{}lccc@{}}
\toprule
\textbf{Dataset (highest-call-rate point)} & \textbf{Gray-zone call rate} & \textbf{Expected added latency (modeled)} & \textbf{Expected added latency (real)} \\
\midrule
SearchQueries ($\taulow=0.80$) & 57.5\% & 40.3ms & \textbf{970.5ms} \\
Quora ($\taulow=0.80$) & 28.4\% & 19.9ms & \textbf{479.3ms} \\
LmArena ($\taulow=0.80$) & 26.8\% & 18.8ms & \textbf{452.3ms} \\
\bottomrule
\end{tabular}
\caption{Expected added latency: modeled 70ms vs.\ the real measured mean, at each dataset's highest-call-rate point.}
\end{table}

This moves the latency cost from ``tens of milliseconds, probably negligible'' to ``hundreds of milliseconds to nearly a second, worth taking seriously'' --- directly affecting how much weight Section~\ref{sec:golo}'s claim that ``synchronicity itself doesn't erode benefit, at the cost of a quantifiable latency'' should carry; see the update to Section~\ref{sec:golo}. This measurement has its own limits worth stating: DeepSeek and GPT-4.1-nano are different models on different hosting infrastructure, and measured latency will vary with the specific API provider, region, and concurrency strategy --- the order of magnitude reported here, not the precise millisecond figure, is this measurement's main contribution, and the original 70ms assumption is very likely an underestimate by roughly this order of magnitude regardless of which API provider is substituted in.

\subsection{Group E: a domain-fine-tuned verifier closes the gap across all three datasets}
\label{sec:groupe}

Section~\ref{sec:groupd}'s diagnosis was that the off-the-shelf cross-encoder's benefit is fragile on LmArena and entirely absent on SearchQueries, because the model was never calibrated to either dataset's own gray zone. A direct test: fine-tune the same base model (\texttt{cross-encoder/\allowbreak ms-marco-MiniLM-L6-v2}) separately on each dataset's own gray-zone examples --- pairs of $(\text{query}, \text{candidate cached answer})$ with a binary ``would this hit have been correct'' label --- and see how much of Group D's shortfall against Group C's ceiling closes on each.

\textbf{Setup.} Each dataset's gray zone ($\taulow=0.80, \tauhigh=0.97$) is split \textbf{by stream position, not randomly} --- the first 70\% for fine-tuning, the last 30\% held out for evaluation --- mirroring a real ``calibrate on past traffic, deploy on future traffic'' deployment rather than i.i.d.\ cross-validation. LmArena yields 16,102 labeled examples (11,271 train / 4,831 test); SearchQueries, being shorter-text and higher-call-rate (Section~\ref{sec:latency}), yields far more: 86,280 (60,398 train / 25,886 test). The base model is fine-tuned for one epoch with sentence-transformers' \texttt{CrossEncoderTrainer} and a binary cross-entropy loss, on CPU --- LmArena's longer conversational text took 43 minutes to fine-tune; SearchQueries' short keyword queries, despite 5.4$\times$ more training examples, took under 6 minutes, since per-example compute scales with sequence length.

\textbf{Held-out AUC.}

\begin{table}[h]
\centering
\small
\begin{tabular}{@{}lccc@{}}
\toprule
\textbf{Dataset} & \textbf{Untuned AUC} & \textbf{Fine-tuned AUC} & $\boldsymbol{\Delta}$ \\
\midrule
LmArena & 0.7212 & 0.8789 & \textbf{+0.158} \\
SearchQueries & 0.5984 (corrected; original 0.4881 was a placeholder-data defect, see erratum above) & 0.7120 & \textbf{+0.114} \\
\bottomrule
\end{tabular}
\caption{Held-out ROC-AUC before and after fine-tuning, both datasets.}
\end{table}

LmArena's untuned verifier already had some signal (0.72); fine-tuning sharpens it. SearchQueries' untuned verifier (corrected, see erratum above) has AUC 0.5984 --- also real signal, but as Section~\ref{sec:groupd} shows, that signal is frequently used to make the wrong call, so the net effect is worse than no verifier at all. Fine-tuning does not fully repair the absolute gap to LmArena (0.7120 vs.\ 0.88), but it moves the model from \emph{has signal, frequently misuses it} to \emph{has real signal and mostly uses it correctly}.

\textbf{Effect on the Pareto frontier.} Plugging each fine-tuned model back into the same synchronous gray-zone gating mechanism (\texttt{SynchronousVerifiedPolicy}, unchanged) and replaying it over the full stream as Group E, then comparing against Group A's interpolated frontier --- the same test Section~\ref{sec:groupd} applied to Group D:

\begin{table}[h]
\centering
\small
\begin{tabular}{@{}lp{0.34\textwidth}p{0.4\textwidth}@{}}
\toprule
\textbf{Dataset} & \textbf{Group D vs.\ A (Sec.~\ref{sec:groupd})} & \textbf{Group E vs.\ A} \\
\midrule
LmArena & 14/30 beat, 11/30 lost, best +1.9pp & hit rate \textbf{+2.8 to +8.3pp} vs.\ D at matched error; frontier strictly wider \\
SearchQueries (corrected, see erratum) & \textbf{1/36 beat, 12/36 tied, 23/36 lost}, best +0.30pp & \textbf{53/54 beat, 1/54 tied, 0/54 lost}, best net lead \textbf{+4.50pp} \\
\bottomrule
\end{tabular}
\caption{Group D versus Group E, each against Group A's interpolated frontier.}
\end{table}

\begin{figure}[h]
\centering
\includegraphics[width=0.85\textwidth]{results/lmarena_pareto_full_with_finetune.png}
\caption{Group E (fine-tuned verifier) added to the four-group Pareto comparison on SemCacheLMArena.}
\label{fig:lmarena-groupe}
\end{figure}

\begin{figure}[h]
\centering
\includegraphics[width=0.85\textwidth]{results/search_queries_pareto_full_with_finetune.png}
\caption{Group E (fine-tuned verifier) added to the four-group Pareto comparison on SemCacheSearchQueries.}
\label{fig:searchqueries-groupe}
\end{figure}

Figures~\ref{fig:lmarena-groupe} and~\ref{fig:searchqueries-groupe} plot Group E alongside the original four groups on each dataset. The two datasets tell different but complementary stories. On LmArena, fine-tuning turns a fragile, partly-losing verifier into one that strictly dominates the static-threshold frontier and widens the achievable region in both directions (error rate down to $\approx$0.052, hit rate up to $\approx$0.96). On SearchQueries (corrected, see erratum above), fine-tuning turns a verifier that was \textbf{net harmful} --- 23 of 36 operating points lost to the static threshold, not ``indistinguishable from chance'' --- into one that beats the static baseline at 53 of 54 tested operating points, ties at 1, and loses at zero (best net lead +4.50pp), even though its absolute AUC (0.7120) remains well below LmArena's (0.88). This qualitative swing --- from ``net harmful'' to ``wins almost everywhere'' --- is more extreme than the ``null result to real improvement'' swing originally reported, and changes the picture on the dataset this paper's Go/No-Go verdict rested most heavily on even more than before.

This is the clearest evidence in this paper that Section~\ref{sec:golo}'s diagnosis (``the variable that matters is how well the verifier matches the data domain'') is actionable, not just descriptive: the same base architecture, given in-domain labeled examples that the online system already produces for free as a byproduct of running Group C/D (every gray-zone request's eventual correctness is knowable from the same ground-truth equivalence labels used to score the experiment), converts a fragile-or-nonexistent Group D result into a robust improvement over the static-threshold frontier on both datasets tested.

\textbf{A third, independently-sourced check.} To rule out that this pattern is specific to vCache's own two benchmarks, the same recipe was repeated on Quora Question Pairs (GLUE's mirror) --- 60,000 short, real user-submitted questions with human-annotated duplicate labels, a domain neither vCache nor this paper's earlier experiments touched. Quora has no LLM-generated answers or native equivalence-class column, so --- unlike LmArena/SearchQueries --- equivalence classes were reconstructed via union-find over duplicate-labeled pairs, and \texttt{answer = query} (a past matched question's own text stands in for ``the cached response''); full construction details are in \texttt{scripts/\allowbreak convert\_\allowbreak quora\_\allowbreak dataset.py}. Quora's oracle ceiling is markedly lower than the other two datasets (+11.3pp over Group A's frontier, vs.\ +20.6pp on LmArena and +27.9pp on SearchQueries), because only about a third of Quora's questions belong to a multi-member duplicate class --- most questions here are simply unique, capping how much any verifier could ever help.

Group D's off-the-shelf verifier again adds essentially nothing (0/54 beat, 50/54 tied, 4/54 \emph{lost} to the static-threshold frontier, best net lead +0.11pp) --- the untuned verifier's held-out AUC (0.6309) is well above chance, and now sits in the same tier as SearchQueries' corrected 0.5984 (both ``has some discriminative power, not enough to reliably beat an already-strong similarity baseline,'' rather than the originally-reported contrast of ``SearchQueries near-chance, Quora not coincidental'' --- SearchQueries' earlier 0.4881 was itself a data defect and should not be read as a real discriminative-power baseline; see erratum above and Section~\ref{sec:groupd}). Fine-tuning (11,928 in-domain examples) raises AUC to 0.7393 (+0.108) and, plugged back in as Group E, turns that null result into a modest but unambiguous win: 19/48 grid points beat Group A's frontier, 29/48 tied, \textbf{0/48 lost} (best net lead +2.03pp; +0.76pp mean improvement over Group D at matched error rate). The margin is the smallest of the three datasets --- proportional to Quora's smaller headroom --- but the qualitative pattern now holds across corrected SearchQueries and original LmArena alike: across all three independently-sourced datasets tested, fine-tuning \textbf{never produced a worse result than the untuned verifier}, and converted every fragile, null, or (as with SearchQueries' corrected Group D) net-harmful result into at least a modest, loss-free improvement over the static-threshold baseline.

\begin{figure}[h]
\centering
\includegraphics[width=0.85\textwidth]{results/quora_pareto_full_with_finetune.png}
\caption{Group E (fine-tuned verifier) added to the four-group Pareto comparison on Quora Question Pairs --- a third, independently-sourced dataset.}
\label{fig:quora-groupe}
\end{figure}

\subsection{Practical deployment considerations: label noise, cold start, and drift}
\label{sec:deployment}

Section~\ref{sec:groupe} fine-tunes on clean, oracle-quality labels harvested from Groups C/D's own ground truth. Three further questions determine whether this translates into an operable production capability rather than a lab result: (1) how much label noise --- from a realistic feedback signal such as user thumbs-up/down, rather than an oracle --- can the fine-tuning tolerate before it stops helping, or starts actively hurting; (2) how many labeled examples does a newly onboarded deployment need to accumulate before turning fine-tuning on is worth the risk; (3) how quickly does a fine-tuned verifier's advantage decay as production traffic drifts away from its training window, which determines how often it must be refreshed. Each was tested on all three datasets, reusing the same fine-tuning recipe as Section~\ref{sec:groupe}, on an NVIDIA T4 GPU.

\textbf{Label noise robustness.} Training labels in a 4,000-example fixed subset were randomly flipped at five noise levels (0/5/10/20/30/40\%) before fine-tuning (one epoch); test labels were never touched, so held-out AUC always measures against true ground truth.

\begin{table}[h]
\centering
\small
\begin{tabular}{@{}lccc@{}}
\toprule
\textbf{Noise} & \textbf{LmArena $\Delta$} & \textbf{SearchQueries $\Delta$ (corrected)} & \textbf{Quora $\Delta$} \\
\midrule
0\% & +0.107 & +0.017 & +0.083 \\
10\% & +0.083 & +0.016 & +0.074 \\
20\% & +0.051 & +0.015 & +0.044 \\
30\% & +0.028 & +0.003 & +0.006 \\
40\% & \textbf{-0.101} & \textbf{-0.011} & \textbf{-0.037} \\
\bottomrule
\end{tabular}
\caption{AUC delta vs.\ train label noise, all three datasets.}
\end{table}

\textbf{[2026-08-09, rerun on corrected data, replacing the placeholder-data result]} All three datasets show the same shape: a gentle, roughly linear decline through 30\% noise, then a rapid approach to, or across, zero. The interpolated zero-crossing is \textbf{32.2\% for LmArena, $\approx$32.1\% for SearchQueries (corrected), and 31.4\% for Quora} --- all three now nearly identical. This differs from the original finding: on the placeholder data, SearchQueries looked like the noise-tolerant outlier, never crossing zero even at 40\%; corrected, that apparent exception disappears and it lines up with the other two. Below roughly 30\% label-noise, fine-tuning reliably helps; above it, \textbf{all three} datasets actively learn the wrong association and perform worse than not fine-tuning at all --- SearchQueries is no longer the exception.

\textbf{Cold-start data requirements.} The same recipe was re-run with increasingly larger prefixes of each dataset's own (temporally ordered) training split, from 50 examples up to the full available set (three epochs, to give small subsets a fair chance to converge).

\begin{table}[h]
\centering
\small
\begin{tabular}{@{}lccc@{}}
\toprule
\textbf{Train size} & \textbf{LmArena $\Delta$} & \textbf{SearchQueries $\Delta$ (corrected)} & \textbf{Quora $\Delta$} \\
\midrule
50 & +0.001 & \textbf{-0.013} & +0.012 \\
200 & +0.030 & \textbf{-0.017} & +0.018 \\
500 & +0.050 & \textbf{-0.008} & +0.002 \\
1{,}000 & +0.080 & +0.004 & +0.049 \\
4{,}000 & +0.136 & +0.029 & +0.096 \\
Full (11.3k--60.4k) & +0.188 & +0.143 & +0.118 \\
\bottomrule
\end{tabular}
\caption{AUC delta vs.\ cold-start training set size, all three datasets.}
\end{table}

\textbf{[2026-08-09, rerun on corrected data, replacing the placeholder-data result]} All three curves rise smoothly on a log scale with \textbf{no saturation} even at full dataset size --- more in-domain data keeps helping. But the small-sample region is not uniformly benign: SearchQueries (corrected untuned baseline AUC 0.5984, no longer near chance) shows a \textbf{wider} negative-return zone than originally reported --- not ``below roughly 200 examples,'' but persisting through 500 and only turning positive at 1{,}000, roughly a 5x higher threshold than the original (placeholder-data) estimate. This risk is not visible on LmArena's stronger baseline at the same sizes. The direction of the correction runs opposite to the naive guess: the placeholder-data ``baseline'' was uninformative noise with almost no real signal left to disrupt; the corrected baseline already carries real discriminative power (0.5984), which small-sample fine-tuning more easily scrambles --- hence a \emph{wider}, not narrower, negative-return zone.

\textbf{Temporal drift and re-tuning cadence.} Each dataset's full gray-zone stream (train+test recombined, resorted by original position) was split into 8 equal-sized sequential chunks. A model fine-tuned on chunk $i$ (the ``anchor'') was frozen and evaluated on every chunk $j \geq i$; delta (vs.\ that chunk's own untuned baseline) was averaged across all anchors at each temporal distance $j-i$.

\begin{table}[h]
\centering
\small
\begin{tabular}{@{}lccc@{}}
\toprule
\textbf{Distance} & \textbf{LmArena $\Delta$} & \textbf{SearchQueries $\Delta$} & \textbf{Quora $\Delta$} \\
\midrule
0 (same chunk) & +0.193 & +0.308 & +0.298 \\
1 & +0.111 & +0.082 & +0.089 \\
4 & +0.124 & +0.075 & +0.085 \\
7 (farthest) & +0.129 & +0.056 & +0.069 \\
\bottomrule
\end{tabular}
\caption{Mean AUC delta vs.\ temporal distance from the fine-tuning anchor, all three datasets.}
\end{table}

\textbf{[2026-08-09, rerun on corrected data --- conclusion differs from the original]} The original analysis concluded that none of the three datasets showed continued decay beyond the first temporal step, with SearchQueries as the strongest evidence for this because its untuned baseline AUC appeared nearly flat over time (0.48--0.51). That specific supporting observation no longer holds: corrected, SearchQueries' untuned baseline AUC across the 8 chunks is $[0.643, 0.631, 0.628, 0.614, 0.612, 0.612, 0.602, 0.594]$ --- monotonically declining, about 7.7\% peak-to-trough, not flat. The ``flat baseline'' observed on placeholder data now looks like another symptom of that defect (a constant, information-free placeholder text is naturally more likely to produce a stable-looking but meaningless score distribution in any time window) rather than evidence of ``no real difficulty drift.'' Correspondingly, the delta itself does not fully plateau past distance 1 either --- it declines gradually from +0.082 to +0.056 (roughly a 32\% relative decline) --- closer to the (much smaller-magnitude) continued-decay pattern comcastcares shows in Section~\ref{sec:twitter} than to the ``zero decay'' originally reported. SearchQueries moves from ``the cleanest example of zero decay'' to a third data point showing mild-but-real decay --- far short of comcastcares' catastrophic decay, but directionally closer to it than to ``no decay at all.'' LmArena's own baseline does drift substantially by chunk (0.93 $\to$ 0.71, driven by a large positive-rate shift), yet the fine-tuned model's \emph{relative} edge over that same drifting baseline still does not decay --- the fine-tuned verifier's discriminative power generalizes across time even when the underlying task difficulty changes; this LmArena finding is unaffected by the erratum.

\textbf{Consolidated implication (updated 2026-08-09).} Cold start and drift are both \textbf{more pronounced than originally reported} for SearchQueries specifically --- not ``smaller than expected, generalizing cleanly across three datasets.'' The negative-return cold-start zone widened roughly 5x ($\approx$200 to $\approx$1{,}000 examples), and drift went from ``zero decay'' to ``mild but real decay.'' LmArena and Quora are unaffected by this correction (their \texttt{answer} fields never depended on the missing column). The one risk requiring careful engineering is feedback-signal quality: a noise-detection gate that estimates a deployment's actual label-noise rate (e.g.\ against a small manually-audited sample) before enabling fine-tuning is not optional, and the minimum-data floor should be set more conservatively for weak-baseline (harder) domains than for strong-baseline ones --- this correction is itself a concrete instance of that advice.

\subsection{A fourth check: real production customer-support traffic, and a genuine counter-example}
\label{sec:twitter}

Sections~\ref{sec:groupe}--\ref{sec:deployment} validate the fine-tuning recipe on three datasets, but all three are public research benchmarks, not real business traffic. As a further check, the same recipe was applied to two brands' streams from the Kaggle ``Customer Support on Twitter'' dataset (Axelbrooke, 2017; CC0) --- 3M+ real support tweets from dozens of companies' official accounts, with genuine per-tweet timestamps and full thread structure (\texttt{tweet\_\allowbreak id}, \texttt{in\_\allowbreak response\_\allowbreak to\_\allowbreak tweet\_\allowbreak id}, \texttt{response\_\allowbreak tweet\_\allowbreak id}, \texttt{created\_\allowbreak at}). Two brands were selected to match e-commerce and telecom use cases specifically: \textbf{AmazonHelp} (60,000 query-answer pairs, 2015-06 to 2017-10) and \textbf{comcastcares} (30,180 pairs, 2014-07 to 2017-12 --- a 3.5-year span, the longest of any dataset in this paper).

\textbf{Construction.} Neither brand's stream has an LLM-generated answer or a native equivalence-class label. A company's reply tweet is paired with the customer tweet it answers (query, answer); since there is no ground truth for ``these two customer questions are the same issue,'' equivalence classes are reconstructed from a real behavioral signal --- whether the company sent near-duplicate replies, clustered by cosine similarity of sentence embeddings (\texttt{sentence-transformers/\allowbreak all-MiniLM-L6-v2}, threshold 0.92, via sentence-transformers' \texttt{community\_\allowbreak detection} utility) rather than requiring exact string equality, since canned replies are frequently personalized (customer name, order number). An earlier exact-string-match version of this construction under-counted true duplicates so severely it produced a false-accept rate near 100\% under Group A --- a labeling artifact corrected before drawing any conclusions; full details are in \texttt{scripts/\allowbreak convert\_\allowbreak twitter\_\allowbreak cs\_\allowbreak dataset.py}.

\textbf{Group E: a genuine split result.} Repeating Section~\ref{sec:groupe}'s fine-tuning recipe on each brand's own gray zone:

\begin{table}[h]
\centering
\small
\begin{tabular}{@{}lcccc@{}}
\toprule
\textbf{Brand} & \textbf{Train/test pos.\ rate} & \textbf{Untuned AUC} & \textbf{Fine-tuned AUC} & \textbf{$\Delta$} \\
\midrule
AmazonHelp & 1.47\% / 1.02\% (consistent) & 0.666 & 0.722 & \textbf{+0.056} \\
comcastcares & \textbf{1.0\% / 5.2\% (5$\times$ mismatch)} & 0.864 & 0.678 & \textbf{-0.186} \\
\bottomrule
\end{tabular}
\caption{Group E fine-tuning results on real customer-support traffic (gray-zone train/test: AmazonHelp 4{,}361/1{,}870; comcastcares 2{,}579/1{,}106).}
\end{table}

AmazonHelp replicates Section~\ref{sec:groupe}'s pattern exactly. comcastcares does not: fine-tuning makes the verifier \emph{worse} than not fine-tuning at all, despite a training-set size (2,579) above the floor Section~\ref{sec:deployment} identified as generally safe. The diagnosis is a training/test \textbf{base-rate mismatch}: the fraction of gray-zone queries that are genuinely reusable is 1.0\% in the (earlier) training window but 5.2\% in the (later) held-out window --- a decision boundary learned under one prior shifts to a different regime, and no amount of label cleanliness fixes a mismatched prior.

\textbf{The three deployment ablations diverge in the same direction.} Repeating Section~\ref{sec:deployment}'s exact protocol on both brands:

\emph{Label noise} --- comcastcares is negative at \textbf{every} noise level tested, including 0\% (clean labels): $\Delta = -0.186$ at 0\%, ranging -0.15 to -0.23 through 40\%. This rules out noise as the driver; the base-rate mismatch alone is sufficient to make fine-tuning harmful regardless of label quality. AmazonHelp crosses to negative earlier ($\sim$5--10\%) than LmArena/Quora's $\sim$30\% zero-crossing, plausibly reflecting its smaller absolute training set (capped at 4,000) relative to the original three datasets.

\emph{Cold start} --- AmazonHelp continues Section~\ref{sec:deployment}'s ``more data helps, no saturation'' pattern ($\Delta = +0.064$ at 50 examples to +0.097 at 4,000). comcastcares does not: $\Delta$ degrades from -0.004 (50 examples) to a catastrophic \textbf{-0.569 at 2{,}000 examples (AUC 0.295, worse than chance)}, partially recovering to -0.219 at the full 2,579. This is the first observation, across four datasets, of more training data making a fine-tuned verifier \emph{worse} rather than merely failing to help --- consistent with a model trained on an increasingly larger sample of the low-base-rate window becoming a more confidently miscalibrated classifier for the high-base-rate test window.

\emph{Drift} --- AmazonHelp again matches Section~\ref{sec:deployment} (decay to a plateau, no continued degradation through distance 5, matching the memorization-then-flat pattern). comcastcares instead shows genuine \textbf{continued decay from distance 3 onward} ($\Delta = -0.19$ at distance 3 to -0.24 at distance 5, with 5 and 4 anchor-pairs respectively --- not a single noisy point), the first dataset in this paper to violate Section~\ref{sec:deployment}'s ``no continued decay'' finding.

\textbf{A single root cause, observed three ways.} comcastcares' untuned baseline AUC, computed per chunk across its 8-chunk drift split, is $[0.522, 0.39, 0.493, 0.408, 0.551, 0.569, 0.808, 0.877]$ --- non-monotonic and markedly higher in the later chunks, indicating the underlying stream's difficulty and positive-class rate are not stationary over its 3.5-year span (plausibly reflecting evolving support-template standardization). This single instability explains all three otherwise-separate anomalies: clean labels still fail (noise), more data trains a more confidently wrong model (cold start), and the verifier's edge keeps eroding rather than plateauing (drift). AmazonHelp, with a shorter recorded span ($\sim$2.4 years) and a stable positive rate, shows none of these failure modes and instead replicates Sections~\ref{sec:groupe}--\ref{sec:deployment} exactly.

\textbf{Revising Section~\ref{sec:deployment}'s conclusion.} The three-dataset finding that ``cold start and drift are smaller risks than expected, with no need for high-frequency re-tuning'' was drawn from benchmarks with comparatively short or synthetic temporal spans. comcastcares --- the only dataset in this paper with multi-year real production history --- is a direct counter-example, and the mechanism responsible (a non-stationary base rate, not merely the passage of time) is identifiable and monitorable. The practical revision: a deployment should track the \emph{stability of the gray-zone positive rate} between its fine-tuning window and current traffic, not just discriminative-power decay, and treat a detected shift as a distinct re-tuning trigger, independent of the calendar-time-based cadence Section~\ref{sec:deployment}'s other three datasets suggested was unnecessary. This is a strictly narrower, not reversed, conclusion: the risk is real but detectable, and appears specifically correlated with the length and stability of the traffic history --- a variable a synthetic re-slicing of a static benchmark (Section~\ref{sec:deployment}'s method) cannot expose, and only genuinely long-horizon production data (as tested here) can.

\subsection{A prototype monitor for the risk Section~\ref{sec:twitter} identifies}
\label{sec:monitor}

Section~\ref{sec:twitter}'s diagnosis is that comcastcares' fine-tuning failure traces to a single, identifiable cause: a shift in the gray-zone positive rate between the fine-tuning window and current traffic. If that diagnosis is right, a lightweight monitor on the positive-rate signal alone --- computed from labels the online system already produces for free, with no embedding inference or model training --- should be able to flag the instability before it degrades Group E's benefit. This is tested directly, on both brands, using two classical change-point detection methods applied to the chronological \texttt{would\_\allowbreak be\_\allowbreak correct} label sequence: (1) a chunked two-proportion z-test comparing each of 8 sequential held-out chunks against the training-window baseline rate, and (2) a Page-Hinkley sequential test, which processes the held-out stream one example at a time and flags the first position where cumulative deviation from the baseline exceeds a threshold --- closer to how an actual online monitor would run.

\begin{table}[h]
\centering
\small
\begin{tabular}{@{}lccc@{}}
\toprule
\textbf{Brand} & \textbf{Train-window pos.\ rate} & \textbf{z-test: first chunk flagged} & \textbf{Page-Hinkley: flagged at} \\
\midrule
comcastcares & 1.01\% & \textbf{chunk 5/8} ($p = 2\times10^{-20}$) & \textbf{held-out example 664/1{,}106} (chunk 5) \\
AmazonHelp & 1.47\% & never (min $p = 0.06$) & never \\
\bottomrule
\end{tabular}
\caption{Positive-rate monitor results: both methods flag comcastcares' shift and neither false-alarms on AmazonHelp.}
\end{table}

Both methods agree on comcastcares --- flagging the shift roughly halfway through the held-out window, well before the point where Section~\ref{sec:twitter}'s drift experiment showed continued AUC decay setting in --- and neither method false-alarms on AmazonHelp, the brand with a stable positive rate and no observed failure mode. This converts Section~\ref{sec:twitter}'s diagnosis from a plausible explanation into a testable, low-cost detection capability: a deployment could run this monitor continuously on data it already has, with no additional inference cost, and use a flagged shift as a concrete re-tuning trigger rather than relying on a fixed calendar schedule or waiting for downstream AUC to visibly degrade.

This remains a prototype, not a validated production monitor: it was tested retrospectively on a static held-out set rather than a genuinely streaming deployment, its threshold hyperparameters ($\alpha=0.01$ for the z-test; $\delta=0.005, \lambda=8.0$ for Page-Hinkley) were not tuned or validated beyond this single comparison, and it has only been tested on two datasets --- one with the failure mode and one without. But the core finding --- that the exact instability responsible for Section~\ref{sec:twitter}'s counter-example is detectable from labels alone, cheaply and without a false alarm on the brand that didn't need one --- is a concrete answer to the open question Section~\ref{sec:twitter} raised, not just a proposal for one.

\subsection{A tested and refuted hypothesis: action-verb bucketing as a pre-filter}
\label{sec:bucketing}

After this paper's publication, a Reddit discussion (r/LLMDevs, 2026-08-03, prompted by this paper's ``cancel/pause subscription'' example --- a high-similarity pair with different correct answers) proposed a remedy that had until now stayed at the design stage: extract an action verb from each query, forbid queries with different action verbs from sharing a cache bucket, and only run similarity/verification within a bucket. The motivation was to shrink the fraction of candidates the verifier actually has to adjudicate down to the ``genuinely ambiguous, same-bucket'' subset, raising that subset's measured discriminative power. The discussion converged on a concrete implementation principle: build the bucket key only from deterministic morphological normalization (lemmatization), never a semantic/embedding-based one --- since the latter would reintroduce the ``axis problem'' into the bucketing step itself. The full design history is recorded in \texttt{RESEARCH\_\allowbreak PROPOSAL.md} \S10, direction 7.

\textbf{Setup.} Using spaCy dependency parsing (the sentence's ROOT token if tagged VERB, else the sole VERB-tagged token, else extraction fails) on each of three datasets' (LmArena, Quora, corrected SearchQueries) gray zone ($\taulow=0.80, \tauhigh=0.97$), one action verb was extracted for the query and for its matched candidate. Three conditions were compared: verifier-only (today's Group D, no bucketing); bucketing-only (predicting correctness purely from whether the two action verbs match, ignoring the verifier's score entirely); bucketing+verifier (reject outright on an action mismatch; the verifier's AUC is measured only on the remaining action-matched candidates).

\begin{table}[h]
\centering
\small
\begin{tabular}{@{}lcccc@{}}
\toprule
\textbf{Dataset} & \textbf{Extraction-eligible} & \textbf{verifier-only AUC} & \textbf{bucketing+verifier AUC} & \textbf{Difference} \\
\midrule
LmArena & 62.7\% (10{,}089/16{,}102) & 0.8644 [0.8546, 0.8734] & 0.7663 [0.7418, 0.7916] ($n{=}4{,}196$) & \textbf{Sig.\ worse ($-0.098$)} \\
Quora & 57.6\% (9{,}819/17{,}040) & 0.6086 [0.5976, 0.6204] & 0.5870 [0.5725, 0.6010] ($n{=}6{,}352$) & Directionally worse ($-0.022$) \\
SearchQueries (corrected) & 27.3\% (23{,}553/86{,}280) & 0.6215 [0.6145, 0.6287] & 0.6150 [0.6066, 0.6232] ($n{=}17{,}195$) & Flat ($-0.007$) \\
\bottomrule
\end{tabular}
\caption{Action-verb bucketing ablation: verifier-only AUC on the extraction-eligible subset vs.\ bucketing+verifier AUC on the bucket-matched subset, all three datasets. ``Sig.\ worse'' means the two confidence intervals do not overlap; the other two rows' confidence intervals overlap (marginally for Quora, fully for SearchQueries).}
\end{table}

The direction is consistent across all three datasets, and it is negative: on none of them does restricting to the action-matched bucket raise the verifier's discriminative power above the unbucketed baseline; on LmArena it produces a statistically significant \emph{regression}. Diagnosis points to a structural blind spot in the bucketing mechanism itself, not an artifact of this particular extractor's implementation: action-verb bucketing effectively screens out candidates whose verbs are simply different and obviously unrelated (bucketing-only precision is not low --- 0.869 on LmArena), but it is powerless against candidates that share the same verb with an opposite object direction --- e.g., ``convert audio to video'' vs.\ ``convert video to audio,'' which share a verb but are judged non-equivalent by ground truth, yet get bucketed together. Once the easy negatives are filtered out, the negatives that remain in the bucket-matched subset are concentrated almost entirely in exactly this class of hard case, which the verifier itself struggles with most --- so the subset's overall separability drops systematically. The stronger the verifier's baseline discriminative power (LmArena's 0.86, well above Quora/SearchQueries' roughly 0.61), the more visible this ``easy negatives filtered away, only hard ones left'' effect becomes, which explains why the regression's magnitude tracks each dataset's own baseline AUC.

\textbf{Conclusion.} The simplest version of action-verb-only bucketing --- also the version the Reddit discussion converged on as ``structurally safe'' --- is refuted, not merely under-supported, by three independently-sourced datasets under the implementation actually tested here. This runs opposite to the original design hypothesis: action-verb bucketing is not a free precision gain, and where the verifier already has strong discriminative power it can actively backfire. This does not mean structured pre-filtering as a broader idea is unworkable --- using an (action, object) pair rather than the action verb alone as the key could, in principle, close the ``same verb, opposite object direction'' blind spot identified here, but that variant was not tested in this experiment and remains a new, more specific open question (Section~\ref{sec:conclusion}).

\subsection{Does the conclusion change once $\tauhigh$ is also tunable}
\label{sec:tauhigh}

Section~\ref{sec:grids} fixed Group C/D/E's $\tauhigh$ at a single value, 0.97, sweeping only $\taulow$ --- Section~\ref{sec:limitations}'s limitations flagged this explicitly: ``Group C/D's Pareto frontier is therefore only a cross-section anchored at $\tauhigh = 0.97$, not a fully expanded two-dimensional grid.'' Adding a \texttt{--tau-high-grid} flag to \texttt{run\_\allowbreak verified.py} makes $\tauhigh$ sweepable too (implementation-wise this only required widening the similarity range \texttt{score\_\allowbreak gray\_\allowbreak zone} covers to $[\min(\taulow), \max(\tauhigh))$, so the whole two-dimensional grid still shares one real-verifier scoring pass --- no extra model inference). Group D was re-swept over $\tauhigh \in \{0.89, 0.92, 0.95, 0.97, 0.99\} \times$ the existing $\taulow$ grid (same win/tie/loss convention as Section~\ref{sec:groupd}, covering all three datasets: LmArena, Quora, SearchQueries corrected):

\begin{table}[h]
\centering
\small
\begin{tabular}{@{}lccccc@{}}
\toprule
\textbf{Dataset} & $\tauhigh$\textbf{=0.89} & $\tauhigh$\textbf{=0.92} & $\tauhigh$\textbf{=0.95} & $\tauhigh$\textbf{=0.97 (orig.)} & $\tauhigh$\textbf{=0.99} \\
\midrule
LmArena & 0/15 win, +0.05pp & 0/20 win, +0.09pp & 2/25 win, +0.28pp & 14/30 win, +1.42pp & \textbf{24/30 win, +5.18pp} \\
Quora & all tie, +0.16pp & all tie, +0.16pp & all tie, +0.11pp & 0/54 win, +0.11pp & all tie, +0.11pp \\
SearchQueries (corr.) & all tie, +0.23pp & 5/24 win, +0.32pp & 5/30 win, \textbf{+0.33pp} & 1/36 win, +0.30pp & 0/36 win, \textbf{$-0.11$pp} \\
\bottomrule
\end{tabular}
\caption{$\tauhigh$ grid scan, Group D, all three datasets. Figures are each slice's best net lead over the Group A frontier; the ``win'' denominator is that slice's total tested points.}
\end{table}

(The $\tauhigh=0.97$ column recomputes Section~\ref{sec:groupd}'s original report for a consistency check --- LmArena's win count matches the original ``14/30 win, best +1.9pp'' exactly, but the net-lead magnitude and the tie/loss split differ slightly (14/30 win, best +1.42pp here), most plausibly bootstrap-resampling noise rather than a real discrepancy; SearchQueries corrected's column, ``1/36 win, +0.30pp,'' matches the original exactly.)

\textbf{LmArena and SearchQueries are sensitive to $\tauhigh$ in opposite directions.} On LmArena, raising $\tauhigh$ from 0.97 to 0.99 --- letting more high-similarity candidates go through the verifier instead of being auto-approved by similarity alone --- pushes the net lead from +1.42pp to +5.18pp (best point at $\taulow=0.92$, threshold=2.0), more than tripling Section~\ref{sec:groupd}'s originally reported number; lower $\tauhigh$ values perform progressively worse (0.89/0.92 are almost entirely ties). SearchQueries (corrected) runs the opposite way: $\tauhigh=0.99$ is the only slice with a negative net lead across the whole sweep ($-0.11$pp, 35/36 points lose), while $\tauhigh=0.95$ slightly beats the original 0.97 anchor. Quora is essentially unaffected by $\tauhigh$ --- all five slices sit at +0.11--0.16pp --- consistent with the existing diagnosis that ``Quora's ceiling is set by the QQP task's own score-separability limit, not by threshold selection.''

\textbf{Mechanism.} $\tauhigh$ sets the boundary of the ``similarity is high enough, skip the verifier and approve directly'' shortcut. Raising $\tauhigh$ moves more candidates that would otherwise be auto-approved by pure similarity into the verifier's jurisdiction instead --- which only pays off when the verifier is more reliable than raw similarity in that band. LmArena's off-the-shelf verifier already has meaningful discriminative power (untuned AUC 0.72), so handing it more high-similarity candidates is, on average, a net win; SearchQueries' verifier is weaker (untuned AUC 0.5984, already diagnosed in Section~\ref{sec:groupd} as ``has signal but often uses it wrong''), so the same change actively introduces more bad calls. In other words, \textbf{whether widening $\tauhigh$ is worthwhile depends on whether the verifier is actually more trustworthy than ``do nothing and approve by similarity'' in that specific similarity band} --- a dataset-specific judgment, not a universally safe choice of $\tauhigh$.

This section turns Section~\ref{sec:limitations}'s ``$\tauhigh$ fixed at a single value'' limitation from a methodological disclaimer into a finding with a concrete mechanism and a direction that varies by dataset: whether $\tauhigh$ is worth exposing as a tunable deployment parameter cannot be answered in general --- it needs the same domain-specific calibration this paper repeatedly argues verifier selection itself needs, consistent with this paper's recurring theme that how well the verifier matches the business data domain, not any single fixed hyperparameter, is what determines net benefit.

\subsection{Does a larger/stronger off-the-shelf reranker close the gap: capacity or training distribution?}
\label{sec:reranker}

Section~\ref{sec:groupe} raised SearchQueries' off-the-shelf verifier (untuned AUC 0.5984) to 0.7120 via in-domain fine-tuning, but never answered a more basic question: without fine-tuning, does simply swapping in a larger or more broadly-trained off-the-shelf model recover part of that same gain? Section~\ref{sec:conclusion}'s future-work item 2 left this as ``whether a different architecture built for short text could narrow the gap further'' --- this section tests it directly.

\textbf{Design.} Two models, each isolating one variable relative to the \texttt{cross-encoder/\allowbreak ms-marco-MiniLM-L6-v2} (6 layers) Group D has used throughout:

\begin{itemize}[nosep]
  \item \texttt{cross-encoder/\allowbreak ms-marco-MiniLM-L12-v2} (12 layers): the same training data and training objective as L6 (MS MARCO long-web-query/passage ranking), just twice the layers --- a pure \textbf{capacity} control. This is also the exact model Baral et al.\ (2026, see Section~\ref{sec:golo}'s external cross-validation) evaluated, reporting a CRR of only 0.427; this section reruns it on this paper's own three datasets and its own AUC metric, directly comparable to that finding.
  \item \texttt{BAAI/\allowbreak bge-reranker-base} (278M parameters, XLM-RoBERTa backbone): an entirely different training-data lineage --- covering a broader range of retrieval tasks, not limited to MS MARCO-style long web queries --- and substantially larger capacity as well, so \textbf{both capacity and training distribution change together}. If this model clearly beats L12, that points to training-distribution mismatch as the dominant factor; if the two perform similarly, it suggests ``broader distribution alone'' doesn't fix things either.
\end{itemize}

Both models score the same already-cached matches in this paper's standard gray zone ($\taulow=0.80$, $\tauhigh=0.97$, matching Sections~\ref{sec:groupd}/\ref{sec:groupe}) --- reusing the existing match trace rather than re-running nearest-neighbor search --- with no fine-tuning, measuring untuned AUC directly, run on the remote GPU server.

\textbf{Results.}

\begin{table}[h]
\centering
\small
\begin{tabular}{@{}lccccc@{}}
\toprule
\textbf{Dataset} & \textbf{L6 (baseline)} & \textbf{L12 (capacity)} & $\bm{\Delta}$ & \textbf{BGE-reranker-base} & $\bm{\Delta}$ \\
\midrule
LmArena & 0.7200 & 0.8461 & \textbf{+0.1261} & 0.8712 & \textbf{+0.1512} \\
SearchQueries (corrected) & 0.5984 & 0.6210 & +0.0226 & 0.6198 & +0.0214 \\
Quora & 0.6309 & 0.6174 & $-0.0135$ & 0.6381 & +0.0072 \\
\bottomrule
\end{tabular}
\caption{Untuned AUC: L6 baseline vs.\ L12 (pure capacity) vs.\ BGE-reranker-base (capacity + training distribution), all three datasets.}
\end{table}

\textbf{On LmArena both models improve substantially, and by similar margins} (+0.126, +0.151) --- but because capacity and training distribution both vary together on this one dataset, this alone can't tell which variable is doing the work.

\textbf{SearchQueries (corrected) is the dataset that actually separates the two hypotheses, and neither is supported.} L12 (+0.0226) and BGE-reranker-base (+0.0214) are nearly identical --- if training-distribution mismatch were the dominant explanation, BGE's much broader retrieval-task coverage should have clearly outpaced L12's same-distribution-but-bigger design, and it didn't. Both plateau around 0.62, still far short of the 0.7120 fine-tuning reaches in Section~\ref{sec:groupe}. In other words, on this paper's weakest dataset, ``swap in a bigger off-the-shelf model'' and ``swap in a differently-trained off-the-shelf model'' are practically indistinguishable in effect, and both fall well short of fine-tuning on that dataset's own data.

\textbf{Quora shows an anomaly worth noting but not over-interpreting}: L12 actually regresses relative to the L6 baseline ($-0.0135$), while BGE shows a small positive (+0.0072). Both effects are small, and this section's AUC figures don't carry bootstrap confidence intervals (consistent with this paper's convention elsewhere --- only hit-rate/error-rate get confidence intervals, not standalone AUC numbers), so noise can't be ruled out; but if ``a bigger same-distribution model performs worse'' replicates in a larger future test, it would be one more independent example of off-the-shelf model unreliability.

\textbf{Conclusion.} On the one dataset in this paper's test set that actually separates ``capacity'' from ``training distribution'' (SearchQueries), neither hypothesis has explanatory power --- not ``the model is too small,'' nor ``the training distribution is wrong.'' An off-the-shelf model reinforced in either direction produces limited, and nearly indistinguishable, gains. This is consistent with the direction of the Baral et al.\ external cross-validation cited in Section~\ref{sec:golo} (their tested L12 also underperformed, on an entirely different dataset/metric) --- and now adds a direct confirmation using this paper's own datasets and metric. Section~\ref{sec:conclusion}'s future-work item 2 --- whether a different architecture built for short text narrows the gap further --- has now been tested along two axes (same-family-but-bigger, and broader-training-distribution), and neither meaningfully closes it; off-the-shelf model selection is reasonably well ruled out within the scope tested here. The in-domain fine-tuning validated in Section~\ref{sec:groupe} remains the only remedy this paper has confirmed to work.

\subsection{From point estimates to finite-sample risk control: calibrating the gray-zone reuse decision with Conformal Risk Control}
\label{sec:crc}

\textbf{Motivation.} Section~\ref{sec:groupd}'s honest calibration (chronological calibration/test split, Youden's J threshold selection) solves the problem of whether a threshold is peeking at the test set, but the threshold it selects is still just a point estimate --- it never answers how much confidence the resulting error-rate figure deserves given a finite calibration sample. Conformal Risk Control (CRC; Angelopoulos, Bates, Fisch, Lei, Schuster, arXiv:2208.02814, current version v4, 2025-06-13) generalizes split conformal prediction to any bounded, monotone, right-continuous loss function, giving a distribution-free guarantee that holds \textbf{exactly at any finite sample size}: $\mathbb{E}[L_{n+1}(\hat\lambda)] \le \alpha$. This section connects that method to CacheVerifier's existing gray-zone verifier threshold, testing whether the decision to reuse a cached hit can be upgraded from a point estimate with no quantifiable confidence into a decision with a user-specified risk budget and a formal guarantee --- not bolting a confidence interval onto the existing error-rate figure, but replacing threshold selection itself with a calibration procedure that takes a risk budget as input.

\textbf{The quantity being controlled.} Among the existing metrics, \texttt{error\_\allowbreak rate} already has the right \emph{shape} --- an unconditional per-query loss, $P(\text{incorrect} \wedge \text{hit})$, with the denominator over all requests --- it just isn't scoped to the gray zone; \texttt{verifier\_\allowbreak fidelity} is already gray-zone-scoped (its design explicitly excludes the $\tauhigh$/$\taulow$ direct-pass branches, to avoid over- or under-stating the verifier's own error rate) but is conditional ($fp/(tp+fp)$). What CRC actually needs to control is the intersection of the two, which had no standalone implementation before this section:
\[
R_{GZ}(\lambda) = P(\text{incorrect} \wedge \text{accept}_\lambda \mid \text{gray zone})
\]
The new \texttt{gray\_\allowbreak zone\_\allowbreak risk\_\allowbreak curve} (\texttt{cacheverifier/\allowbreak metrics/\allowbreak core.py}) computes this curve. \textbf{Implementation detail:} the accept rule must be strictly right-continuous ($\text{accept}_\lambda(x) := \text{score}(x) > \lambda$, not $\ge$) to satisfy the precondition of CRC's Theorem 1 --- under an inclusive rule, a tied candidate lands on the wrong side of its own discontinuity. This detail has no observable effect on any number reported elsewhere in this paper (continuous cross-encoder scores essentially never tie), but it is a necessary condition for the formal guarantee to hold, not a stylistic choice; the first implementation's ``subtract the point's own index'' approach gave two candidates tied at the exact same score two different risk values, caught in the act by a synthetic test (\texttt{tests/\allowbreak test\_\allowbreak crc.py::\allowbreak test\_\allowbreak b\_\allowbreak tied\_\allowbreak scores\_\allowbreak use\_\allowbreak strict\_\allowbreak inequality\_\allowbreak not\_\allowbreak inclusive}) and fixed with \texttt{searchsorted}.

\textbf{The CRC selector} (Theorem 1, eq.~4, read directly from the paper's own pages, not implemented from memory):
\[
\hat\lambda = \inf\left\{ \lambda : \frac{n}{n+1}\cdot \hat{R}_n(\lambda) + \frac{B}{n+1} \le \alpha \right\}
\]
where $n$ is the calibration sample size and $B$ bounds the loss ($B=1$ in this paper, since $R_{GZ}$ is a 0/1 loss). \textbf{When the candidate set is empty, the algorithm itself defines $\hat\lambda := \lambda_{\max}$} (reject the entire gray zone) --- this is not a failure mode: under this paper's construction, $L_i(\lambda_{\max})=0$ holds exactly for every calibration point (no candidate's score can strictly exceed the sample's own maximum), so the guarantee still holds even along this fallback path, and it should never be described as ``CRC being infeasible.'' This condition degenerates into a dataset-independent closed-form floor --- $n \ge \lceil 1/\alpha - 1 \rceil$ (for $B=1$): $\alpha = 5\%/2\%/1\%/0.5\%$ correspond to 19/49/99/199 respectively. All three datasets clear this floor by 4$\times$--217$\times$ at every calibration size actually used (the tightest case is Quora's existing 909 at $\taulow=0.95$; at the $\taulow=0.80$ band this section uses, the three datasets have 8{,}051/8{,}520/43{,}140 respectively) --- sample size was never the reason reuse was limited in this section's experiments.

\textbf{Protocol R and Protocol T.} CRC's guarantee requires calibration and test samples to be exchangeable. The gray-zone candidates' similarity retrieval is itself stateful --- match quality shifts systematically as the cache grows (see the drift audit below) --- so a chronological split (the ``first half calibration, second half test'' convention Section~\ref{sec:groupd}'s honest calibration already uses) does not satisfy this assumption. But for a fixed, already-realized finite pool of samples --- regardless of what process generated it --- a uniform random split makes the calibration set exchangeable with any test point, purely by the symmetry of the randomization itself; this is the standard justification for split conformal methods and requires no stationarity assumption whatsoever about the underlying generating process. This section therefore runs two parallel protocols: \textbf{Protocol R} (uniform random split, supporting the strict claim ``resampling uniformly from this same fixed benchmark pool,'' a formal CRC result) and \textbf{Protocol T} (chronological split, a deployment-transfer/drift-robustness experiment, explicitly not an application of the CRC theorem and not a claim about real production traffic).

$\tauhigh$ is fixed at 0.97 (the original anchor used throughout most of this paper's experiments, not the grid-search-recommended value newly introduced in Section~\ref{sec:tauhigh} --- using that recommended value would reintroduce exactly the ``post-hoc grid search'' problem Section~\ref{sec:tauhigh} itself exposed, since that sweep used \texttt{run\_\allowbreak verified.py}'s standard pipeline with no calibration/test split). $\taulow$ is fixed at 0.80 (the widest value in the grid, matching the verifier scores already cached, requiring no new model inference).

\textbf{Protocol R results} (50/50 random splits on each dataset, $M=200$ independent repeats, reporting mean realized risk, its 95\% CI, the single-run exceedance probability $P(\text{risk}>\alpha)$, mean reuse, and efficiency relative to the full-data empirical oracle, $\eta = U_{CRC}/U_{oracle}$):

\begin{table}[h]
\centering
\footnotesize
\begin{tabular}{@{}lccp{2.2cm}p{2.2cm}p{2.2cm}p{2.2cm}@{}}
\toprule
\textbf{Dataset} & \textbf{AUC} & \textbf{$n_{cal}$} & $\alpha$=5\% & $\alpha$=2\% & $\alpha$=1\% & $\alpha$=0.5\% \\
\midrule
LmArena & 0.720 & 8{,}051 & risk 0.0499, reuse 68.98\%, $\eta$ 0.999 & risk 0.0200, reuse 40.67\%, $\eta$ 1.001 & risk 0.0100, reuse 21.32\%, $\eta$ 1.002 & risk 0.0050, reuse 9.87\%, $\eta$ 1.022 \\
Quora & 0.631 & 8{,}520 & risk 0.0499, reuse 10.01\%, $\eta$ 0.997 & risk 0.0199, reuse 4.34\%, $\eta$ 0.996 & risk 0.0100, reuse 2.37\%, $\eta$ 1.000 & risk 0.0050, reuse 1.21\%, $\eta$ 0.985 \\
SearchQueries (corr.) & 0.598 & 43{,}140 & risk 0.0498, reuse 13.99\%, $\eta$ 0.999 & risk 0.0199, reuse 6.10\%, $\eta$ 0.997 & risk 0.0100, reuse 3.24\%, $\eta$ 0.999 & risk 0.0050, reuse 1.67\%, $\eta$ 0.997 \\
\bottomrule
\end{tabular}
\caption{Protocol R (random split, $M=200$): mean realized risk, mean reuse, and oracle efficiency $\eta$, four target risk levels, all three datasets.}
\end{table}

(Youden's J's own single-split reference point is reported separately, since it is not alpha-targeted: LmArena mean\_test\_risk=0.0509, Quora 0.2806, SearchQueries 0.2095 --- these three numbers differ enormously, but that only reflects each gray zone's own base rate of incorrectness, not a signal about CRC's calibration quality.)

On all three datasets, at all four alpha levels, mean\_risk sits almost exactly on target --- not conservatively below it. This is by design: the CRC selector picks the loosest $\lambda$ satisfying the constraint, so it should hug the boundary rather than leave safety margin. \textbf{Precisely because of this, roughly half of single runs will show a realized risk slightly above $\alpha$ ($P(\text{risk}>\alpha)$ ranges 0.435--0.525 across the four levels) --- this is what a tightly (but correctly) bound guarantee looks like, not under-coverage; the $\mathbb{E}[L] \le \alpha$ constraint bounds the mean, not any individual realization.} $\eta$ lands within 3\% of 1.0 across all three datasets and all four alpha levels, and the dataset with the largest calibration sample (SearchQueries, 43{,}140) sits closer to 1.0 than the two smaller ones (LmArena/Quora, roughly 8{,}000--8{,}500) --- \textbf{at the sample sizes tested here, CRC's finite-sample guarantee costs almost no utility relative to an oracle that knows the full distribution, and that cost shrinks further with more calibration data, exactly as theory predicts.}

\textbf{Headline finding: AUC ranking does not predict risk-constrained operational-utility ranking.} SearchQueries has the lowest AUC (0.598) of the three datasets, yet its reuse rate is higher than Quora's (AUC 0.631) at \textbf{every one of the four alpha levels} --- 1.67\% vs.\ 1.21\% at $\alpha$=0.5\%. A verifier that ranks worse by AUC can still be operationally more useful under a fixed false-reuse risk budget. This is not a hypothetical illustration --- it is an actual rank reversal measured across this section's three datasets, and it is closer to what a deployer actually cares about (``can this verifier be used under this specific risk constraint'') than reporting an AUC number alone: AUC measures ranking quality, CRC reveals risk-constrained operational utility, and the two are not the same thing.

\textbf{Protocol T results} (chronological split, single run, first half calibration / second half test, the same split point Section~\ref{sec:groupd}'s honest calibration uses):

\begin{table}[h]
\centering
\small
\begin{tabular}{@{}lcccl@{}}
\toprule
\textbf{Dataset} & \textbf{cal.\ incorrect} & \textbf{test incorrect} & \textbf{ratio} & \textbf{exceeds $\alpha$ at all 4 levels?} \\
\midrule
LmArena & 0.4088 & 0.0574 & 0.140 (easier) & No --- conservative throughout \\
Quora & 0.5511 & 0.6563 & 1.191 (harder) & \textbf{Yes --- up to 80\% relative overshoot} \\
SearchQueries (corr.) & 0.5147 & 0.5170 & 1.005 (~cancels) & Yes, mild (~20--25\% overshoot) \\
\bottomrule
\end{tabular}
\caption{Protocol T (chronological split, single run): calibration- vs.\ test-half incorrect rate and guarantee status, all three datasets.}
\end{table}

This result is directly explained by the direction gray-zone composition drifts over stream position, quantified with an audit that needs no new inference --- just binning the already-cached trace into deciles of stream position: LmArena's gray-zone incorrect rate falls from 63.5\% at decile 0 to 5.0\% at decile 9; Quora runs the opposite direction, rising from 43.6\% to 67.6\%; SearchQueries rises then falls, and its net effect happens to nearly cancel at exactly the 50/50 split point. LmArena's drift direction happens to favor chronological calibration (calibrating on the harder first half and applying it to the easier second half is naturally conservative); Quora's drift runs the opposite way, calibrating on the easier first half systematically under-preparing for the harder second half --- \textbf{this is this paper's first concrete, measured (not hypothetical) demonstration of an exchangeability failure}, qualitatively different from Protocol R's roughly 50\% single-run exceedance rate (where the guarantee still holds, just tightly bound): here, all four alpha levels exceed target in the same direction on Quora, which is a systematic pattern, not sampling noise. \textbf{The formal guarantee reported under Protocol R requires the random-split assumption to hold; treating a threshold calibrated on real chronological order as carrying a CRC-theorem guarantee about future production traffic is not safe on data like Quora's.}

\textbf{vCache methodology audit: the conclusion is not to attempt a direct numeric comparison.} Section~\ref{sec:groupb} already discusses Group B in detail (this paper's line-by-line port of vCache's \texttt{VerifiedDecisionPolicy}, \texttt{cacheverifier/\allowbreak cache/\allowbreak adaptive\_\allowbreak threshold.py}); to judge whether its \texttt{target\_\allowbreak error\_\allowbreak rate} and this section's CRC $\alpha$ can be compared numerically, that port's implementation was re-examined, and vCache's own abstract was checked to confirm it ``learns an optimal threshold for each cached prompt.'' Three structural mismatches, any one of which alone would be enough to make a direct numeric comparison meaningless:

\begin{enumerate}[nosep]
  \item \textbf{Decision granularity.} vCache maintains a separate threshold \textbf{per individual cache entry}, updated online as that entry accumulates its own observation history ($\text{min\_observations}=6$); this section's CRC calibrates \textbf{one global threshold over the entire pooled gray zone}. ``vCache's error rate'' is really an average over many entry-level policies at different stages of maturity; ``CRC's certified risk'' is a single number for one shared policy.
  \item \textbf{Verification timing.} vCache's \texttt{observe()} only updates an entry's history on a miss (the explore branch) --- the ported implementation's own documentation states plainly that ``only MISSES ever get their correctness checked... verification, when it happens at all, never sits on the hit path,'' matching vCache's own \texttt{\_\allowbreak \_\allowbreak update\_\allowbreak cache}, which runs only on the explore branch. This is \textbf{asynchronous} verification: a hit is trusted unconditionally in the moment, and verification, when it happens, only informs future decisions. This section's gray-zone gating is \textbf{synchronous}: it decides before serving the current hit. \texttt{target\_\allowbreak error\_\allowbreak rate} and this section's $\alpha$ describe guarantees under two different causal structures, not two settings of the same dial.
  \item \textbf{Guarantee type.} vCache's explore probability $\tau$ comes from a fitted one-dimensional logistic regression of correctness on similarity, plus a delta-method/normal-approximation confidence bound (\texttt{scipy.stats.norm.ppf}), patched with a hardcoded small-sample empirical variance lookup table (\texttt{\_\allowbreak VARIANCE\_\allowbreak MAP}) for the approximation's known weakness at very few per-entry observations (6--48) --- a parametric, asymptotic-flavored construction. CRC (Theorem 1) is distribution-free and exact at any finite $n$. Comparing ``vCache's guaranteed error rate'' against ``CRC's certified risk'' would be comparing an approximate, model-dependent bound to an exact, model-free one --- not the same kind of claim even before asking what population each is defined over.
\end{enumerate}

This section therefore does not report ``vCache's error rate is $X\%$, CacheVerifier's CRC-certified risk is $Y\%$, ours is better/worse'' --- any apparent numeric edge could just as easily reflect one of these three mismatches as a real difference in mechanism quality. The defensible framing is structural, filled in here with verified facts rather than placeholders:

\begin{table}[h]
\centering
\small
\begin{tabular}{@{}lll@{}}
\toprule
\textbf{Dimension} & \textbf{This section (CRC)} & \textbf{vCache} \\
\midrule
Decision granularity & one global threshold, pooled gray zone & per-cache-entry threshold \\
Verification timing & synchronous (gates current hit) & asynchronous (miss/explore only) \\
Guarantee type & distribution-free, exact, finite-sample & parametric, asymptotic-flavored \\
Calibration & offline, batch calibration/test split & online, per-entry, incremental \\
Cold start & documented \texttt{COLD\_\allowbreak START\_\allowbreak WARNING\_\allowbreak THRESHOLD} & per-entry $\text{min\_observations}=6$ \\
Adapts to local difficulty & no & in principle, per cached prompt \\
\bottomrule
\end{tabular}
\caption{Structural comparison: this section's CRC calibration vs.\ vCache. Not a numeric benchmark --- see text.}
\end{table}

A genuine head-to-head, if ever wanted, would need new engineering on one side or the other --- reimplementing this paper's gray-zone gate as a per-entry asynchronous policy, or reimplementing vCache's decision rule under a single pooled synchronous threshold --- not simply running the two existing implementations against each other. That is scoped as a separate, explicitly unfinished future direction, not part of this section's conclusions.

\textbf{Conclusion.} This section upgrades the gray-zone reuse decision from a point estimate with no quantifiable confidence into a decision with a user-specified risk budget and an exact finite-sample guarantee under a random-split assumption, and that guarantee costs almost no utility at the sample sizes tested here (8{,}051--43{,}140; $\eta$ within 3\% of 1.0 throughout). This upgrade surfaces two facts that were never quantified before: first, a verifier's AUC ranking does not equal its operational-utility ranking under a specific risk budget (SearchQueries turns a lower AUC into higher reuse than Quora); second, calibrating on historical traffic and deploying to future traffic --- an operation nearly every production system performs --- is only as safe as the traffic's own exchangeability, and Quora's real, systematic (not incidental) guarantee violation shows this cannot be assumed by default. Sample size itself, at the scales tested in this section, was never the reason reuse was limited --- the limit comes from the verifier's own discriminative power, consistent with the diagnosis in Sections~\ref{sec:groupd}/\ref{sec:groupe}, just now expressed in the formal language of a risk budget.

\subsection{E': rewrite instead of reject --- a negative ablation against the TweakLLM paradigm}
\label{sec:rewrite}

\textbf{Motivation.} Everywhere else in this paper, a verifier rejection is converted into a plain \texttt{miss} (falling through to a fresh LLM call, treated as necessarily correct in this simulation, zero error risk, but zero cache benefit either). TweakLLM (Cheema et al., 2025, arXiv:2507.23674, already listed in the related-work table) proposes a different routing: instead of discarding a rejected candidate, edit its cached answer with a lightweight LLM into a version that fits the new query, and serve that. Section~\ref{sec:conclusion}'s future-work item 9 had left this an untested open question --- where ``reject $\to$ rewrite $\to$ serve'' lands on this paper's error-rate/hit-rate Pareto frontier relative to the existing binary gate. This section is the first measurement.

\textbf{Method.} Building on Group E's existing honest calibration framework (same as Section~\ref{sec:groupd}: the gray zone is split by stream position into calibration/test halves, and the threshold is selected with Youden's J on the calibration half only), this section takes the candidates rejected by that threshold on the \textbf{held-out test half}, and splits them by \texttt{would\_\allowbreak be\_\allowbreak correct} (whether the candidate and the new query share an equivalence class):

\begin{itemize}
\item \textbf{false-reject stratum} (the candidate was actually correct; the verifier rejected it in error)
\item \textbf{true-reject stratum} (the candidate genuinely was a different question; the rejection was correct)
\end{itemize}

100 examples are sampled per stratum per dataset (LmArena/Quora/SearchQueries-corrected; population sizes 420/693/7{,}004 for false-reject and 298/3{,}668/15{,}411 for true-reject --- all far above the sampling target). For each sample: DeepSeek (\texttt{deepseek-chat}, the same API-hosted-small-model substitution principle used throughout this paper) edits the candidate's answer to fit the new query (an editing prompt, not from-scratch generation), then the same model, given that query's own reference answer, judges whether the rewritten answer is semantically correct (YES/NO, not requiring a verbatim match). \texttt{RequestOutcome(action="hit", correct=<verdict>)} directly reuses this paper's existing \texttt{error\_\allowbreak rate}/\texttt{bootstrap\_\allowbreak ci} machinery --- no new statistics code. This is a sampled ablation, not a full-trace simulation, scoped to 3 datasets $\times$ 2 strata $\times$ 100.

\textbf{Results.}

\begin{table}[h]
\centering
\small
\begin{tabular}{lccc}
\toprule
\textbf{Dataset} & \textbf{false-reject incorrect (95\% CI)} & \textbf{true-reject incorrect (95\% CI)} & \textbf{rewrite latency mean/p95 (ms)} \\
\midrule
LmArena & 58.0\% [49.0, 68.0] & 61.0\% [51.0, 71.0] & 3{,}201/6{,}159 (false), 4{,}067/7{,}637 (true) \\
Quora & 33.0\% [24.0, 43.0] & 30.0\% [21.0, 39.0] & 5{,}840/8{,}225 (false), 6{,}181/7{,}891 (true) \\
SearchQueries (corrected) & 24.0\% [16.0, 32.0] & 22.0\% [14.0, 30.0] & 2{,}665/3{,}909 (false), 2{,}935/4{,}603 (true) \\
\bottomrule
\end{tabular}
\caption{E' rewrite-then-judge incorrect rates, false-reject vs. true-reject strata, all three datasets.}
\end{table}

\textbf{Headline finding: the false-reject and true-reject incorrect rates fall inside each other's 95\% confidence intervals on all three datasets, with no dataset showing the expected-direction difference.} This is not the intuitive result --- in the false-reject stratum the candidate was already correct, so a rewrite that faithfully preserves it should pass judgment easily, but empirically it is no more reliable than the true-reject stratum. Manual review of both strata's rewrite examples shows the editing itself is generally good quality (e.g.\ precisely substituting ``shepherd'' for ``farmer,'' leaving the rest intact) --- the failures concentrate in the judging step: the reference answer is an \textbf{independently written} text for that specific new query, and even a semantically sound rewrite can be marked ``not matching'' over surface differences in numeric convention or structure --- a stricter bar than the equivalence\_id-based label this paper otherwise relies on. A safety-relevant edge case also surfaced in LmArena's true-reject stratum: a query asking for pornographic-content streaming sites matched an unrelated ``YouTube downloader software'' candidate (a correct true reject); the rewrite did not cleanly refuse the way the reference answer does, but instead grafted the original candidate's content onto a hedge --- a more ambiguous, partially-compliant output, a risk inherent to the ``rewrite instead of reject'' idea itself, independent of the judging methodology.

\textbf{Conclusion.} Within the scope tested here, ``reject $\to$ rewrite $\to$ serve'' shows no sign of a net benefit over the existing binary gate. Under the binary gate, a rejection is a \texttt{miss} --- zero error risk, at the cost of one full LLM call. E' converts this previously zero-risk population into ``hits'' with a 22\%--61\% error rate, and the rewrite call's own latency (mean 2.7--6.2 seconds) is not clearly lower than a fresh generation's would be either. This is a negative ablation: not ``the TweakLLM paradigm doesn't work,'' but that its simplest form --- reusing a rewritten answer with no additional correctness check --- shows no measurable net benefit in this paper's error-rate/hit-rate framework.

\textbf{Limitations.} (1) Judging compares the rewritten answer against that query's own independently labeled reference answer rather than the original candidate, plausibly inflating the false-reject stratum's apparent error rate with reference-paraphrase noise unrelated to rewrite quality. (2) 68.8\% of SearchQueries-corrected's reference answers were themselves generated by DeepSeek, and this section's rewriting and judging also use DeepSeek --- a same-model-family bias cannot be ruled out for that dataset's notably lower error rate. (3) Rewrite and judge calls used bounded concurrency rather than this paper's otherwise-strict sequential calling convention; each call's own latency measurement remains real, but this is not fully methodologically consistent with the Section~\ref{sec:latency} Oracle-latency measurement. (4) Only the ``always rewrite unconditionally'' simplest policy was tested --- a second verification pass on the rewritten answer (a double gate) was not tested, an independent, untested follow-up.

\subsection{Is looking only at the nearest neighbor not enough: testing a Top-K candidate cascade}
\label{sec:topk}

\textbf{Motivation.} Everywhere else in this paper, \texttt{SynchronousVerifiedPolicy} (Section~\ref{sec:verifiers}; \texttt{cacheverifier/cache/synchronous\_\allowbreak verified.py}) retrieves a single nearest neighbor only --- \texttt{VectorCacheStore.query} hardcodes \texttt{k=1}. If that one candidate falls in the gray zone and the verifier rejects it, the request is judged a \texttt{miss} outright, even when the semantic second-nearest neighbor is in fact the correct answer and was never given a chance to be seen by the verifier. This gap was flagged by an external, independent re-derivation of this project (uninvolved in writing this paper, working from the public repository and cross-checking source code file by file; every one of this paper's already-published, checkable numbers it touched --- the 1687.8ms latency, the 4.4x--29.1x Group B fix, the 0.5984$\to$0.7120 SearchQueries AUC gain, among others --- checked out accurately). This section is the first test of that specific gap.

\textbf{A prior monotonicity argument narrows the scope of the test.} HNSW returns neighbors in descending similarity order, so rank $m$'s similarity is necessarily $\le$ rank $m{-}1$'s. Consequently, if the rank-1 candidate is already $<\taulow$, every lower-ranked candidate is also necessarily $<\taulow$ --- looking further down the list is \textbf{provably} useless on that branch. The only branch that can plausibly benefit is ``rank-1 falls in the gray zone $[\taulow, \tauhigh)$ and is rejected by the verifier'': in that case it is worth trying rank-2, rank-3, \ldots\ until some candidate's similarity drops below $\taulow$ (past which point the same monotonicity argument guarantees going further is pointless too). Accordingly, \texttt{VectorCacheStore} gained a \texttt{query\_\allowbreak topk} method, and a parallel cascade decision path was added alongside (not inside) \texttt{SynchronousVerifiedPolicy} --- no class any already-published result depends on was modified:

\begin{verbatim}
for candidate in top_K (descending similarity):
    if candidate.similarity < tau_low: break   # monotonicity: later
                                                 # candidates can't be better
    if candidate.similarity >= tau_high: return HIT
    if verifier(query, candidate).approved: return HIT
    # else: rejected, try the next-ranked candidate
return MISS
\end{verbatim}

\textbf{Phase 0: a pure ground-truth ceiling, calling no verifier at all} (\texttt{scripts/topk\_\allowbreak ceiling\_\allowbreak diagnostic.py}). Existing embeddings are reused to build a single top-5-neighbor HNSW trace pass; for every request whose rank-1 candidate is gray-zone-and-wrong, a pure count checks whether ranks 2--5 contain a candidate whose similarity is still $\ge \taulow$ (hence actually reachable by the cascade) and is in fact correct:

\begin{table}[h]
\centering
\small
\begin{tabular}{lcccc}
\toprule
\textbf{Dataset} & \textbf{rank-1 gray-zone \& wrong} & \textbf{K=5 raw ceiling} & \textbf{K=5 reachable ceiling (\% of subset)} & \textbf{as \% of all requests} \\
\midrule
LmArena & 3{,}755 / 60{,}000 & 451 & 450 (11.98\%) & 0.75\% \\
SearchQueries (corrected) & 44{,}511 / 150{,}000 & 9{,}529 & 8{,}487 (19.07\%) & 5.66\% \\
Quora & 10{,}285 / 60{,}000 & 1{,}296 & 973 (9.46\%) & 1.62\% \\
\bottomrule
\end{tabular}
\caption{Phase 0: pure ground-truth Top-K ceiling, no verifier calls.}
\end{table}

The ceiling is non-trivial on all three datasets (not noise-level), and largest on SearchQueries --- precisely the dataset where the off-the-shelf verifier's discriminative power is weakest (Section~\ref{sec:groupd}, AUC 0.598), suggesting retrieval-side recall and verifier discriminative power are two separately quantifiable, independent bottlenecks.

\textbf{Phase 2: a cascade with a real verifier plugged in} (\texttt{cacheverifier/experiments/topk\_\allowbreak sweep.py} + \texttt{scripts/run\_\allowbreak topk\_\allowbreak cascade.py}, K=2). This continues Section~\ref{sec:verifiers}'s ``score once, re-threshold many times'' design: \texttt{score\_\allowbreak cascade\_\allowbreak candidates} only scores the (record, rank) pairs the cascade could \textbf{actually reach} under a fixed $\tauhigh$ --- i.e.\ candidates where every earlier-ranked sibling also falls in $[\min(\taulow), \tauhigh)$ --- skipping downstream candidates whose sibling already hit directly or already dropped below $\taulow$ (verified that this pruning changes zero published numbers, and only removes scoring calls \texttt{replay\_\allowbreak cascade} would never have looked up anyway). All three datasets, with both the off-the-shelf and the domain-fine-tuned verifier (the same checkpoints as Section~\ref{sec:groupe}), were run at full scale on a remote GPU server (Tesla T4): cross-encoder inference on real (not synthetic-short) query/answer text ran at roughly 4.3--4.6 examples/sec on CPU versus roughly 145 examples/sec on GPU ($\sim$33x); the largest candidate volume (161{,}403 pairs for SearchQueries's K=2 scoring pass) would have taken over a day on local CPU. This is the only section in this paper that uses a GPU rather than the CPU path used by default elsewhere, and it does not change any numerical methodology.

\textbf{Results: exact-match on the same $(\taulow, \text{threshold})$ grid point, bootstrap 95\% CI significance, K=1 baseline vs.\ K=2 cascade.}

\begin{table}[h]
\centering
\small
\begin{tabular}{llccccc}
\toprule
\textbf{Dataset} & \textbf{Verifier} & \textbf{grid pts} & \textbf{hit\_rate up (sig.)} & \textbf{down (sig.)} & \textbf{error\_rate up (sig.)} & \textbf{mean $\Delta$ hit} \\
\midrule
LmArena & off-the-shelf & 30 & 25 & 0 & 0 & +0.80pp \\
LmArena & fine-tuned & 54 & 29 & 0 & 0 & +0.61pp \\
Quora & off-the-shelf & 54 & 0 & 0 & 1 & +0.10pp \\
Quora & fine-tuned & 48 & 1 & 0 & 1 & +0.11pp \\
SearchQueries (corrected) & off-the-shelf & 36 & 23 & 0 & 19 & +1.18pp \\
SearchQueries (corrected) & fine-tuned & 54 & 28 & 0 & 23 & +0.81pp \\
\bottomrule
\end{tabular}
\caption{K=2 cascade vs.\ K=1 baseline, matched-grid-point comparison.}
\end{table}

\textbf{The three datasets tell three different stories, consistent with --- not a coincidental repeat of --- this paper's recurring ``dataset-dependent, not one-size-fits-all'' pattern already seen in Sections~\ref{sec:groupd} and \ref{sec:groupe}.}

\textbf{LmArena --- close to a free lunch.} Regardless of which verifier is used, error\_rate never moves significantly at any grid point, while hit\_rate rises significantly. One concrete operating point ($\taulow=0.86$, threshold$=-2.0$, off-the-shelf): K=1 hit\_rate$=0.9130$ [0.9106, 0.9151], error\_rate$=0.0638$ [0.0619, 0.0658]; K=2 hit\_rate$=0.9208$ [0.9188, 0.9231] (CIs don't overlap, significantly up), error\_rate$=0.0652$ [0.0633, 0.0673] (CIs overlap, no significant change) --- looking at one more candidate carries almost no observable extra error cost.

\textbf{Quora --- essentially no effect.} The mean gain under both verifiers sits around 0.1pp, consistent with Phase 0's already-small ceiling (1.62\%) --- not a failure of the cascade mechanism itself, just a narrow well to draw from on this dataset.

\textbf{SearchQueries --- a real trade-off, not a free lunch; fine-tuning mitigates but does not eliminate it.} Under the off-the-shelf verifier, 19 of 36 grid points (53\%) also show a significant error\_rate increase; switching to the fine-tuned verifier brings that down to 23 of 54 (43\%) --- the right direction, but not eliminated. One concrete operating point ($\taulow=0.80$, threshold$=7.0$, off-the-shelf, this dataset's best net-gain point): K=1 hit\_rate$=0.5531$ [0.5512, 0.5555], error\_rate$=0.1620$ [0.1599, 0.1638]; K=2 hit\_rate$=0.5878$ [0.5858, 0.5901] (significantly up), error\_rate$=0.1807$ [0.1785, 0.1825] (also significantly up) --- giving a verifier that is already comparatively weak (Section~\ref{sec:groupd}, AUC 0.598) a second chance also gives it a second chance to be wrong, exactly the risk anticipated at design time, not a surprise discovered after the fact.

\textbf{K and fine-tuning are complementary levers, not substitutes for one another.} At a fixed, reasonable error-rate ceiling, the four combinations (K=1/K=2 $\times$ off-the-shelf/fine-tuned) always hand the highest hit\_rate to whichever combination uses both --- e.g.\ on SearchQueries at error\_rate $\le 5\%$: K=1+off-the-shelf$=0.2838$, K=1+fine-tuned$=0.3038$, K=2+off-the-shelf$=0.2845$, K=2+fine-tuned$=0.3063$, with the ``fine-tuned + K=2'' combination the highest of the four; Quora's four numbers at the same ceiling (0.1172/0.1311/0.1172/0.1327) show the same ordering. This particular ``fix an error ceiling, find the best point'' comparison is limited on LmArena by that dataset's coarser threshold grid (5 candidate values, versus 8--9 for Quora/SearchQueries) and can occasionally produce a misleading negative number from a step artifact between adjacent grid points --- the matched-grid-point-plus-CI test above (25--29 points significantly up, 0 significantly down) is the more trustworthy evidence on LmArena, which is why this section leads with that method rather than ``fix the error rate, find the optimum.''

\textbf{Limitations.} (1) Only K=2 was tested; the Phase 0 ceiling table shows K=3/4/5 still carry diminishing but non-zero marginal returns (LmArena rises from K=2's implied 6.26\% to K=5's 11.98\%, SearchQueries from roughly 12--13\% to 19.07\%) --- whether a larger K is worth the extra verifier calls (worst-case cascade latency stacks with K; see Section~\ref{sec:latency}'s already-non-negligible single-call synchronous verification latency) is an untested open question. (2) Only the cross-encoder verifier family was tested (off-the-shelf and fine-tuned), similar in scope to Section~\ref{sec:reranker}'s ``a larger/more-broadly-trained off-the-shelf reranker was also tested but did not close the gap'' --- whether the cascade mechanism itself is sensitive to verifier architecture choice was not tested. (3) The ``K and fine-tuning are complementary'' finding rests on the ``fix the error rate, find the best hit\_rate'' comparison, already known to be less reliable than matched-point comparison on LmArena (see above); its strong evidence comes mainly from Quora and SearchQueries, the two datasets with finer grids. (4) The latency cost analysis is throughput/arithmetic only (the sequential-vs.-parallel scoring trade-off for multiple candidates was discussed at design time; see the corresponding memory record) --- no real end-to-end call-latency measurement was made the way Section~\ref{sec:latency} did for the Oracle. (5) Significance for error\_rate increases, like hit\_rate increases, used only pointwise non-overlapping bootstrap CIs, without the multiple-comparison correction Sections~\ref{sec:groupd} and~\ref{sec:crc} applied --- some false-positive significant results among 36--54 grid points would be expected by chance, but the observed rate (SearchQueries, 43\%--53\%) is far higher than any reasonable false-positive rate would explain, and is unlikely to be a multiple-comparison artifact.

\subsection{The CRC gate's self-selection feedback loop: a genuine online closed-loop test}
\label{sec:selfselect}

\textbf{Motivation.} Section~\ref{sec:crc}'s Protocol R and Protocol T both test static splits of a fixed, already-completed historical trace (random vs.\ chronological order) --- neither is a genuine closed loop. If the CRC gate's own reuse/reject decisions feed back into future cache state --- approved candidates persisting and getting hit more, rejected ones getting silently crowded out of the candidate pool --- that could induce a self-selection feedback loop: the calibration distribution itself gets systematically shaped by the verifier's own past decisions, not just drifting naturally with time. This was hypothesized to potentially be a stronger failure mode than the exchangeability violation Protocol T already demonstrated, but it had never been tested in isolation.

Before designing this test, a more basic premise was checked first: \textbf{every single experiment} elsewhere in this paper --- Groups A--E, Section~\ref{sec:crc}'s CRC Protocol R/T, Section~\ref{sec:topk}'s Top-K cascade --- inherits one convention from vCache's own benchmark harness (\texttt{cacheverifier/\allowbreak experiments/\allowbreak verified\_\allowbreak sweep.py::\allowbreak build\_\allowbreak match\_\allowbreak trace}, \texttt{cacheverifier/\allowbreak experiments/\allowbreak runner.py::\allowbreak ExperimentRunner.run}, both implemented this way): every request gets inserted into the cache \textbf{unconditionally}, regardless of whether it was a hit or a miss. That means the candidate pool's composition at any point in the stream has never actually depended on any policy's history of decisions --- the self-selection mechanism this section tests has never been simulated anywhere in this paper before. Not ``not yet finished testing'' --- ``never started.''

\textbf{Implementation.} A new script, \texttt{scripts/\allowbreak crc\_\allowbreak closed\_\allowbreak loop\_\allowbreak self\_\allowbreak selection.py}, does not modify \texttt{verified\_\allowbreak sweep.py}/\texttt{runner.py} (every other experiment in this paper depends on their current behavior) and instead implements a genuine online closed loop from scratch. The same record stream is run through three regimes in sequence:

\begin{verbatim}
baseline:          insert every record regardless of hit/miss
                    (= the convention every other section of this
                       paper uses)
                    + threshold calibrated once from an initial
                      warm-up slice, then frozen
self_select:        insert only on miss (a hit reuses; no duplicate
                       write)
                    + threshold fixed at baseline's own calibrated
                      value
self_select_recal:  insert only on miss
                    + threshold re-derived online via
                      crc_select_threshold: every 500 new gray-zone
                      observations, recalibrate from the most recent
                      2,000 (sliding window)
\end{verbatim}

\texttt{self\_\allowbreak select} changes exactly one variable relative to \texttt{baseline} (write semantics) --- the threshold value is identical, transplanted as a plain number from \texttt{baseline}'s own calibration rather than re-derived on \texttt{self\_\allowbreak select}'s own data --- so any difference between the two can be attributed to ``does cache content get shaped by its own decisions'' alone, not to the threshold also happening to differ. \texttt{self\_\allowbreak select\_\allowbreak recal} adds a second variable (online recalibration) on top of \texttt{self\_\allowbreak select}, testing whether recalibration can compensate for whatever damage self-selection causes. Sanity check: \texttt{baseline}'s Quora gray-zone candidate count (17{,}040) matches this paper's existing static trace cache file (the same one Section~\ref{sec:crc} uses) exactly, digit for digit --- confirming the new harness faithfully reproduces this paper's existing mechanism rather than being an independent implementation that might carry its own bugs.

Regime comparisons use only post-warm-up observations (\texttt{baseline} alone goes through a warm-up period where every gray-zone candidate is rejected under a fixed fail-closed policy --- matching \texttt{cacheverifier-service}'s own real cold-start default --- before its threshold is calibrated; \texttt{self\_\allowbreak select}/\texttt{self\_\allowbreak select\_\allowbreak recal} use the transplanted, already-calibrated threshold from their very first observation and never go through this warm-up period. Including the warm-up period in the comparison would pull \texttt{baseline}'s overall risk down artificially via this zero-risk stretch, producing a spurious difference unrelated to the self-selection mechanism itself). Risk differences use an unpaired percentile bootstrap (\texttt{bootstrap\_\allowbreak risk\_\allowbreak difference}, 2{,}000 resamples, 95\% CI; once two regimes diverge, their matched candidate sequences are no longer the same, so there is no natural way to pair observations one-to-one --- this is deliberately unpaired, not paired).

\textbf{Quantitative results} ($\alpha=0.02$; ``direct-hit rate'' and ``cache sparsification'' are both taken from \texttt{baseline}/\texttt{self\_\allowbreak select}'s actual runs):

\begin{table}[h]
\centering
\small
\begin{tabular}{@{}lccccc@{}}
\toprule
\textbf{Dataset} & \textbf{direct-hit} & \textbf{self\_select cache} & \textbf{self-selection alone} & \textbf{online recal.} & \textbf{recal.\ vs.\ baseline} \\
\midrule
Quora & 3.7\% & 56{,}912/60{,}000 ($-5.1\%$) & $-0.0001$, CI $[-0.0036,+0.0034]$ \textbf{n.s.} & $-0.0055$, CI $[-0.0090,-0.0021]$ & $-0.0057$, CI $[-0.0091,-0.0023]$ \textbf{beats baseline} \\
SearchQueries (corr.) & 28.1\% & 102{,}929/150{,}000 ($-31.4\%$) & $+0.0068$, CI $[+0.0050,+0.0085]$ \textbf{sig.\ harm} & $-0.0195$, CI $[-0.0211,-0.0180]$ & $-0.0127$, CI $[-0.0142,-0.0112]$ \textbf{beats baseline} \\
LmArena & 72.8\% & 13{,}586/60{,}000 ($-77.4\%$) & $+0.0305$, CI $[+0.0274,+0.0337]$ \textbf{risk $>3\times$} & $-0.0239$, CI $[-0.0270,-0.0206]$ & $+0.0065$, CI $[+0.0038,+0.0093]$ \textbf{still worse} \\
\bottomrule
\end{tabular}
\caption{Closed-loop self-selection: realized-risk differences (unpaired bootstrap, 95\% CI) across three regimes and three datasets. ``n.s.'' = not significant; all other entries are significant (CI excludes zero).}
\end{table}

The three datasets sit precisely on one monotonic dose-response curve, not three unrelated dataset quirks: the higher the direct-hit rate (i.e., the larger the share of queries similar enough to an existing cache entry that the verifier is never even consulted), the more a write-on-miss-only policy --- the semantics that more closely match a real production system --- sparsifies the cache, and the larger the harm self-selection alone causes. Quora (3.7\%/5.1\%) shows no detectable effect; SearchQueries (28.1\%/31.4\%) causes a moderate, significant harm; LmArena (72.8\%/77.4\%) causes a strongly significant harm that more than triples realized risk. This is the first real, measured confirmation of the self-selection closed-loop mechanism anywhere in this paper --- not a theoretical hypothesis anymore.

One non-monotonic wrinkle is worth recording honestly. Online recalibration fully compensates for self-selection's harm on both Quora and SearchQueries --- even beating the original baseline --- but not on LmArena, where recalibration still leaves a significant residual harm relative to the original (non-self-selecting) baseline. This suggests that once the harm magnitude clears some threshold, recalibration alone is no longer sufficient to fully offset it; that threshold sits somewhere between SearchQueries' 28\% and LmArena's 73\% direct-hit rate, but the three datasets merely happened to land on either side of it rather than being designed to locate it --- its exact value remains an open question.

\textbf{[2026-08-29 update] Locating this threshold: write probability instead of synthetic data construction, interpolating directly inside LmArena.} Two earlier approaches to locating this threshold were considered and rejected. (1) Sweeping \tauhigh\ to artificially raise or lower a dataset's direct-hit rate --- rejected, because \tauhigh\ isn't just a measure of ``how much redundancy the traffic itself contains,'' it is also the boundary at which similarity alone is trusted without the verifier (Section~\ref{sec:tauhigh} already showed moving \tauhigh\ is never a neutral operation); an effect measured this way would confound ``changing self-selection exposure'' with ``the direct-hit shortcut itself becoming less reliable,'' not a clean single-variable manipulation. (2) Injecting synthetic exact-duplicate records into a low-redundancy dataset (e.g.\ Quora) to artificially pull its direct-hit rate up --- rejected, because this introduces a new ``is synthetic duplicate data realistic enough'' debate, requires more engineering, and doesn't directly use the one real dataset already shown to exhibit the effect.

The final choice instead generalizes \texttt{run\_\allowbreak regime}'s binary switch (\texttt{insert\_\allowbreak on\_\allowbreak hit}, whether to write on a hit) into a continuous parameter \texttt{insert\_\allowbreak on\_\allowbreak hit\_\allowbreak probability} --- on a hit, the cache is still rewritten with probability $p$ ($p=1$ is the original baseline, $p=0$ is \texttt{self\_\allowbreak select}, and intermediate values correspond to a real policy such as ``occasionally rewrite popular entries,'' e.g.\ a TTL-driven periodic refresh). This keeps the embedding model, verifier, \taulow/\tauhigh, and threshold calibration protocol all unchanged --- the only manipulated variable is the rewrite probability on a hit --- requires no synthetic data construction, and this parameter itself directly corresponds to a real, implementable product mitigation, not just a diagnostic knob.

On LmArena (reusing the already-calibrated \texttt{frozen\_\allowbreak threshold}, not re-derived), the ``self-selection + online recalibration'' protocol was run once for each $p \in \{0.1, 0.25, 0.5, 0.75\}$ (\texttt{scripts/\allowbreak crc\_\allowbreak closed\_\allowbreak loop\_\allowbreak write\_\allowbreak probability\_\allowbreak sweep.py}), compared against the original baseline via an unpaired bootstrap:

\begin{table}[h]
\centering
\small
\begin{tabular}{@{}lcccl@{}}
\toprule
\textbf{$p$} & \textbf{Realized risk} & \textbf{Diff vs.\ baseline} & \textbf{95\% CI} & \textbf{Verdict} \\
\midrule
0.0 (already tested) & 0.0203 & $+0.0065$ & $[+0.0038,+0.0093]$ & sig.\ residual harm \\
0.1 & 0.0198 & $+0.0060$ & $[+0.0031,+0.0088]$ & sig.\ residual harm \\
0.25 & 0.0190 & $+0.0052$ & $[+0.0025,+0.0080]$ & sig.\ residual harm \\
0.5 & 0.0186 & $+0.0049$ & $[+0.0021,+0.0076]$ & sig.\ residual harm \\
0.75 & 0.0179 & $+0.0042$ & $[+0.0013,+0.0070]$ & sig.\ residual harm \\
\bottomrule
\end{tabular}
\caption{Write-probability sweep on LmArena: realized risk vs.\ the original insert-always baseline, unpaired bootstrap, 95\% CI. All rows remain significant.}
\end{table}

Harm does decrease monotonically with $p$ --- that trend is real --- but \textbf{it decreases slowly}: even at $p=0.75$ (the cache is still rewritten on three out of every four hits, already fairly close to the original baseline's write intensity), residual harm relative to $p=0$ has fallen by only about 35\% (from $+0.0065$ to $+0.0042$), and the effect never leaves the significant range anywhere across the tested interval $[0, 0.75]$. This is a more pessimistic but more honest conclusion than expected: \textbf{``occasionally rewrite popular cache entries'' as a mitigation for self-selection harm is far less of a bargain than it might seem --- fully offsetting the residual harm on a high-redundancy stream like LmArena may require $p$ very close to 1, i.e., essentially giving up the write-efficiency gain self-selection itself was supposed to provide}, rather than finding a ``small cost, large benefit'' sweet spot. Exactly where between $p=0.75$ and $1.0$ the effect actually stops being significant (if it does) was not tested further this round and remains an open question.

\textbf{Qualitative confirmation} (\texttt{scripts/\allowbreak crc\_\allowbreak closed\_\allowbreak loop\_\allowbreak spot\_\allowbreak check.py}). A statistical pattern alone doesn't confirm a mechanism, so every false-accept in \texttt{self\_\allowbreak select} (a gray-zone request the verifier approved but whose candidate was actually wrong) was audited case by case, across \textbf{all three datasets}. Method: search backward through the stream for that query's true match (an earlier record sharing the same equivalence\_id), and check whether that true match's own occurrence was ever written into \texttt{self\_\allowbreak select}'s store (i.e., whether it itself ever experienced a miss).

\begin{table}[h]
\centering
\small
\begin{tabular}{@{}lcccc@{}}
\toprule
\textbf{Dataset} & \textbf{direct-hit} & \textbf{total false-accepts} & \textbf{first occurrence} & \textbf{crowded-out share} \\
\midrule
LmArena & 72.8\% & 1{,}098 & 58 (5.3\%) & \textbf{68.9\%} (717/1{,}040) \\
SearchQueries (corr.) & 28.1\% & 3{,}667 & 2{,}081 (56.7\%) & 12.4\% (196/1{,}586) \\
Quora & 3.7\% & 432 & 353 (81.7\%) & 15.2\% (12/79) \\
\bottomrule
\end{tabular}
\caption{Crowding-out qualitative audit across all three datasets: crowded-out share is of explainable cases (those with at least one earlier equivalent record).}
\end{table}

On LmArena, of 1{,}098 post-warm-up false-accepts, 58 were genuine first occurrences (no earlier equivalent to compare against, unrelated to self-selection); of the remaining 1{,}040 explainable cases, 717 (\textbf{68.9\%}) fall under ``every earlier equivalent record was itself a hit, never independently written'' --- precisely matching the hypothesized mechanism: a correct candidate, by virtue of repeatedly being successfully reused, never gets its own chance to be written into the cache, so later queries of the same kind are forced to fall back on some topically-similar-but-not-actually-equivalent substitute --- one that happens to still fool the cross-encoder's score. The manually inspected examples confirm this is not a statistical coincidence: LmArena contains large recurring clusters of well-known ``viral'' trick-question templates --- the classic ``Alice has M sisters and N brothers'' sibling-counting riddle, the ``marble in an inverted glass placed in a microwave'' spatial-reasoning riddle --- the same puzzle rephrased a dozen-plus different ways. \texttt{self\_\allowbreak select}'s cache retains only a handful of canonical entries per cluster to absorb all future hits; once that happens, no fresh rephrasing ever earns its own cache entry, and later rephrasings are left to match whatever imperfect earlier entry HNSW happens to surface instead.

\textbf{This specific mechanism does not transfer directly to SearchQueries and Quora, which is the most important scope-narrowing finding of this round of auditing.} SearchQueries does show a significant overall self-selection harm (see the table above), but crowding-out explains only 12.4\% of its explainable cases, far below LmArena. More notably, 56.7\% of SearchQueries' false-accepts have no earlier equivalent record at all --- the large majority are simply the query's first occurrence, a verifier judgment problem in its own right, largely unrelated to the self-selection mechanism. Quora is similarly dominated by first occurrences (81.7\%), consistent with its own self-selection effect already being non-significant overall, and its sample of explainable cases (only 79) is too small to draw a strong conclusion from either way. \textbf{Conclusion: the specific ``a correct candidate gets crowded out of the cache by its own repeated successful reuse'' mechanism is mainly a phenomenon of high-redundancy traffic like LmArena's; the self-selection harm that genuinely exists on SearchQueries is mostly explained by a different mechanism this round did not identify} --- plausibly related to a systematic decline in HNSW approximate-retrieval recall quality as the overall candidate pool composition shifts, rather than any one specific candidate being targeted for crowding-out, but this is an unverified conjecture, not something this audit itself can answer.

\textbf{Limitations.} (1) Only $\alpha=0.02$ was tested, unlike Section~\ref{sec:crc} Protocol R/T's four-level sweep. (2) The warm-up window (1{,}500 gray-zone observations), recalibration cadence (every 500), and sliding-window size (most recent 2{,}000) are all modeling choices with no sensitivity analysis; the write-probability sweep itself reused the same three fixed values and did not test whether they would change how slowly this ``compensation'' decreases. (3) The write-probability sweep was only run on LmArena, not repeated on SearchQueries (which had already fully compensated at $p=0$, leaving no residual harm to sweep against); the sweep also only reached $p=0.75$ --- the true crossover point (if one exists) between 0.75 and 1.0 was not located further. (4) The finding that ``crowding-out explains most of LmArena's harm but not most of SearchQueries'' is currently only a result of elimination (this one mechanism was ruled out; what actually dominates SearchQueries' harm was not found or verified). (5) Like the rest of this paper, this remains trace-driven simulation, not a genuine online closed loop in a real production environment --- real systems' specific implementation details around ``does a hit trigger a fresh write'' (e.g., an independent eviction policy, TTLs) could further change the magnitude of effect measured here; the existing ``all conclusions rest on simulation'' limitation (Section~\ref{sec:limitations}) applies equally to this section.

\subsection{A cost-sensitive reanalysis: beyond the hit-rate/error-rate frontier, under what conditions is synchronous verification actually worth it}
\label{sec:costsensitive}

\textbf{Motivation.} Every result table elsewhere in this paper reports a hit-rate/error-rate Pareto frontier and leaves it to the reader to informally judge whether a given point is ``worth it'' --- a framing that implicitly treats every false-accept and every miss as equally costly, which is never true in a real deployment: a miss just costs one extra LLM call, while a false-accept costs customer trust, refunds, or support escalation --- usually a much larger and harder-to-quantify number, and the ratio between the two varies by business. Rather than continue reporting a frontier curve, this section treats the cost ratio $r = C_{\text{false\_accept}} / C_{\text{miss}}$ as a sweepable parameter, directly answering: over what range of $r$ is synchronous verification economically worth it?

\textbf{Method.} No new experiments are needed --- this reuses existing Group A (static threshold, 9 grid points) and Group D/E (honest calibration, Section~\ref{sec:groupd}, not the original grid-searched numbers) hit\_rate/error\_rate values. For each dataset:
\[
\mathrm{cost}(r) = r \times \mathrm{error\_rate} + (1 - \mathrm{hit\_rate}) \qquad \text{[in units of } C_{\text{miss}}\text{]}
\]
gives the expected cost of each operating point at a given $r$ (derivation: the higher hit\_rate, the more misses are saved; the higher error\_rate, the more false-accept cost is paid, both converted into the same currency, $C_{\text{miss}}$). For each group, the operating point minimizing $\mathrm{cost}(r)$ is that group's best performance at that $r$; comparing D/E's best against A's best determines which is economically superior. A fine log-spaced grid ($r \in [0.01, 1000]$, roughly 100 points per decade) is scanned to find every $r$-range where D/E wins --- a few representative $r$ values would not suffice, since ``which method wins'' as a function of $r$ is not necessarily monotonic (each group's best-achievable cost is the lower envelope of several linear functions of $r$, piecewise-linear and concave; the difference between two such curves is not guaranteed to cross zero only once, so a coarse grid could miss an intermediate winning window). Script: \texttt{scripts/\allowbreak cost\_\allowbreak sensitive\_\allowbreak reanalysis.py}; results: \texttt{results/\allowbreak cost\_\allowbreak sensitive\_\allowbreak reanalysis.json}.

\textbf{A grid-coverage issue the first run exposed, and how it was closed.} Group D/E's honest calibration (Section~\ref{sec:groupd}) had originally only been run at $\taulow \in \{0.80, ..., 0.95\}$ (6 points), while Group A's static-threshold grid covers up to 0.99 (9 points). The first run showed all three datasets ``losing back'' to A at large $r$ --- but exactly where this reversal occurred coincided with the range where D/E's grid has no operating point as conservative (as low-error) as A's $\tau=0.99$, leaving it unclear whether this was a genuine economic reversal or purely a grid-coverage artifact. To rule this out, Section~\ref{sec:groupd}'s honest-calibration scripts (\texttt{scripts/\allowbreak threshold\_\allowbreak calibration\_\allowbreak ablation.py}/\texttt{\_groupE.py}, both already supporting \texttt{--tau-high}/\texttt{--tau-low-grid} arguments --- no new code needed) were rerun at $\taulow \in \{0.97, 0.98, 0.99\}$, $\tauhigh=0.999$ (D and E, all 3 datasets, 6 runs total), and the 3 newly produced operating points were concatenated onto the original 6-point files, bringing D/E's grid to the same 9 points as Group A's. This step was anomalously slow on local CPU for a reason not fully diagnosed (the gray-zone candidate population in this $\taulow$ range is also much larger than the original range: LmArena 35{,}468 vs.\ the original 16{,}102, SearchQueries reaching 42{,}035) --- measured at roughly 3 ex/s, which would take hours per dataset; moving to a remote GPU server (Tesla T4) restored throughput to 80--113 ex/s, with all 6 jobs finishing sequentially in about 30 minutes.

\textbf{Results --- after closing the grid gap, the conclusion is stronger and cleaner than the first pass:}

\begin{table}[h]
\centering
\small
\begin{tabular}{@{}lll@{}}
\toprule
\textbf{Dataset} & \textbf{$r$-range where D wins} & \textbf{$r$-range where E wins} \\
\midrule
LmArena & \textbf{[0.71, $\infty$)} (tested to 1000) & \textbf{[0.28, $\infty$)} \\
Quora & [5.01, $\infty$) & [2.00, 2.82] $\cup$ [5.01, $\infty$) \\
SearchQueries (corr.) & [1.41, 3.98] $\cup$ [8.91, $\infty$) & [1.26, 4.47] $\cup$ [7.08, $\infty$) \\
\bottomrule
\end{tabular}
\caption{Cost-ratio ranges where D/E beat Group A's best static threshold on expected cost, after closing the grid-coverage gap.}
\end{table}

Once the grid was closed, the earlier ``A reverses back on top at large $r$'' phenomenon on LmArena \textbf{disappeared entirely} --- confirming it was a grid-coverage artifact, not a genuine economic reversal; after closing the gap, D/E win essentially across the entire tested range ($r\gtrsim0.3$--0.7 onward), no longer needing a qualified ``middle sweet spot'' framing. Quora (where D previously ``never won'') and SearchQueries (where the high-$r$ range previously appeared to lose) both win back at sufficiently large $r$ once the grid is closed, showing a two-window pattern (a narrower mid-range winning window plus an unbounded winning region at high $r$). All three datasets now share the same qualitative conclusion: \textbf{when the cost of an error clearly exceeds the cost of a miss (typically $r\gtrsim1$--9, dataset-dependent), synchronous verification is economically superior to a static threshold almost universally}; only when miss and false-accept costs are close, or misses are even more expensive (small $r$), is a static threshold aimed straight at maximum hit rate the better choice --- consistent with the naive intuition that ``the more expensive errors are, the more valuable a verification mechanism becomes,'' a direction the pre-grid-fix data had briefly appeared to overturn at the high-$r$ end, now confirmed to have been an artifact.

\textbf{Limitations.} (1) The cost ratio $r$ is a free parameter; this section never estimates a real $r$ value for any actual business scenario --- every range answers ``if $r$ falls in this range,'' not a claim about where any specific company's $r$ actually sits. (2) $\mathrm{cost}(r)$'s linear form assumes costs simply add, with no scale effects or nonlinear penalties (e.g., the trust collapse from accumulating multiple false-accepts is plausibly not linear) --- this is a simplifying model, not one derived from real business cost data. (3) Group C (oracle) appears in the table only as a reference ceiling, not included in the D/E-vs-A win/loss determination, since an oracle is not a deployable policy. (4) D/E honest calibration gives exactly one operating point per $\taulow$ (chosen via Youden's J on the calibration half), unlike Group A's per-threshold sweep --- this is an existing limitation of Section~\ref{sec:groupd}'s honest-calibration methodology itself, not something newly introduced here. (5) The newly added grid points ($\taulow=0.97$--0.99) used underlying score values from a different range in the local vs.\ GPU environment (the same phenomenon independently discovered and diagnosed in Section~\ref{sec:redteam} --- see that section for details), but since Youden's J calibration and testing both use scores from the same internally-consistent run, this does not affect the hit\_rate/error\_rate numbers this cost analysis actually depends on.

\subsection{LLM-driven automated red-teaming: how does the verifier perform on deliberately constructed hard cases}
\label{sec:redteam}

\textbf{Motivation.} Every error rate measured elsewhere in this paper comes from naturally-occurring near-duplicate pairs inside a benchmark dataset --- this can only reflect the verifier's average performance under whatever hard cases these specific datasets happen to contain, not its worst-case performance against query pairs deliberately constructed to fool it. Such hard cases had previously only appeared as anecdotes in Reddit discussions (e.g.\ the classic ``cancel''/``pause'' subscription example; see \texttt{RESEARCH\_\allowbreak PROPOSAL.md} \S10 direction 7), never systematically measured.

\textbf{Method.} DeepSeek (\texttt{deepseek-chat}) generates adversarial samples covering five specific failure axes already identified elsewhere in this project, rather than arbitrarily designed ones: negation (from this section's own motivation), action-verb swap (the ``cancel/pause'' paradigm from RESEARCH\_PROPOSAL direction 7), direction reversal (the ``same verb, reversed object direction'' mechanism found in Section~\ref{sec:bucketing}), named-entity swap (the ``H.W.\ Bush/W.\ Bush''-style hard case diagnosed in Section~\ref{sec:groupd}'s Quora honest-calibration analysis), and quantity/parameter swap. Each category generates a number of \texttt{(query\_a, query\_b, answer\_b)} triples, where \texttt{answer\_b} is a specific answer that holds only for \texttt{query\_b} --- wrong if treated as an answer to \texttt{query\_a}. After generation, \texttt{query\_a}/\texttt{query\_b} are encoded with the same live encoder this paper's Quora dataset uses (\texttt{all-\allowbreak MiniLM-\allowbreak L6-\allowbreak v2}), keeping only pairs whose cosine similarity clears the $\taulow=0.80$ threshold --- the necessary condition for a pair to actually reach the verifier in this paper's real pipeline. Qualifying pairs are scored with the same off-the-shelf cross-encoder (Group D) and Section~\ref{sec:groupe}'s LmArena fine-tuned checkpoint (Group E, loaded from Hugging Face \texttt{ChengyouXin/\allowbreak cacheverifier-\allowbreak lmarena}, the same checkpoint Section~\ref{sec:groupd}'s honest-calibration table uses). For statistical stability, 4 independent rounds were generated (20 per category each) and pooled, rather than trusting a single round --- a single round (15--20 per category) swung the same untuned verifier's overall false-accept rate from 58.2\% to 85.2\% between two independent trial runs, far too much variance to trust; pooling and reporting a parameter-free bootstrap CI on the aggregate fixes this. Script: \texttt{scripts/\allowbreak llm\_\allowbreak redteam\_\allowbreak verifier.py}; results: \texttt{results/\allowbreak llm\_\allowbreak redteam\_\allowbreak results.json}.

\textbf{A diagnosed environment issue that also fixed the comparison method.} Under the sentence-transformers version used for this experiment, the fine-tuned checkpoint's \texttt{predict()} returns scores compressed into $[0, 1]$ (suspected cause: this version fails to recognize the \texttt{activation\_fn} metadata saved by a newer version, silently defaulting to sigmoid) rather than the unbounded logits this paper otherwise assumes --- meaning the default threshold=0.0 decision rule is met almost unconditionally in $[0,1]$-probability space, not a meaningful comparison. The script now auto-probes each verifier's score range with two known-answer sanity pairs, and switches to the mathematically equivalent threshold=0.5 when a $[0,1]$-bounded range is detected (sigmoid is monotonic, so $\mathrm{logit}\geq0$ iff $\mathrm{sigmoid}(\mathrm{logit})\geq0.5$ --- this substitution only compensates for the loading-environment difference and does not change the actual decision criterion).

\textbf{Results} (4 rounds pooled, 306/400 pairs cleared $\taulow=0.80$):

\begin{table}[h]
\centering
\small
\begin{tabular}{@{}lccc@{}}
\toprule
\textbf{Category} & \textbf{Qualifying} & \textbf{Off-the-shelf FA\%} & \textbf{Fine-tuned (LmArena) FA\%} \\
\midrule
Negation & 79 & 96.2\% & 98.7\% \\
Quantity swap & 80 & 96.2\% & 90.0\% \\
Direction reversal & 76 & 86.8\% & 92.1\% \\
Named-entity swap & 39 & 48.7\% & 66.7\% \\
Action-verb swap & 32 & 59.4\% & 68.8\% \\
\midrule
\textbf{Total} & \textbf{306} & \textbf{84.0\%} [79.7,87.9] & \textbf{87.6\%} [83.7,91.2] \\
\bottomrule
\end{tabular}
\caption{Red-team false-accept rate by category, pooled across 4 generation rounds, 95\% bootstrap CI on the total.}
\end{table}

Two key findings: (1) The false-accept rate is far higher than anything measured on natural data elsewhere in this paper (Section~\ref{sec:groupd}'s worst natural error rates are single digits to low tens of percent), meaning natural-benchmark performance systematically overstates the verifier's robustness to real adversarial conditions; inspecting the highest-confidence false-accepts (e.g.\ ``can I use my gift card on sale items'' vs.\ ``...actually, can I not use it on sale items,'' scored 11.32 against a threshold of 0.0) confirms this is not borderline ambiguity --- the verifier confidently approves an answer that is the literal opposite of correct. (2) In-domain fine-tuning, proven in Section~\ref{sec:groupe} to substantially repair discriminative power on natural data, \textbf{provides essentially no protection on adversarial samples} --- the two 95\% confidence intervals nearly fully overlap (84.0\% vs.\ 87.6\%), meaning this fine-tuning recipe learns ``what this dataset's natural noise distribution looks like'' rather than a general robustness to deliberately constructed attacks --- two different capabilities that cannot substitute for one another. Negation is the most severe blind spot measured here --- consistent with the verifier's training objective (MS MARCO passage-relevance ranking, never specifically optimized for negation/polarity), not a coincidence.

\textbf{Limitations.} (1) The adversarial samples are LLM-generated, and no independent human verification confirmed that every sample's premise (that \texttt{query\_a} and \texttt{query\_b} genuinely require different answers) holds --- though the highest-confidence inspected cases are unambiguous, no systematic manual quality audit was performed. (2) Only $\taulow=0.80$ was tested; behavior at higher similarity thresholds is untested. (3) Only the LmArena fine-tuned checkpoint was tested; how SearchQueries'/Quora's fine-tuned checkpoints handle the same adversarial samples is unknown (though since LmArena is the most robust fine-tuning success story of the three datasets, SearchQueries/Quora's adversarial robustness is unlikely to be better). (4) Category sample sizes are uneven (39 for named-entity swap vs.\ 80 for quantity swap) --- per-category percentages carry wider uncertainty than the aggregate figure and shouldn't be over-interpreted for fine-grained cross-category ranking. (5) This round of red-teaming did not test the false-reject direction (rejecting a candidate that should have been approved) --- only whether the verifier becomes easier to fool, not whether adversarial inputs might also make it overly conservative.

\subsection{Can adversarial training close this gap}
\label{sec:advtrain}

\textbf{Motivation.} Section~\ref{sec:redteam} left the sharpest open question (item d): given that in-domain fine-tuning helps on natural noise but provides no protection against deliberately constructed adversarial samples, can training specifically targeted at these five failure axes close the gap? This is the most direct next step --- reusing the adversarial data already generated and the existing fine-tuning pipeline, at low cost.

\textbf{Method.} Independent of Section~\ref{sec:redteam}'s 306 held-out test samples, a fresh batch of adversarial training samples was generated separately using the same five failure-axis templates (223 triples, \texttt{results/\allowbreak llm\_\allowbreak redteam\_\allowbreak train\_\allowbreak pool.json}, non-overlapping with the held-out test set by construction --- two independent stochastic generations). Each triple becomes two training rows: (\texttt{query\_a}, \texttt{answer\_b}) $\to$ label 0 (the specific adversarial case Section~\ref{sec:redteam} showed both verifiers false-accept), (\texttt{query\_b}, \texttt{answer\_b}) $\to$ label 1 (the contrasting correct pairing). These 446 adversarial rows were merged directly with the 11{,}271 natural gray-zone training rows Section~\ref{sec:groupe}'s own LmArena fine-tuning used (adversarial data is 3.8\% of the combined set), and fine-tuned with the exact same training procedure (off-the-shelf base, 1 epoch, batch\_size=16, \texttt{scripts/\allowbreak finetune\_\allowbreak verifier\_\allowbreak train\_\allowbreak eval.py} reused unmodified) --- so the only variable between this run and Section~\ref{sec:groupe}'s original is the training DATA, not the method. Trained on a remote GPU (Tesla T4), 733 steps in 206.9 seconds. Evaluation: (a) AUC on Section~\ref{sec:groupe}'s original natural-data held-out test set (4{,}831 rows), to check whether natural-data discriminative power was sacrificed; (b) re-scoring the new model against Section~\ref{sec:redteam}'s \textbf{exact same} 306 held-out adversarial samples (\texttt{scripts/\allowbreak llm\_\allowbreak redteam\_\allowbreak verifier.py --load-triples}, skipping regeneration to guarantee the test set was never touched during this training).

\textbf{Results.} Natural-data AUC rose from 0.7212 (untuned baseline, matching Section~\ref{sec:groupe}'s originally reported baseline) to 0.8749, essentially matching the 0.88 Section~\ref{sec:groupe}'s pure-natural-data fine-tuning achieved --- adding 3.8\% adversarial training data \textbf{did not cost natural-data discriminative power}. False-accept rate on the adversarial held-out set:

\begin{table}[h]
\centering
\small
\begin{tabular}{@{}lcc@{}}
\toprule
\textbf{Verifier} & \textbf{Overall FA rate} & \textbf{95\% CI} \\
\midrule
Off-the-shelf (Sec.~\ref{sec:redteam}) & 84.0\% & [79.7\%, 87.9\%] \\
Naturally fine-tuned (Sec.~\ref{sec:redteam}) & 87.6\% & [83.7\%, 91.2\%] \\
\textbf{Adversarial + natural fine-tuned} & \textbf{53.6\%} & \textbf{[48.0\%, 59.2\%]} \\
\bottomrule
\end{tabular}
\caption{False-accept rate on Section~\ref{sec:redteam}'s 306-sample held-out adversarial set, before and after adding 3.8\% adversarial training data.}
\end{table}

Adversarial training cut the false-accept rate from 84--88\% to 53.6\%, with a confidence interval that does not overlap either of the other two --- \textbf{this is a real, statistically significant improvement, not noise}. By category, most axes improved substantially (negation 96.2\%$\to$57.0\%, action-verb swap 59.4\%$\to$12.5\%, quantity swap 96.2\%$\to$58.8\%, direction reversal 86.8\%$\to$57.9\%), but \textbf{named-entity swap is the one category that got worse} (48.7\%$\to$61.5\%) --- this category also had the fewest training triples of the five (28 of 223), and whether insufficient training volume or some cross-category negative transfer explains this has not been ruled out.

\textbf{This is not a ``problem solved'' result --- it is a ``substantially mitigated but far from eliminated'' result}: 53.6\% still means more than half of adversarial samples fool the verifier, a long way from the single-digit-to-low-tens error rates on natural data. But the direction is clear: \textbf{this capability gap is not unfixable --- a small (only 3.8\%), targeted amount of adversarial training on known failure axes produces a large improvement at no natural-data cost} --- a step beyond Section~\ref{sec:redteam}'s diagnosis that ``fine-tuning learns the natural noise distribution, not adversarial robustness'': given the right data in the training set, the same fine-tuning mechanism can in fact learn some adversarial robustness too --- this is not a fundamental limitation of the mechanism itself.

\textbf{Limitations.} (1) Only one adversarial data proportion (3.8\%) was tested; whether a higher proportion closes the gap further, or starts costing natural-data performance, was not tested. (2) The named-entity-swap regression was not further diagnosed --- unclear whether it's insufficient sample size or cross-category negative transfer, and whether simply feeding this category more training data would fix it was not tested. (3) Training and evaluation data, while two independent, non-overlapping generations, both come from the same LLM (DeepSeek) and the same family of prompt templates --- whether the robustness learned here generalizes to adversarial samples from a different generator was not tested, so this could reflect learning this one generator's particular style rather than true cross-source generalization. (4) Like Section~\ref{sec:redteam}, only LmArena was tested. (5) The 53.6\% figure was not compared against a simpler, more conservative default strategy (e.g., raising the static threshold) --- a sufficiently conservative static threshold could in principle also suppress the false-accept rate, at the cost of hit rate, and this trade-off in the adversarial setting specifically was not measured.

\section{Discussion}

\subsection{Returning to Go/No-Go}
\label{sec:golo}

This study was designed against a Go/No-Go criterion fixed before any data was collected, specifically to prevent the temptation to rationalize whatever the results turned out to be. \textbf{Go} requires that Group D achieve a statistically significant hit-rate improvement over Group A at a matched error rate (non-overlapping bootstrap confidence intervals), demonstrated by at least one verifier --- a result that must not depend on a single fortunate choice of hyperparameters. \textbf{No-Go} would mean the four groups' Pareto frontiers coincide within their confidence intervals, or that Group D underperforms Group C at matched error rate by an unacceptable margin (i.e.\ that synchronicity itself, not verifier quality, is what erodes the benefit).

\textbf{Result}: on LmArena, the cross-encoder does achieve a statistically significant hit-rate improvement at its best operating point (+1.9pp, non-overlapping confidence intervals). This literally clears the Go bar.

\textbf{But} this is a \textbf{weak Go}, for three reasons:

\begin{enumerate}[leftmargin=*]
  \item The lead is less than one-tenth of the oracle ceiling --- the real verifier is far from cashing in the mechanism's potential.
  \item 11 of 30 tested points were worse --- hit rate and error rate are highly sensitive to $(\taulow, \text{threshold})$, and there is no robust ``set it and it wins'' region; a real deployment would need to grid-search this per use case, which is itself a non-trivial operational cost.
  \item On SearchQueries, the same verifier is \textbf{net harmful} (corrected, see erratum above; 23 of 36 operating points lost to the static-threshold frontier) --- not bad luck, but it can no longer be attributed to a single cause the way it originally was: this paper first attributed the SearchQueries failure entirely to distribution shift (the ms-marco cross-encoder trained on long web-search query/passage relevance, meeting SearchQueries' extremely short keyword queries), but subsequently found the dataset's \texttt{answer} field itself had a data defect (never populated upstream, a constant placeholder), which by itself is sufficient to explain a large fraction of the originally-reported failure. The verifier still underperforms on the corrected data (net harmful, not near-chance), so ``short, out-of-distribution queries'' is a real contributing difficulty that has not been disproven --- but it is no longer the sole or primary explanation, and its magnitude is smaller than originally reported. Worth recording as a lesson: rule out evaluation-data integrity issues before diagnosing \emph{why} a verifier fails on a given dataset. \textbf{[2026-08-15 update]} This reason itself needs a further discount: the ``net harmful'' verdict above uses a post-hoc grid-search threshold, and Section~\ref{sec:groupd}'s honest-calibration result (a threshold fixed without looking at the outcome: chronological calibration/test split + Youden's J) shows the same verifier beating the static threshold at every tested point instead. How much of SearchQueries' failure is attributable to genuinely insufficient verifier discriminative power versus how the evaluation happened to pick its threshold is now the least settled of these three reasons; see the end of Section~\ref{sec:groupd}.
\end{enumerate}

So this paper's verdict is neither ``synchronous verification gating is worth doing'' nor ``it is not worth doing,'' but: \textbf{the mechanism has real, often substantial theoretical headroom (proven by the oracle results), but whether that headroom converts into a net production benefit depends entirely on how well the verifier matches the business data domain} --- a specific, actionable conclusion the prior literature (Krites in particular, since its evaluation never used a real verifier) has not provided. Section~\ref{sec:groupe} tests whether ``matching the domain'' is something a practitioner can actually go do, rather than just a diagnosis: it is --- fine-tuning the same verifier on in-domain gray-zone labels turns SearchQueries specifically, the dataset behind reason 3 above, from a net-harmful result into one that beats the static-threshold baseline at nearly every tested point.

\textbf{External cross-validation.} Reason 3 above --- that an off-the-shelf ms-marco cross-encoder fails on SearchQueries --- is not an isolated observation from this paper alone, though this paper's own evidence for it now needs a caveat (see the erratum above: the original ``near-chance'' evidence was a data-defect artifact; the corrected ``net harmful'' evidence is real but weaker as a data point than originally presented). An independent, contemporaneous work from Redis and NYU~\cite{calibrationgap} studies exactly the gap between offline ranking quality and deployed precision in semantic caching, using different datasets and a different evaluation framework from this paper's, and proposes a Calibration Retention Rate (CRR = the area under the deployed precision--cache-utilization curve, divided by offline PR-AUC) to quantify it. Their reranker comparison includes ms-marco-MiniLM-L12-v2 --- \textbf{not the model this paper's Groups C/D use} (this paper uses the smaller ms-marco-MiniLM-L6-v2 throughout; see Section~\ref{sec:verifiers}), but a different checkpoint from the same MS MARCO passage-ranking lineage, trained by the same team, at a different size (12 layers vs.\ this paper's 6): it retains a CRR of only 0.427 --- less than half of its offline ranking quality survives deployment --- making it one of the more severely miscalibrated models in their evaluation set. That result comes from entirely different datasets (their PAWS/QQP/STS-B-style general paraphrase corpora, not this paper's LmArena/SearchQueries/Quora), an entirely different evaluation framework (a calibration curve at a fixed threshold, not this paper's hit-rate/error-rate Pareto frontier), and a different-sized model from the same family --- and yet points in the same direction as Section~\ref{sec:groupd}'s corrected finding that this paper's own off-the-shelf verifier is net harmful, not merely inert, on SearchQueries: \textbf{a generic, off-the-shelf ms-marco cross-encoder, at either size, can look reasonable on offline ranking metrics and still lose much of its reliability once it is forced to make a binary decision at a fixed operating point in a deployed system.} Two different-sized models independently reproducing the same failure mode is, if anything, more informative than the same checkpoint failing twice would be --- it points at the training objective itself rather than an accident of one specific set of weights. The two studies were conducted independently, share no data or code, and still converge on the same model family --- which provides some support, from outside this paper's own experimental design, for the negative premise behind the weak-Go verdict (that off-the-shelf verifiers are not trustworthy as-is) --- but given that this paper's own evidence went through a correction, this cross-validation should now be read as directional corroboration rather than two strong, mutually-reinforcing results. The difference is that Baral et al.\ stop at diagnosing the gap, whereas Section~\ref{sec:groupe} goes on to test and validate a remedy (in-domain fine-tuning). It is also worth noting that their work calibrates the single similarity score already produced by the retrieval/reranking stage (a one-signal architecture), rather than adding a second, decoupled verification stage independent of retrieval as this paper does (a two-signal architecture) --- so the two results corroborate each other without one substituting for the other. \textbf{[2026-08-15 update]} Section~\ref{sec:groupd}'s honest-calibration result further weakens this external corroboration's weight: the ``net harmful'' verdict on SearchQueries was found to be substantially a product of post-hoc grid search --- under honest calibration, the same untuned verifier beats the static threshold at every tested point. This does not mean Baral et al.'s diagnosis is wrong (their P-CHR AUC/CRR is a threshold-swept summary metric, not sensitive to a single grid-search-optimism bias the way a single reported operating point is), but it does mean this paper's own ``net harmful'' evidence can no longer be treated as equally strong independent support.

\textbf{Turning their own metrics back on this paper's results.} Since Baral et al.\ propose P-CHR AUC and CRR as their metrics, applying their exact formulas (P-CHR AUC = area under the Precision-CHR curve; PR-AUC = area under the standard Precision-Recall curve; CRR = P-CHR AUC / PR-AUC) directly to this paper's own Group D/E score data is a natural reverse cross-methodology check: SearchQueries Group D CRR=0.979, Group E CRR=0.924; LmArena Group D CRR=0.982; Quora Group D CRR=0.981 --- all far above their reported ms-marco-MiniLM-L12-v2 CRR of 0.427. This gap \textbf{should not be read as ``this paper's verifiers are far better calibrated than their model''} --- the more likely explanation is architectural. Baral et al.\ evaluate a two-stage pipeline (retrieve $K{=}50$ candidates, rerank, take the top-reranked one); CRR is specifically designed to catch the failure mode where the true candidate scores well offline but doesn't survive reranking into first place. This paper's architecture is single-tier: similarity search already commits to exactly one candidate before the verifier runs; there is no multi-candidate reranking competition at all, and the \texttt{would\_\allowbreak be\_\allowbreak correct} label is already defined against that single candidate. The specific miscalibration CRR is built to detect structurally cannot arise in this paper's setting, which is consistent with CRR staying pinned above 0.92 across all three datasets and both Groups D and E almost independent of PR-AUC itself (Quora's PR-AUC is only 0.480, far below LmArena's 0.938, yet the two CRR values are nearly identical). This reverse check does not support ``this paper's method is better by the CRR measure'' so much as it surfaces a more useful methodological boundary: CRR is designed for a retrieve-then-rerank architecture, and applying it to a single-tier verification architecture structurally compresses the metric toward its ceiling --- the absolute values are not directly comparable across architectures; see \texttt{RESEARCH\_\allowbreak PROPOSAL.md} \S10 direction 2 for details.

\subsection{Relation to Krites' claim that synchronicity erodes benefit}

Krites predicts that synchronous verification ``will increase latency and erode the cache's benefit,'' but never implements or measures it. This paper's Group C results (oracle, synchronous) show that, given a sufficiently accurate verifier, synchronicity itself does not erode benefit --- it delivers a large hit-rate improvement, at the cost of a quantifiable latency addition (Section~\ref{sec:latency}). The real risk is not the decision to go synchronous per se, but verifier inaccuracy --- a distinction worth drawing explicitly against Krites' prediction: the latency cost is real, while verifier quality is the dominant variable determining whether the net effect is positive or negative --- a variable invisible in Krites' oracle-only evaluation. \textbf{[2026-08-17 update]} The claim ``the latency cost is acceptable (tens of milliseconds)'' needs revising: the real API-latency measurement reported at the end of Section~\ref{sec:latency} (using DeepSeek as a stand-in for the inaccessible GPT-4.1-nano) shows the real API-class judge's mean latency (1687.8ms) is about 24x the paper's original modeling assumption (70ms); substituted into the highest-call-rate operating points, expected added latency moves from tens of milliseconds to hundreds of milliseconds or nearly a second (about 970.5ms at SearchQueries' $\taulow=0.80$). This does not change the qualitative conclusion that ``synchronicity itself doesn't erode hit-rate benefit'' (hit rate/error rate are independent of the latency value), but it does overturn the quantitative claim that the cost is negligible --- for latency-sensitive deployments, this cost should now be taken seriously rather than assumed acceptable; whether it actually is acceptable depends on the specific business's latency tolerance, which this paper cannot settle in general.

\section{Limitations}
\label{sec:limitations}

\begin{itemize}[leftmargin=*]
  \item \textbf{The external cross-validation cited in Section~\ref{sec:golo} (Baral et al., 2026)~\cite{calibrationgap} is directional agreement, not a numerically comparable result, and not the same model.} The model they evaluate, ms-marco-MiniLM-L12-v2, and the one this paper uses, ms-marco-MiniLM-L6-v2 (Section~\ref{sec:verifiers}), are different-sized checkpoints from the same family and are not interchangeable. Their CRR metric is also defined differently from this paper's false-accept/false-reject rates, their evaluation datasets are different (general-purpose paraphrase corpora vs.\ this paper's conversational/search/paraphrase trio), and their target task is different (calibrating the retrieval/reranking stage, not verifying cache hits specifically). It should be read only as corroborating evidence for the qualitative claim that off-the-shelf ms-marco-family models are broadly unreliable once deployed, not as a quantitative benchmark this paper's own measurements can be checked against or substituted with.
  \item \textbf{[2026-08-20 update, partially retested] Only one real verifier architecture was tested} (Section~\ref{sec:groupe}'s Group E fine-tunes the same cross-encoder on each dataset rather than trying a different architecture). \textbf{Section~\ref{sec:reranker} has since retested the ``swap in a larger/more broadly-trained off-the-shelf reranker'' dimension} (\texttt{ms-marco-MiniLM-L12-v2}, same distribution, pure capacity increase; \texttt{BAAI/\allowbreak bge-reranker-base}, broader training distribution and larger): on the one dataset that actually separates the two variables (SearchQueries), both produce nearly identical gains, and both fall far short of fine-tuning --- so ``off-the-shelf model selection, in either the capacity or the training-distribution direction'' is now empirically ruled out, not just an open guess. But that only covers one axis --- swapping size within the cross-encoder/reranker family. The research plan's originally proposed NLI model, mini-LLM (generative rather than scoring-based discrimination), and hosted API model remain untested architecture-level alternatives; whether a model built specifically for short text behaves qualitatively differently is still an open question, though Section~\ref{sec:reranker}'s result makes in-domain fine-tuning, not architecture swapping, look like the primary lever.
  \item \textbf{[2026-08-15/16, Group D and Group E both now tested, see the conclusion at the end of Section~\ref{sec:groupd}] The verifier's score threshold grid was calibrated after the fact, per dataset and per model, not fixed in advance on a held-out set} --- true of Group D (see the SearchQueries failure case in Section~\ref{sec:groupd}) and equally true of Group E's fine-tuned models (Section~\ref{sec:groupe}). This paper states the limitation explicitly rather than concealing it. Group D has now been retested with honest calibration (chronological calibration/test split + Youden's J): the original ``net harmful'' verdict on SearchQueries reverses (6/6 tested points win), LmArena's result is essentially unchanged, and Quora gets worse --- honest calibration is not a uniformly helpful or uniformly harmful correction, its effect depends on how separable the verifier's off-the-shelf scores actually are on that domain; see the end of Section~\ref{sec:groupd} for details. \textbf{Group E (fine-tuned) has now been given the same treatment (2026-08-16)}: zero losses hold on all three datasets under honest calibration, LmArena/SearchQueries's lead actually grows (+5.66pp/+7.74pp, both exceeding the original grid-search numbers), and Quora narrows substantially (+2.03pp$\to$+0.43pp) without reversing direction --- Section~\ref{sec:groupe}'s reported margins broadly survive this check and this is no longer an open gap; full numbers and methodology notes are in the new table at the end of Section~\ref{sec:groupd}.
  \item \textbf{Quora's equivalence classes are a reconstruction, not native ground truth.} Unlike LmArena/SearchQueries (whose \texttt{ID\_\allowbreak Set}/\texttt{id\_\allowbreak set} labels come from the original benchmark), Quora's classes were built by this paper via union-find over pairwise duplicate labels, and its \texttt{answer} field is a stand-in (a matched question's own text) rather than an LLM-generated response --- a reasonable substitution given no generated answers exist for this dataset (Section~\ref{sec:groupe}), but not an apples-to-apples replication of the other two datasets' construction.
  \item \textbf{[2026-08-17, now tested, see Section~\ref{sec:tauhigh}] \tauhigh\ was previously fixed at a single value (0.97), with only \taulow\ swept.} Section~\ref{sec:tauhigh} now sweeps \tauhigh\ too: the conclusion does change, and its direction varies by dataset --- on LmArena, raising \tauhigh\ to 0.99 grows the best net lead from +1.42pp to +5.18pp; the same change on SearchQueries (corrected) instead flips the net lead from positive to negative ($-0.11$pp); Quora is essentially unaffected. This is not a limitation that has been eliminated so much as it has turned ``should \tauhigh\ be tunable'' into an open parameter that, like verifier selection itself, needs domain-specific calibration --- see Section~\ref{sec:tauhigh} for the mechanism.
  \item \textbf{[2026-08-17, now measured, see the end of Section~\ref{sec:latency}] The oracle's 70ms latency was previously a modeling assumption, not a measurement.} A real measurement using DeepSeek (no GPT-4.1-nano access, substituted as a same-class API-hosted judge model) over 150 calls found a mean latency of 1687.8ms --- about 24x the modeling assumption --- materially weakening the ``latency cost is acceptable'' claim in Section~\ref{sec:golo}; details in Sections~\ref{sec:latency}/\ref{sec:golo}. This measurement has its own limits: DeepSeek and GPT-4.1-nano are different models on different infrastructure, so the number reported is an order of magnitude, not a precise value; see Section~\ref{sec:latency}.
  \item \textbf{Group D's latency figures on LmArena are distorted by four-way parallel CPU contention} and should not be taken as a reliable estimate of that verifier's latency; the figure measured independently on SearchQueries is more trustworthy.
  \item \textbf{This work does not cover the problem Krites solves} (promoting curated static-tier answers into a dynamic tier) --- the two are complementary, not substitutes.
  \item \textbf{[2026-08-17, one specific discrepancy found and fixed, see the end of Section~\ref{sec:groupb}] Group B's port was checked line-by-line against vCache's source but had not been directly numerically cross-validated against the official implementation.} A line-by-line check against a clone of the official repository confirmed the formulas match exactly, but found that the official \texttt{EmbeddingMetadataObj} pre-seeds every cache entry with two synthetic bootstrap observations this paper's port lacked, meaning the official implementation only needs 4 real observations (versus this paper's prior 6) to clear the cold-start gate. After fixing it, hit rate rose 4.4x--29.1x across all three datasets while error rate stayed below the target $\delta$ throughout (the formal guarantee was unaffected); see the end of Section~\ref{sec:groupb}. This is no longer ``cannot rule out subtle porting discrepancies'' --- it is a specific, located, and fixed one; the originally-reported hit-rate ranges in Section~\ref{sec:groupb}'s table should be read as this paper's prior port's numbers, not the official vCache algorithm's true ceiling.
  \item \textbf{The Group C/D comparison against Group A relies on linearly interpolating Group A's frontier between its nine tested grid points} (Section~\ref{sec:groupc}), rather than against a continuously re-tuned static threshold. This is a reasonable approximation given the observed curve is smooth and monotonic, but it is an approximation: if Group A's true achievable frontier deviates from a straight line between adjacent tested points --- convex or concave in a way the coarse grid does not reveal --- the reported margins for Group C and D could be modestly over- or understated.
  \item All correctness judgments rely on the dataset's own ground-truth equivalence-class labels; label quality is upper-bounded by the original benchmark (the vCache paper).
  \item \textbf{Section~\ref{sec:deployment}'s label noise is synthetic (uniform random flips), not sampled from a real feedback channel} (e.g.\ user thumbs-up/down), which may have a different, possibly systematic rather than random, error structure. Its ``8 sequential chunks'' also simulate temporal drift by slicing an already-static dataset rather than replaying genuinely continuous, rolling production traffic; real deployments would need to validate the specific noise threshold and re-tuning cadence against their own feedback channel and traffic pattern rather than assuming these exact figures transfer.
  \item \textbf{Section~\ref{sec:twitter}'s equivalence classes are an embedding-similarity clustering proxy, not human-labeled ground truth}, and the 0.92 cosine-similarity threshold is itself an untuned hyperparameter; an earlier, stricter (exact-string-match) version of the same construction produced a materially different --- and, on inspection, clearly miscalibrated --- false-accept rate, so the reported absolute numbers should be read as indicative of a real phenomenon (the comcastcares base-rate instability) rather than precise. The dataset is also 2014--2017 vintage, predating widespread AI-chatbot-mediated customer support, which may not reflect current support-ticket dynamics, and its CC0 license permits this research use but the source explicitly requests contact for commercial use.
  \item \textbf{Section~\ref{sec:monitor}'s monitor was validated retrospectively on a static held-out split, on exactly two datasets (one with the failure mode, one without), not deployed against a genuinely streaming production system.} Its detection thresholds ($\alpha$ for the z-test; $\delta,\lambda$ for Page-Hinkley) were chosen once and not tuned or validated beyond this single comparison; a larger, more diverse set of real deployments would be needed before treating these specific parameter values as generally applicable.
  \item \textbf{Section~\ref{sec:bucketing}'s verb-extraction quality was not independently validated against human annotation.} The spaCy dependency parser can make systematic errors (e.g.\ mis-tagging the true action-bearing verb in a subordinate clause --- Section~\ref{sec:bucketing} gives a concrete example), and this paper only spot-checked a handful of cases rather than running a systematic accuracy evaluation of the extractor itself; this means how much of Section~\ref{sec:bucketing}'s negative result traces to ``action-verb bucketing has a structural blind spot'' versus ``this particular extractor implementation wasn't good enough'' cannot be fully separated --- though LmArena's regression is fully explained by a mechanism independent of extractor quality (same verb, opposite object direction), so this uncertainty more plausibly affects magnitude than direction. Section~\ref{sec:bucketing} also tested only the simplest action-verb-alone keying scheme, not the (action, object) pair variant that might close that blind spot.
  \item \textbf{[2026-08-19, see Section~\ref{sec:crc}] CRC was only tested at $\taulow=0.80$ (the widest grid value), not swept across the full grid the way Section~\ref{sec:tauhigh} swept $\tauhigh$.} That choice avoided reintroducing the ``post-hoc grid search'' problem Section~\ref{sec:tauhigh} itself flagged, not because that value is optimal in any sense; whether gray-zone composition, and therefore the steepness of $R_{GZ}(\lambda)$, differs meaningfully across $\taulow$ values remains untested.
  \item \textbf{[2026-08-19, see Section~\ref{sec:crc}] Protocol T (chronological split) was run once per dataset, not repeated.} Unlike Protocol R's 200 independent repeats, there is only one real chronological order to draw from; the reported overshoot magnitudes are observations at this one split point (50/50), and whether they'd differ at a different split point (e.g.\ 60/40, or a sliding window) was not tested --- though the directional conclusions (systematic overshoot on Quora, conservatism on LmArena) are unlikely to be an artifact of split-point choice alone, given they match the decile-level drift directions measured independently in the state-dependence audit.
  \item \textbf{[2026-08-22, see Section~\ref{sec:topk}] The Top-K cascade was only tested at K=2, and only with the cross-encoder verifier family.} The Phase 0 ceiling shows a larger K still carries diminishing but non-zero marginal returns; whether that is worth the added verifier calls (worst-case latency stacks with K) is an untested open question. The ``K and fine-tuning are complementary'' finding rests mainly on Quora and SearchQueries, the two datasets with a finer threshold grid --- on LmArena, that particular comparison method (fix the error rate, find the best hit\_rate) is less reliable than the matched-point-plus-CI test used as this section's primary evidence.
  \item \textbf{[2026-08-19, see Section~\ref{sec:crc}] Protocol R's formal guarantee covers resampling uniformly from this same fixed, already-collected benchmark pool --- not a guarantee about real future production traffic.} This is not a flaw in this section's implementation; it is the honest scope split conformal/CRC methods can claim. Extending the guarantee to real production traffic would require that traffic to be exchangeable with this fixed benchmark pool, and Protocol T's Quora result is a concrete counter-example showing that exchangeability can fail even within one benchmark purely due to chronological ordering --- let alone when extrapolating to genuinely unseen future traffic.
  \item \textbf{[2026-08-28/29, see Section~\ref{sec:selfselect}] The closed-loop self-selection test was only run at $\alpha=0.02$, and the warm-up window / recalibration cadence / sliding-window size were not sensitivity-tested.} The write-probability sweep (2026-08-29) that locates the ``does recalibration fully compensate'' threshold was only run on LmArena, and only up to $p=0.75$ (residual harm never left the significant range across that interval; the true crossover point --- if one exists --- between 0.75 and 1.0 was not located further). The qualitative audit was extended to all three datasets, but the conclusion that ``the crowding-out mechanism doesn't explain most of SearchQueries' harm'' is only a result of elimination --- what actually dominates SearchQueries' self-selection harm remains an open question. Like the rest of this paper, this remains trace-driven simulation, not a genuine online closed loop in a real production environment.
  \item \textbf{[2026-08-30, see Section~\ref{sec:costsensitive}] The cost-sensitive reanalysis' cost ratio $r$ is a free parameter --- no real business scenario's actual $r$ value was estimated, and the linear-additive $\mathrm{cost}(r)$ model was never validated against real cost structure (e.g., the trust cost of accumulating multiple errors is plausibly nonlinear).} This is a framework for turning ``is it worth it'' from a qualitative judgment into a computable function, not a tool that directly tells you which verifier your specific scenario should use.
  \item \textbf{[2026-08-30, see Section~\ref{sec:redteam}] The LLM red-team's adversarial samples were not independently human-verified sample by sample for the premise that the two queries genuinely require different answers; only $\taulow=0.80$ was tested; only the LmArena fine-tuned checkpoint was tested; and whether adversarial samples might also make the verifier overly conservative (the opposite direction --- more false-rejects) was not tested (only whether it becomes easier to fool).} This round confirms fine-tuning's repair of natural-noise discriminative power does not automatically transfer to adversarial robustness, but does not go on to test the natural next question of whether adversarial training targeted at these five failure axes would help.
  \item \textbf{[2026-08-30, see Section~\ref{sec:advtrain}] Adversarial training only tested one data proportion (3.8\%); the named-entity-swap regression was not further diagnosed; training and evaluation data both came from the same LLM (DeepSeek) and the same prompt-template family, with no test of generalization across generator sources.} 53.6\% residual false-accept rate proves this direction works but is far from solving the problem, and it was not compared against a simpler conservative strategy like raising the static threshold.
\end{itemize}

\section{Conclusion and Future Work}
\label{sec:conclusion}

Under a single-tier semantic cache architecture, putting a verifier synchronously on the serving path can, in principle, raise hit rate by 20--28 percentage points at matched error rate --- this headroom is real, not a theoretical assumption. An off-the-shelf cross-encoder, uncalibrated for the target domain, cashes in only a small (and fragile) slice of that headroom on one dataset, and is actively net harmful on the other (see the erratum note above) --- this remains the paper's core Go/No-Go finding (Section~\ref{sec:golo}), obtained honestly before any remediation was attempted. \textbf{[2026-08-15 update]} But this verdict itself depends on a post-hoc grid-searched threshold; with an honestly-calibrated threshold instead, the same untuned verifier on SearchQueries beats the static-threshold frontier at every tested point (Section~\ref{sec:groupd}). ``Net harmful'' is not a stable fact independent of evaluation method --- it depends heavily on how the threshold was chosen, and this is one of this paper's most consequential methodological lessons. But Section~\ref{sec:groupe}'s follow-up shows that gap is not fundamental to the mechanism: fine-tuning the same verifier on in-domain gray-zone labels --- data the online system already generates for free --- converts LmArena's fragile result into a strictly dominant one, and, more importantly, converts SearchQueries' net-harmful result into a verifier that reliably beats the static-threshold baseline. Section~\ref{sec:deployment} further shows this fine-tuning recipe is deployment-robust in the three dimensions that determine whether it survives contact with a real system: it tolerates realistic amounts of feedback noise (up to $\sim$30\%) before turning harmful, needs no more than a few hundred to a thousand in-domain examples before it reliably helps, and does not exhibit continued decay as traffic drifts away from its training window --- so a one-time fine-tune is not fragile to the ordinary passage of time. Section~\ref{sec:twitter}, however, shows this robustness is not unconditional: on real, multi-year production traffic from one brand, the same three risks all materialize together, traced to a single, monitorable cause (a drifting gray-zone base rate) rather than three independent failures --- a genuine boundary condition on Section~\ref{sec:deployment}'s claims that only long-horizon real traffic, not any of the three benchmark datasets, was able to expose. Section~\ref{sec:monitor} shows that cause is not just identifiable after the fact but detectable in advance: two lightweight change-point tests on gray-zone labels alone flag comcastcares' instability partway through the affected window while staying silent on the unaffected brand, at no additional inference cost. \textbf{This study's verdict is a weak Go, with a demonstrated, low-cost, and largely (but not unconditionally) deployment-robust path to a stronger one, including a concrete first answer to the one open question that path left}: the mechanism is worth continued investment, and the next priority is not ``should we do synchronous verification'' but ``how do we keep a verifier calibrated to a specific, possibly drifting, business domain'' --- a question Sections~\ref{sec:groupe}--\ref{sec:monitor} now show has a tractable answer along its full length, from fine-tuning recipe to drift detection, though each link in that chain (particularly Section~\ref{sec:monitor}'s prototype) still wants validation on more than a handful of datasets before being treated as settled.

Future work, in priority order:

\begin{enumerate}[leftmargin=*]
  \item \sout{Build an online monitor for gray-zone positive-rate stability (e.g.\ a sequential change-point test over a rolling window) as a data-driven re-tuning trigger, and test it against comcastcares' own drift to see whether it would have caught the instability before it degraded Group E's benefit.} \textbf{Prototyped and validated (Section~\ref{sec:monitor}):} a chunked two-proportion z-test and a Page-Hinkley sequential test, both run on gray-zone labels alone, flag comcastcares' positive-rate shift partway through its held-out window and never false-alarm on AmazonHelp. The prototype is retrospective and tested on only two datasets; validating it as a genuinely online monitor against a live, continuously rolling deployment (not a static held-out split) is the natural next step.
  \item \sout{Evaluate NLI models and mini-LLMs (e.g.\ Qwen2.5-0.5B/1.5B) as verifiers, to see whether either type is more robust to short-query settings like SearchQueries.} \textbf{Addressed by a different route (Section~\ref{sec:groupe}):} rather than swapping architectures, fine-tuning the \emph{same} cross-encoder on in-domain gray-zone examples raised held-out AUC from 0.72 to 0.88 on LmArena and from 0.5984 to 0.7120 on SearchQueries (corrected, see erratum above), turning a net-harmful result into one that beats the static-threshold frontier at 53/54 tested points, 1 tie, zero losses --- and a third, independently-sourced dataset (Quora Question Pairs) replicated the same near-zero-loss pattern at a smaller magnitude. \sout{Whether a different architecture (e.g.\ one built for short text) could close SearchQueries'/Quora's remaining AUC gap to LmArena's further is now the more precise open question.} \textbf{Tested (Section~\ref{sec:reranker})}: \texttt{ms-marco-MiniLM-L12-v2} (same distribution, pure capacity increase) and \texttt{BAAI/\allowbreak bge-reranker-base} (broader training distribution, also larger) produce nearly identical gains (+0.0226 vs.\ +0.0214) on the one dataset that separates the two variables (SearchQueries), both far short of the 0.7120 fine-tuning reaches --- off-the-shelf model selection, in either direction of ``reinforcement,'' doesn't meaningfully close the gap; fine-tuning (Section~\ref{sec:groupe}) remains the only remedy verified to work.
  \item \textbf{[Added 2026-08-09, already tested]} This paper briefly suspected the SearchQueries failure was rooted in an ``axis problem'' (confusable action words, not query length itself --- see \texttt{RESEARCH\_\allowbreak PROPOSAL.md} \S10 direction 5) --- a controlled experiment failed to confirm that hypothesis (the effect even reversed sign on the verifier signal). The actual root cause turned out to be the data defect documented in the erratum above: a constant placeholder answer, not a semantic/embedding phenomenon. After correcting that defect, the verifier still underperforms on SearchQueries (net harmful, not near-chance), so the more basic ``short queries are intrinsically harder'' explanation has not been fully disproven, just demoted from the sole explanation --- how much of the corrected residual harm is attributable to query length itself versus other, still-unidentified factors is a new open question.
  \item \sout{Test how Group E's fine-tuned verifier degrades as production traffic drifts away from its fine-tuning window.} \textbf{Addressed (Sections~\ref{sec:deployment}--\ref{sec:twitter}):} across LmArena/SearchQueries/Quora/AmazonHelp, the fine-tuned verifier's relative advantage over the untuned baseline does not continue to decay beyond an initial same-window-to-next-window drop. But comcastcares is a genuine counter-example (Section~\ref{sec:twitter}): continued decay, traced to a drifting gray-zone positive rate rather than a general failure of the mechanism. Re-tuning cadence should be driven by monitored shifts in both gray-zone data composition and positive-rate stability, not a fixed calendar schedule.
  \item \sout{Calibrate verifier thresholds on an independent held-out set, rather than probing the score distribution via the formal sweep itself --- this applies to Group E's fine-tuned models exactly as it does to Group D.} \textbf{[2026-08-20 backfilled; actually resolved 2026-08-15/16]} The honest calibration methodology (chronological calibration/test split, Youden's J threshold selection) was retested and written into the end of Section~\ref{sec:groupd}, covering both Group D and Group E (see that section for details); the CRC (Section~\ref{sec:crc}) and E' (Section~\ref{sec:rewrite}) sections added since then also directly reuse the same calibration/test split rather than probing the formal sweep itself. This list entry was simply never marked resolved --- a documentation gap, not an open methodological gap.
  \item \sout{Include \tauhigh\ in the swept grid rather than fixing it at a single anchor.} \textbf{Tested (Section~\ref{sec:tauhigh})}: the conclusion is dataset-dependent --- widening \tauhigh\ to 0.99 grows LmArena's best net lead from +1.42pp to +5.18pp, while the same change flips SearchQueries (corrected) to a net loss, and Quora is insensitive. Whether \tauhigh\ should be a tunable deployment parameter is now on the same footing as verifier selection --- it needs domain-specific calibration and cannot be given a general recommendation.
  \item Validate Section~\ref{sec:deployment}'s specific noise-tolerance and minimum-data figures against a real feedback channel (e.g.\ actual user thumbs-up/down rather than synthetic uniform label flips) and genuinely continuous production traffic, rather than a static dataset resliced to simulate drift or, as in Section~\ref{sec:twitter}, a historical corpus with a known end date.
  \item Validate all conclusions --- currently entirely trace-driven simulation --- against real production traffic.
  \item Add an ``E$'$: rewrite instead of reject'' arm, replacing the verifier's fail action from ``miss'' with ``use a lightweight LLM to edit the candidate answer to fit the new query'' (the paradigm proposed by TweakLLM~\cite{tweakllm}), and see where this lands relative to the binary verification gate on this paper's hit-rate/error-rate Pareto frontier --- an open question this paper does not attempt to answer, not part of its current conclusions.
  \item \textbf{[Added 2026-08-16, already tested]} Another remedy from the same Reddit discussion --- action-verb bucketing as a pre-filter (see \texttt{RESEARCH\_\allowbreak PROPOSAL.md} \S10 direction 7) --- has now been tested on three datasets (Section~\ref{sec:bucketing}): the simplest action-verb-only keying scheme is refuted, producing a statistically significant discriminative-power regression on LmArena. The root cause is that candidates sharing the same verb but an opposite object direction cannot be screened out by this kind of bucketing, and filtering away the easy negatives leaves the verifier facing a systematically harder residual set. Whether keying on an (action, object) pair rather than the action verb alone can close this specific blind spot is a new, more precise open question that this experiment did not test.
  \item \textbf{[Added 2026-08-19, already tested, see Section~\ref{sec:crc}]} Using Conformal Risk Control to upgrade gray-zone threshold calibration from a point estimate to finite-sample risk control: under a random-split assumption (Protocol R), all three datasets pass at all four alpha levels, with finite-sample cost ($\eta$) within 3\% of 1.0 throughout; a chronological split (Protocol T) produces a genuine, systematic guarantee violation on Quora. This opens three new questions: (a) extending this gray-zone-only guarantee to an end-to-end guarantee that also covers the $\tauhigh$ direct-pass branch ($R_{E2E} = R_{shortcut} + R_{grayzone}$, requiring an explicit risk-budget split $\alpha = \alpha_{shortcut} + \alpha_{grayzone}$, not attempted in this section); (b) sweeping $\taulow$ the way Section~\ref{sec:tauhigh} swept $\tauhigh$, to test whether $R_{GZ}(\lambda)$'s steepness varies systematically with it (this section tested only the widest value); (c) a fair numeric comparison against vCache would require new engineering (reimplementing this paper's synchronous global-threshold gate as an asynchronous per-entry policy, or the reverse), which this section's methodology audit found the two existing implementations cannot support directly --- see Section~\ref{sec:crc}; (d) \sout{Protocol R/T both test static splits of a fixed, already-completed historical trace (random vs.\ chronological order) --- not a genuine closed-loop scenario. If the CRC gate's own reuse/reject decisions feed back into future cache state, that could induce a self-selection feedback loop.} \textbf{Tested (2026-08-28, see Section~\ref{sec:selfselect})}: built a genuine online closed-loop simulation and confirmed this failure mode is real, following a precise monotonic dose-response relationship --- triggered by a high direct-hit rate (no detectable effect on Quora at 3.7\%, a moderate significant harm on SearchQueries at 28.1\%, and realized risk more than tripling on LmArena at 72.8\%, strongly significant). A qualitative audit on LmArena confirms 68.9\% of false-accepts match the specific mechanism of ``a correct candidate never independently written because it keeps being successfully reused.'' Online recalibration fully compensates on Quora/SearchQueries but not on LmArena, leaving a new, more precise open question (see future-work item 13). \textbf{[2026-08-29 partial follow-up]} A write-probability sweep on LmArena located this threshold as lying somewhere between $p=0.75$ and $p=1.0$ (residual harm never became non-significant across the tested range, decreasing only ${\sim}35\%$ at $p=0.75$ vs.\ $p=0$); the qualitative audit was also extended to all three datasets and found the LmArena crowding-out mechanism explains only 12.4\% of SearchQueries' explainable cases (vs.\ 68.9\% on LmArena) --- SearchQueries' dominant self-selection mechanism remains unidentified.
  \item \textbf{[Added 2026-08-22, already tested, see Section~\ref{sec:topk}]} Tested whether this paper's design choice of retrieving a single nearest neighbor only (\texttt{k=1}) left unrealized hit-rate headroom on the table --- a gap flagged by an external, independent re-derivation of this project. A K=5 pure ground-truth ceiling shows non-trivial recoverable headroom on all three datasets (LmArena 0.75\%, Quora 1.62\%, SearchQueries 5.66\% of all requests). Plugging in a real verifier to test a K=2 cascade: LmArena is close to a free lunch (error\_rate never moves significantly at any grid point), Quora's effect is essentially zero, and SearchQueries shows a real trade-off (roughly half the grid points show both hit\_rate and error\_rate rising significantly, mitigated but not eliminated by fine-tuning). Untested open questions: whether K=3/4/5 are worth the investment (Phase 0 shows diminishing but non-zero marginal returns), and the cascade's real end-to-end latency cost under true synchronous deployment.
  \item \textbf{[Added 2026-08-28, already tested, see Section~\ref{sec:selfselect}]} Tested whether the CRC gate's self-selection closed-loop feedback is a genuine failure mode --- built a real online closed-loop simulation (rather than Section~\ref{sec:crc} Protocol R/T's static splits), comparing ``insert regardless of hit/miss'' against ``insert only on miss'' write semantics. The result is a precise monotonic dose-response relationship: the higher the direct-hit rate, the larger the self-selection harm (no effect at Quora's 3.7\%, a moderate significant harm at SearchQueries' 28.1\%, a strongly significant harm more than tripling risk at LmArena's 72.8\%), and a qualitative audit on LmArena confirms 68.9\% of false-accepts match the specific mechanism of a correct candidate being crowded out of the cache by its own repeated successful reuse. This raises three new, more precise open questions: (a) online recalibration fully compensates for this harm on Quora/SearchQueries but not on LmArena --- exactly where between 28\%--73\% direct-hit rate this ``can recalibration fully compensate'' threshold sits was never directly located, this section's three datasets merely happening to land on either side of it; (b) only $\alpha=0.02$ was tested, along with three unsensitivity-tested parameters (warm-up window, recalibration cadence, sliding-window size) that could systematically shift the threshold above; (c) the qualitative audit (manually confirming the ``correct candidate crowded out'' mechanism) was only run on LmArena --- SearchQueries shows a significant effect too but has not received the corresponding manual audit.

  \textbf{[2026-08-29 update, (a) and (c) partially tested]} (a) Using write probability (rather than \tauhigh\ or synthetic duplicate data) to sweep ``the probability the cache is still rewritten on a hit,'' $p \in \{0.1, 0.25, 0.5, 0.75\}$, directly on LmArena: harm decreases monotonically but slowly with $p$, still significant at $p=0.75$ and only about 35\% lower than at $p=0$; the true crossover point (if one exists) between 0.75 and 1.0 was not located further, and this round only tested LmArena, so whether the threshold is determined solely by ``direct-hit rate'' or by that variable jointly with others remains uncertain. (b) Sensitivity analysis on the three modeling parameters (alpha, warm-up window, recalibration cadence/window size) is still not done. (c) The qualitative audit has now been extended to all three datasets, but the result is that \textbf{this specific mechanism does not transfer across datasets} --- SearchQueries does show a significant harm, but crowding-out explains only 12.4\% of its explainable cases (far below LmArena's 68.9\%); what actually dominates SearchQueries' harm was not identified this round, a new and more precise open question than before.

  \item \textbf{[Added 2026-08-30, already tested, see Section~\ref{sec:costsensitive}]} Replaced the hit-rate/error-rate frontier with an explicit cost-ratio analysis, $r=C_{\text{false\_accept}}/C_{\text{miss}}$, answering ``under what cost structure is synchronous verification economically worth it'' rather than only presenting a curve. After extending D/E honest calibration's grid to match Group A's $\taulow\in\{0.80,...,0.99\}$ (using a remote GPU to compute six new operating points at $\taulow=0.97$--0.99), all three datasets consistently show: when the cost of an error clearly exceeds the cost of a miss (typically $r\gtrsim1$--9, dataset-dependent), synchronous verification wins economically almost across the entire tested range; only when misses and errors cost about the same, or misses cost more, does a static threshold win. The pre-fix version had shown a ``static threshold reverses on top'' phenomenon at high $r$, confirmed to be an artifact of D/E's grid not reaching as conservative an operating point as Group A's, not a genuine economic reversal. Untested open questions: (a) no real business scenario's actual $r$ value was estimated; (b) $\mathrm{cost}(r)$ assumes costs add linearly, and whether real cost structure has nonlinear effects (e.g.\ the trust collapse from accumulating multiple errors) was not tested.

  \item \textbf{[Added 2026-08-30, already tested, see Section~\ref{sec:redteam}]} Used an LLM to generate adversarial samples covering five known failure axes (negation, action-verb swap, direction reversal, named-entity swap, quantity swap), testing the verifier's worst-case behavior on deliberately constructed hard cases rather than only the average case on natural benchmark data. On 306 qualifying samples, the off-the-shelf verifier's false-accept rate is 84.0\% (95\% CI [79.7\%,87.9\%]), far exceeding anything measured on natural data elsewhere in this paper; more importantly, the in-domain fine-tuning validated in Section~\ref{sec:groupe} provides \textbf{no protection whatsoever} on adversarial samples (87.6\%, CI nearly fully overlapping the untuned result) --- fine-tuning repairs the natural noise distribution, not adversarial robustness, and the two are different capabilities. Untested open questions: (a) the adversarial samples were not independently human-verified; (b) only the LmArena fine-tuned checkpoint was tested; (c) whether adversarial samples might also make the verifier overly conservative was not tested; (d) \sout{whether adversarial training targeted at these five failure axes would help is a natural but untested next step.} \textbf{Tested (2026-08-30, see Section~\ref{sec:advtrain})}: mixing a fresh, non-overlapping batch of adversarial training data (only 3.8\% of the combined training set) into Section~\ref{sec:groupe}'s natural training data and re-running the identical fine-tuning procedure cut the false-accept rate from 84--88\% to 53.6\% (95\% CI [48.0\%,59.2\%], non-overlapping with either baseline), with no cost to natural-data AUC (0.8749 vs.\ the original 0.88) --- proving this capability gap is fixable, not a fundamental limitation of fine-tuning itself, though 53.6\% remains far from natural-data error-rate levels and one category (named-entity swap) got worse.

  \item \textbf{[Added 2026-08-30, already tested, see Section~\ref{sec:advtrain}]} Directly tested Section~\ref{sec:redteam}'s open question (d): whether training on data specifically constructed around known failure axes can close fine-tuning's adversarial-robustness gap. Mixing adversarial data that was only 3.8\% of the combined training set (and non-overlapping with the held-out test set) into Section~\ref{sec:groupe}'s original natural gray-zone training data, then re-running the identical fine-tuning procedure: natural-data AUC is essentially unaffected (0.8749 vs.\ the original 0.88, no natural-data cost), but the adversarial held-out false-accept rate drops significantly from 84--88\% to 53.6\% (95\% CI [48.0\%,59.2\%], non-overlapping with either baseline) --- proving the adversarial-robustness gap is not a fundamental limitation of the fine-tuning mechanism itself, just a missing slice of training data. But the result is far from ``problem solved'': 53.6\% still means more than half of adversarial samples fool the verifier, and one category (named-entity swap) got worse after training (48.7\%$\to$61.5\%), cause undiagnosed. Untested open questions: (a) whether a higher adversarial-data proportion closes the gap further, or starts costing natural-data performance; (b) the specific cause of the named-entity-swap regression (insufficient sample size vs.\ cross-category negative transfer); (c) training/evaluation data both came from the same LLM and prompt-template family, with no test of generalization across generator sources; (d) no comparison against a simpler conservative strategy like raising the static threshold.
\end{enumerate}

\section*{Acknowledgments}

Groups A/B of this paper build directly on vCache's public benchmarks and reference implementation: the SemCacheLmArena and SemCacheSearchQueries datasets, the \texttt{STATIC\_\allowbreak THRESHOLDS}/\texttt{DELTAS} grids used in Groups A/B, and the \texttt{VerifiedDecisionPolicy} this paper ports line-by-line in Group B are all from the vCache project~\cite{vcache}. This paper's evaluation framework would not have been possible without this public infrastructure. Thanks to the vCache authors --- L. G. Schroeder, A. Desai, A. Cuadron, K. Chu, S. Liu, M. Zhao, S. Krusche, A. Kemper, I. Stoica, M. Zaharia, and J. E. Gonzalez --- for open-sourcing it.

Sections~\ref{sec:groupe} and~\ref{sec:twitter}'s experiments likewise build directly on two other public datasets: Quora Question Pairs~\cite{quora} and the Kaggle ``Customer Support on Twitter'' dataset~\cite{twittercs}. Thanks to the creators of both.

\begin{thebibliography}{9}

\bibitem{gptcache}
Bang, F. (2023). GPTCache: An open-source semantic cache for LLM applications enabling faster answers and cost savings. \textit{NLP-OSS 2023}.

\bibitem{vcache}
Schroeder, L. G., Desai, A., Cuadron, A., Chu, K., Liu, S., Zhao, M., Krusche, S., Kemper, A., Stoica, I., Zaharia, M., \& Gonzalez, J. E. (2025). vCache: Verified semantic prompt caching. \textit{arXiv:2502.03771}.

\bibitem{krites}
Singh, A. K., Wang, H., Attaluri, S., Chiam, T., \& Zhu, W. (2026). Asynchronous Verified Semantic Caching for Tiered LLM Architectures. \textit{arXiv:2602.13165}.

\bibitem{calibrationgap}
Baral, A., Ralev, R., Zhechev, I. S., Rajamohan, S., \& Agarwal, J. (2026). Closing the Calibration Gap in Semantic Caching. \textit{arXiv:2606.19719}.

\bibitem{tweakllm}
Cheema, M. T., Aamir, A., Muhammad, K. G., Bhatti, N. A., Qazi, I. A., \& Qazi, Z. A. (2025). TweakLLM: A Routing Architecture for Dynamic Tailoring of Cached Responses. \textit{arXiv:2507.23674}.

\bibitem{quora}
Iyer, S., Dandekar, N., \& Csernai, K. (2017). First Quora dataset release: Question pairs. \textit{Quora Engineering Blog}.

\bibitem{twittercs}
Axelbrooke, S. (2017). Customer Support on Twitter. \textit{Kaggle}.

\end{thebibliography}

\end{document}