Architecture Decision Records for the choices in this project that had a real alternative worth naming. Formatted as: context, decision, what was given up, and what would make us revisit it. Skipped: choices with no serious alternative (e.g. "use Python" for an ML-adjacent service).
Context. Generic RAG chunkers slide a fixed-size window (e.g. 512 tokens, 50-token overlap) across raw text, blind to document structure. 3GPP specs have an unusually regular, deep clause hierarchy where the section number is the citable unit -- "per TS 24.501 §5.5.1.2" is a complete, checkable reference; "per TS 24.501, somewhere around 60% through the document" is not.
Decision. ingest/chunker.py never lets one chunk span two leaf
clauses. A short clause is one chunk; a long clause becomes several
chunks that all keep the same clause_path, so a citation always
resolves to one section number a reader can look up, even if it doesn't
capture the clause's full text.
Trade-off. A fixed-window chunker is simpler and handles arbitrary document formats without a custom parser. We gave that up for a chunker that only works on documents with a real structural convention -- which 3GPP specs have, and which is exactly why a from-scratch chunker was worth building instead of reaching for an off-the-shelf splitter.
Revisit if: the corpus grows to include specs whose authoring template
doesn't follow the "number + tab + title" heading convention (older
legacy .doc specs sometimes don't) -- the parser would need a second
heading-detection strategy, not a different chunking policy.
Context. Dense embeddings retrieve by meaning and miss exact-token matches (a query for "AMF" can retrieve mobility-management prose in general while missing the one clause that names "AMF" verbatim next to the fact the user needs). BM25 retrieves by exact term overlap and misses paraphrase and synonymy. 3GPP questions hit both failure modes: acronym- literal lookups ("what does 33.501 say about SUCI") and conceptual questions with no exact-term overlap to the answer ("how does a UE avoid sending its permanent identity in the clear").
Decision. Run both arms, fuse with Reciprocal Rank Fusion
(retrieval/fusion.py), then rerank the fused candidates with a
cross-encoder. RRF specifically (not a weighted score blend) because
cosine similarity and BM25 scores live on incomparable scales with no
principled conversion between them -- RRF sidesteps that by fusing on
rank, not score.
Trade-off. Two retrieval arms plus a fusion step plus a rerank step is more moving parts than "just use embeddings," and it's real latency (three model calls -- embed, then rerank -- per query, plus a lexical search). We accepted that cost because on this corpus the two arms fail on disjoint query types, and reranking the fused, deduplicated candidate set is cheaper than reranking either arm's full candidate list separately.
Revisit if: eval data shows one arm dominates recall across the whole query distribution -- at that point the second arm is cost without benefit and should be dropped, not kept for symmetry.
Context. A chat UI usually streams the model's tokens as they're generated for responsiveness. This pipeline's whole premise is that a generated sentence isn't trustworthy until the verification gate has checked it -- which requires the complete answer, because claim-splitting and entailment scoring operate on finished sentences.
Decision. /api/chat/stream streams SSE events for pipeline stage
transitions (retrieving → generating → verifying) and then one
answer event carrying the fully-verified result. No partial generation
text is ever sent.
Trade-off. This is a strictly worse "feels alive" experience than token streaming during the several seconds an answer takes -- there's no incremental text to watch appear. We chose it anyway because token streaming here would mean showing the user sentences that might be retracted a moment later when verification strips them, which is a worse experience than a short wait: a system that visibly takes back what it just said undermines the whole trust story this project is built around.
Revisit if: verification latency becomes the dominant cost and stage events alone feel too coarse -- a finer-grained "claim N of M verified" event could be added without ever showing unverified text.
Context. Qdrant supports sparse vectors natively, but computing them
still requires an encoder -- typically SPLADE (a learned sparse model) or
classic BM25. rank_bm25 and similar libraries exist on PyPI.
Decision. index/sparse.py implements BM25 term weighting directly:
tokenize, build vocabulary + document frequencies at fit time, encode with
the standard Robertson/Sparck-Jones formula. No trained model, no extra
dependency.
Trade-off. A learned sparse model (SPLADE) generally outperforms classic BM25 on modern IR benchmarks, and a library would have taken less code. We chose hand-rolled classic BM25 because it needs no training data or GPU, is fully deterministic (reproducible eval numbers), and — the concrete reason this matters day-to-day — when a citation looks wrong, the sparse-side contribution to its score can be recomputed by hand from the formula instead of treated as a black box.
Revisit if: eval data shows sparse-arm recall is the retrieval bottleneck -- at that point a learned sparse encoder is worth the added complexity.
Context. Two places in the pipeline could decide "the corpus doesn't have this": before generation (based on retrieval scores alone) or after (based on what the model says about the evidence it was given).
Decision. retrieval/pipeline.py's sufficiency check runs first and
can skip generation entirely (qa.py: if not retrieval.sufficient: return abstain(...)). Generation only happens once retrieval has already
decided the evidence is worth answering from.
Trade-off. A model asked to answer from three barely-relevant chunks
will often produce something plausible-sounding rather than declining, so
relying solely on the model's own "I don't know" judgement (post-hoc) is
measurably weaker than not asking in the first place. Catching it before
generation also means a weak-evidence question costs nothing beyond the
retrieval call -- no generation tokens, no verification pass. The cost is
that the sufficiency threshold (min_rerank_score, min_evidence_chunks)
is a blunt, corpus-wide instrument; it can't take question difficulty into
account the way a smarter, per-question judgement might.
Revisit if: the eval's adversarial abstention rate is high but the golden set's answered rate drops too -- that's the signal the threshold is too conservative and needs per-category tuning rather than one global cutoff.
Context. A single verification method could have been chosen: just mechanical citation checking, or just an LLM asked "is this claim supported?".
Decision. Layer the checks by cost and failure mode: mechanical (free, catches fabricated citations) → local NLI (free, catches claims that don't follow from what they cite) → LLM judge (costs tokens, opt-in, reserved for claims the NLI model is genuinely unsure about).
Trade-off. Three layers is more code and more to explain than one. The reason it's worth it: each layer catches a failure mode the layer before it structurally cannot. Mechanical checking cannot tell if a real citation actually supports the claim; a general-purpose NLI cross-encoder, trained on short sentence pairs, is unreliable on 3GPP's compound conditional prose specifically in the middle confidence band -- exactly where escalation to a judge is targeted, rather than judging every claim (which would make L3 the dominant cost) or trusting every mid-band claim either way (which is where an entailment-only design would actually fail).
Revisit if: judge escalation rate is consistently near 0% or near 100% on real traffic -- either means the entailment score isn't tracking actual difficulty and the NLI model may need fine-tuning on telecom-domain sentence pairs instead.
Context. The generator and the judge both need an LLM call. The system could be written directly against the Anthropic SDK.
Decision. generation/providers/base.py defines a two-method
interface (complete); anthropic_provider.py, openai_provider.py, and
ollama_provider.py implement it, selected by TGPP_LLM__PROVIDER.
Nothing outside generation/providers/ imports an SDK directly.
Trade-off. An abstraction layer is code that a hard dependency doesn't
need, and it means no call site can use a provider-specific feature
without either generalizing the interface or reaching around it. Given up
deliberately: a reviewer without an Anthropic key can still run the whole
pipeline (TGPP_LLM__PROVIDER=ollama, fully offline, zero cost), and the
generator can't accidentally couple to something Claude-specific that
would make swapping providers a rewrite instead of a config change.
Revisit if: a provider-specific capability (e.g. Claude's fine-grained citation blocks) becomes load-bearing for verification quality -- at that point the abstraction needs an optional capability-detection method rather than staying purely least-common-denominator.
Context. A natural instinct for "grounded, factual" generation is
temperature=0. Claude Opus 5 rejects non-default sampling parameters
outright (400 error) — confirmed against the current Anthropic API
reference while building this, not assumed from an older model's
behaviour that no longer holds.
Decision. No sampling parameters are sent at all. Determinism-for- grounding comes from the evidence-block prompt structure and, downstream, from the verification gate -- not from suppressing sampling variance.
Trade-off. Sampling temperature was never a reliable grounding
mechanism anyway (a temperature=0 model can still confidently
hallucinate), so this cost nothing real; the correction still matters
because it's the kind of stale assumption that silently breaks a whole
service the day a model version changes, and every future model swap in
this project checks current API constraints before reusing an old
request shape.
Context. Dense and sparse retrieval could live in separate Qdrant collections, queried independently, or as two named vectors on the same point in one collection.
Decision. One collection, one point per chunk, named vectors dense
and sparse (index/store.py).
Trade-off. Two collections would allow independently scaling or
sharding each retrieval arm. For a corpus this size (low thousands of
chunks) that scaling headroom isn't needed, and the single-collection
design buys a real correctness guarantee instead: one upsert writes both
representations atomically, so there is no code path where the dense and
sparse indexes can drift out of sync with each other -- there's exactly
one set of chunk IDs to keep consistent, not two.
Revisit if: the corpus grows enough that dense and sparse search need independent scaling (different replica counts, different hardware) -- that's the point where the operational cost of two collections starts paying for itself.