Skip to content

RUNBOOK.md

Latest commit

 

History

History
105 lines (82 loc) · 4.6 KB

RUNBOOK.md

File metadata and controls

105 lines (82 loc) · 4.6 KB
Raw

Runbook

Operating notes for running this service, not building it -- config reference, health checks, and what to do when something specific breaks.

Configuration

All configuration is environment variables, loaded via .env in local dev (pydantic-settings, see src/tgpp_rag/settings.py). Nested groups use TGPP_<GROUP>__<FIELD>. Full reference with defaults and comments: .env.example.

The only two variables you actually need to touch to get running:

Variable Purpose
TGPP_LLM__PROVIDER anthropic | openai | ollama
ANTHROPIC_API_KEY (or OPENAI_API_KEY) Required only for the matching provider

Starting the service

docker compose up -d qdrant     # or run Qdrant however you like
uv run tgpp ingest              # one-time (or whenever the catalog changes)
uv run tgpp serve                # http://localhost:8000

Or the whole stack via Docker:

docker compose up -d

Health checks

Endpoint Meaning
GET /api/health Process is up. Always 200 once the app has started.
GET /api/ready Qdrant is reachable, BM25 is fitted, and the collection is non-empty. Returns HealthResponse with the specifics -- use this for load-balancer readiness, not /health.

A 503 from /api/chat or /api/chat/stream with a message about the BM25 vocabulary means tgpp ingest hasn't been run yet in this environment.

Common failures

"no BM25 vocabulary at data/index/bm25.json" Ingestion hasn't run in this environment yet, or data/index/ wasn't mounted as a persistent volume (see docker-compose.yml -- the bm25_index volume exists specifically so this survives a container restart). Run tgpp ingest.

Ingestion fetches 0/12 specs 3gpp.org is behind Cloudflare bot protection that 403s a bare/default HTTP client User-Agent (ingest/fetcher.py sets a browser-shaped one for this reason -- verified empirically while building this, not assumed). If it still fails, 3gpp.org's archive HTML structure may have changed; check fetcher._ZIP_LINK_RE against a manual view of one spec's archive page.

A spec fetches but produces 0 chunks The parser's heading regex (ingest/parser_docx.py::_CLAUSE_HEADING_RE) expects the standard 3GPP authoring template ("5.4.1.2\tTitle" on a Heading N-styled paragraph). A spec using a different template, or an older .doc-only spec with no .docx in its archive, will silently parse to nothing. Check the ingestion log for the per-spec chunk count (tgpp ingest prints a summary table) -- a 0 in that table is the signal, not a crash.

Claude requests failing with stop_reason: "refusal" on security questions Expected for some TS 33.501 / cybersecurity-adjacent phrasing -- Claude Opus 5 runs safety classifiers that occasionally decline benign but security-flavoured requests. TGPP_LLM__ENABLE_REFUSAL_FALLBACK=true (default) retries against TGPP_LLM__REFUSAL_FALLBACK_MODEL automatically; if it's still refusing, the pipeline correctly reports it as an abstention rather than erroring -- check Answer.abstain_reason for the category.

Embedding/rerank/NLI model downloads are slow on first run All three (BAAI/bge-base-en-v1.5, BAAI/bge-reranker-base, cross-encoder/nli-deberta-v3-base) download from HuggingFace on first use and cache under HF_HOME (.cache/huggingface in the Docker image, the default ~/.cache/huggingface locally). Pre-warm the cache in CI or a build step if cold-start latency matters.

High latency per question Expected shape: dense + sparse search (~tens of ms with a small corpus) + cross-encoder rerank (CPU-bound, scales with candidates_per_arm) + one LLM generation call (seconds) + local NLI scoring per claim (fast) + judge escalation only for borderline claims if enabled (adds a call per escalated claim). If latency is dominated by rerank, lower TGPP_RETRIEVAL__CANDIDATES_PER_ARM; if dominated by generation, that's provider/network latency, not this pipeline.

Rotating the LLM provider

Change TGPP_LLM__PROVIDER and restart -- no code change. The judge (verification L3) can run on a separate, typically cheaper model via TGPP_LLM__JUDGE_MODEL; leave it unset to reuse the generator's model.

Re-indexing after a spec updates

tgpp ingest is idempotent per-file (the fetcher skips a re-download if the resolved version's .docx is already cached) but always re-fits BM25 and re-upserts every chunk on each run -- point IDs are derived deterministically from chunk ID (uuid5), so re-running overwrites the same points rather than duplicating them. Delete data/raw/ to force a fresh fetch if you suspect a stale cached archive.