Release v0.12.0

Minor rather than patch: a new subset, three new tools, and two visible
contract changes since v0.11.0.

Full-text coverage is now disclosed. The server reads the PUBLIC dataset,
which masks OCR per row by `OCR_is_public`: only 7,480/12,287 articles
(61%) and 1,298/1,501 publications (86%) carry full text. Keyword search
still reaches every item's title, subjects and AI abstract, so nothing is
invisible — but the full-text half of a match covered only those shares,
silently, and a keyword count read as a corpus-wide census.
get_collection_stats now returns `fulltext_coverage` and `fulltext_note`,
computed from the boolean flag (1-7 ms) rather than a
`length(trim(OCR))` scan (344 ms on publications); the caveat is mirrored
in the handshake instructions, SKILL.md, and the smoke test's pins, so an
upstream policy change trips an alarm instead of shifting every count.

`search` is now two-phase. Columns tagged `heavy` in SUBSET_FIELDS (the
OCR blobs) are excluded from the new FAST_TEXT_COLS; pass 1 matches
curated metadata and pass 2 scans OCR only when pass 1 under-fills the
page. Measured against the July 2026 dataset: 3.09 s -> 0.20 s for a
common term, 4.31 s -> 0.25 s for two tokens, with recall unchanged and
`deep_scan` reporting which path ran.

New in this release:

- The `images` subset (30 fieldwork photographs), with search_images,
  get_image, and semantic_search_images — cross-modal, ranking against
  the multimodal `embedding_image`, which is the only real handle on a
  subset where 28 of 30 have no caption. Tools: 27 core + 3 semantic.
- Audiovisual transcriptions. The subset gained an `OCR` column (4/47
  filled) that is now its body and part of its search surface; it was
  bodied on `descriptionAI`, which is empty for all 47 rows, so every
  fetch answered "(no full text available)" while transcriptions sat
  unread.
- MCP prompts (iwac_research brief/extended, iwac_overview) — the only
  workflow channel a skill-less client such as ChatGPT receives.
- An MCP App: get_temporal_distribution declares `_meta.ui.resourceUri`,
  so hosts supporting the extension render an interactive chart that can
  call the tool back to switch granularity. `_meta` is inert elsewhere,
  so every other client gets byte-identical output. Palette and type
  mirror the IwacVisualizations Omeka module's chart tokens.

Fixes:

- resolveLimit reports clamps at the LOW end too: `limit: 0` returned one
  row with nothing saying why, which reads as "that is all there is".
  Out-of-range excerpt parameters come back in `parameter_note`.
- A `search` query whose every token is under 2 characters is refused
  rather than answered with `count: 0`, which was indistinguishable from
  a term that is genuinely unattested.
- `url`/`title` fall back to the derivable canonical item page. ChatGPT
  builds citation metadata only when `url` is a non-empty string, and the
  result compaction drops empty strings.
- countryFilterIfExists folded into pipeValueFilterIfExists (it had
  become a pure passthrough).
- Biome 2.5.5; documented the win32-arm64 lint segfault and the x64
  emulation workaround, since the pre-commit lint could not run locally.

Verified: typecheck, biome, 48 unit tests, offline fixture and HTTP
round-trips, and the live smoke test against the real dataset.

Author: fmadore

.agents/skills/iwac-mcp/SKILL.md

@@ -14,13 +14,13 @@ description: |
 
 # IWAC MCP Research Workflow
 
-Structured methodology for academic research using the IWAC MCP server's 27 possible tools (25 core + 2 optional semantic). Adapted from ALA-compliant archival research practices. Applies to server **v0.9.0+** — all matching is accent- and case-insensitive; result objects use short English keys (`id`, `date`, `polarity`, `centrality`, `subjectivity`, `description_ai`, `url`); list/search tools return a pagination envelope (`count`, `total_matches`, `offset`, `limit`, `has_more`, `next_offset`); and enumerated filters are validated (see **Reading Results & Errors** below). The essentials of this guidance are mirrored in the server's MCP `instructions` string (`mcpb/src/index.ts`) for skill-less clients — when updating one, update the other.
+Structured methodology for academic research using the IWAC MCP server's 30 possible tools (27 core + 3 optional semantic). Adapted from ALA-compliant archival research practices. Applies to server **v0.9.0+** — all matching is accent- and case-insensitive; result objects use short English keys (`id`, `date`, `polarity`, `centrality`, `subjectivity`, `description_ai`, `url`); list/search tools return a pagination envelope (`count`, `total_matches`, `offset`, `limit`, `has_more`, `next_offset`); and enumerated filters are validated (see **Reading Results & Errors** below). The essentials of this guidance are mirrored in the server's MCP `instructions` string (`mcpb/src/index.ts`) for skill-less clients — when updating one, update the other.
 
 ## Prerequisites
 
 Load reference files **as needed**, not all upfront:
 
-1. **references/tools-by-phase.md** — all 27 possible tools with parameters, defaults, and verified filter vocabularies. Read before the first search of a session.
+1. **references/tools-by-phase.md** — all 30 possible tools with parameters, defaults, and verified filter vocabularies. Read before the first search of a session.
 2. **references/research-domains.md** — French search terms and transliteration variants by domain. Read when crafting search-term variants (Extended mode, or when a Brief search comes back thin).
 3. **references/biases-and-limitations.md** — collection biases, coverage gaps, sentiment caveats. Read before writing the synthesis.
 4. **references/capabilities-overview.md** — plain-language description of the collection and recommended ways into the data. Read when the user asks what you can do (see "Capability Questions" below).
@@ -29,7 +29,7 @@ For data schema and Omeka S API details, defer to the `iwac-data` skill.
 
 ## Capability Questions
 
-When the user asks what you can do with IWAC ("what can you do?", "qu'est-ce que tu peux faire ?", "what's in this collection?", "how could I search this?"), do **not** launch the research workflow, present the depth choice, or enumerate the 27 tools. Read **references/capabilities-overview.md** and answer in plain language, in the user's language:
+When the user asks what you can do with IWAC ("what can you do?", "qu'est-ce que tu peux faire ?", "what's in this collection?", "how could I search this?"), do **not** launch the research workflow, present the depth choice, or enumerate the 30 tools. Read **references/capabilities-overview.md** and answer in plain language, in the user's language:
 
 1. One short paragraph on what the collection is and covers.
 2. The main ways into the data (keyword, curated themes, people/organizations, semantic, sentiment, periodicals, scholarship) — described as research moves, not tool names.
@@ -41,7 +41,7 @@ Close by inviting a research question. Present the Brief/Extended choice only on
 
 Mirror the user's question language in the report, synthesis, capability answer, and follow-up questions. If the user asks in English, write the final answer in English; if they ask in French, write it in French; if mixed, use the dominant language.
 
-Always formulate keyword/substr search strings and concept keywords in French for press articles, publications, documents, and index searches, including when the user's question is in another language. Translate concepts before keyword searching (`pilgrimage` -> `pèlerinage`, `secularism` -> `laïcité`, `Islamic education` -> `enseignement islamique`). For academic references, run title/abstract keyword searches in French **and** English when relevant; metadata/filter labels such as `reference_type` and `language` use French values. This rule does **not** apply to `semantic_search_articles` or `semantic_search_publications`: semantic embedding queries may be written in any language. Keep proper names, quoted titles, and canonical filter values exact; do not translate item titles or citation text.
+Always formulate keyword/substr search strings and concept keywords in French for press articles, publications, documents, and index searches, including when the user's question is in another language. Translate concepts before keyword searching (`pilgrimage` -> `pèlerinage`, `secularism` -> `laïcité`, `Islamic education` -> `enseignement islamique`). For academic references, run title/abstract keyword searches in French **and** English when relevant; metadata/filter labels such as `reference_type` and `language` use French values. This rule does **not** apply to `semantic_search_articles`, `semantic_search_publications` or `semantic_search_images`: semantic embedding queries may be written in any language. Keep proper names, quoted titles, and canonical filter values exact; do not translate item titles or citation text.
 
 ## Research Depth
 
@@ -209,3 +209,4 @@ See **references/research-domains.md** for comprehensive term lists by domain.
 6. **Search incrementally.** Keep limits low, search one dimension at a time, avoid retrieving full OCR text unless needed.
 7. **Publications are mostly entire issues.** Individual articles within an issue are not separated; use the table of contents where one exists (17 of 25 series) and `get_publication_fulltext` keyword excerpts to localise content inside an issue.
 8. **Mind the 1990-91 press-system break.** Pre-1991 articles (~11% of the corpus) come almost entirely from state or single-party organs; the private press only emerges with political liberalisation. Temporal comparisons crossing 1990 compare two different press systems (see biases-and-limitations.md §6).
+9. **Full text is masked per item.** The server reads the *public* dataset, where OCR ships only for items whose content is public on islam.zmo.de — about **61% of articles** (7,480/12,287) and **86% of publications** (1,298/1,501). Titles, subjects and AI abstracts cover every item, so nothing is invisible to discovery, but the full-text half of a keyword match reaches only those shares. Read `fulltext_coverage` from `get_collection_stats`, treat keyword totals as a **floor rather than a census**, and say so whenever a count carries an argument.

.agents/skills/iwac-mcp/references/tools-by-phase.md

@@ -6,9 +6,10 @@
 
 ### search
 Cross-subset search for skill-less clients and quick discovery.
-- `query` (required): one concept, name, or short phrase. Tokens are AND-ed across each subset's searchable fields; use French concepts for primary-source discovery, and French/English terms for references.
+- `query` (required): one concept, name, or short phrase. Tokens are AND-ed across each subset's searchable fields; use French concepts for primary-source discovery, and French/English terms for references. A query whose every word is under 2 characters is **refused** (it would otherwise return an empty result indistinguishable from a real absence).
 - `limit` (default 20, max 50)
-- Returns `results` with namespaced ids (`articles:28576`, `references:11045`), `title`, `url`, `category`, plus a `ranking` note. There is no numeric relevance score; for precise filters use the granular `search_*` tools.
+- Returns `results` with namespaced ids (`articles:28576`, `references:11045`, `images:12237`), `title`, `url`, `category`, plus a `ranking` note and `deep_scan`. There is no numeric relevance score; for precise filters use the granular `search_*` tools.
+- **Two passes.** Pass 1 matches curated metadata only (titles, subjects, AI abstracts, tables of contents) and answers in ~0.2 s; if that fills fewer than `limit` results, pass 2 also scans the full OCR (~2.5 s) and `deep_scan: true` says so. A common term therefore never pays for the OCR scan, and a rare one still finds everything — but treat `search` as discovery, not a census: for exhaustive counts use `search_*` or `get_temporal_distribution`.
 
 ### fetch
 Fetch one item returned by `search`.
@@ -17,8 +18,8 @@ Fetch one item returned by `search`.
 
 ## Phase 1: Scoping Tools
 
-### get_collection_stats
-Overall collection statistics: subset record counts, articles by country, date range, newspaper count.
+### get_collection_stats *(reports `fulltext_coverage` since v0.12.0)*
+Overall collection statistics: subset record counts, articles by country, date range, newspaper count, and **`fulltext_coverage`** — how many items in each subset actually carry OCR in this public dataset (~61% of articles, ~86% of publications; the rest are masked per row by `OCR_is_public`). Read it before reporting any keyword total as a corpus-wide figure.
 - No parameters. Use first to understand scale. (First call may trigger the parquet download.)
 
 ### get_country_comparison
@@ -118,16 +119,26 @@ Search the 26 archival documents (Islamic association reports, flyers, project d
 - Call with no arguments to list all 26.
 
 ### list_audiovisual
-The ~47 audiovisual items (July 2026) — all Nigeria, incl. Hausa/Arabic recordings. (AI descriptions not yet populated in the dataset.)
+The ~47 audiovisual items (July 2026) — all Nigeria, incl. Hausa/Arabic recordings. (AI descriptions still empty for all 47; **4 items now carry a transcription**.)
 - `country` (optional), `limit` (default 20, max 50), `offset`
 - Returns: id, title, creator, publisher, country, date, medium, extent, subject, spatial, language, media_url, url
 
 ### search_audiovisual
-Search the audiovisual subset by title/metadata. Useful because AI descriptions are currently empty.
-- `keyword` (optional): substring over title, creator, publisher, subject, spatial, language, source, and AI description where present
+Search the audiovisual subset by title/metadata **and transcription**. Useful because AI descriptions are still empty.
+- `keyword` (optional): substring over title, creator, publisher, subject, spatial, language, source, the transcription (`OCR`, 4/47 rows) and AI description where present
 - `country` (optional), `language` (optional exact pipe value), `medium` (audio | video), `subject` (optional exact tag), `limit` (default 20, max 50), `offset`
 - Returns the same summary fields as `list_audiovisual`
 
+### search_images *(new July 2026)*
+The 30 fieldwork photographs (mosques, radio stations, schools, signage, street scenes). Captions are almost absent (2/30), so prefer `subject`/`spatial` filters or `semantic_search_images` over keywords.
+- `keyword` (optional): substring over title, creator, subject, spatial and the rare caption
+- `country`, `subject` (exact pipe value), `spatial` (exact pipe value), `creator` (substring), `date_from`, `date_to`, `limit` (default 20, max 50), `offset`
+- Returns: id, title, creator, date, country, spatial, coordinates (`"lat, lng"`), subject, image_url, url
+
+### semantic_search_images *(optional — requires semantic search enabled + Google API key)*
+**Cross-modal.** Ranks against `embedding_image`, a multimodal embedding of the photograph *itself* in the same 768-dim space as the text vectors — so describing what an image shows works even though 28 of 30 have no caption.
+- `query` (describe the visual content, any language), `country` (optional), `limit` (default 10, max 30)
+
 ### semantic_search_publications *(optional — requires semantic search enabled + Google API key)*
 Semantic similarity over publication **tables of contents** via Gemini embeddings. TOC coverage (verified June 2026): **325/1,501 issues (~22%)**, all embedded — **complete for 17 of the 25 series** (Le Rendez-Vous 78, Plume Libre 49, L'Appel 48, Alif 32, La Preuve 28, An-Nasr Trimestriel 16, Le CERFIste 13, Al-Azan 13, ASSALAM 11, Al Mawadda 11, Al Maoulid Info 7, Le Pacific 6, Al Maoulid Magazine 5, AJMCI Infos 4, Al Muwassat Info 2, Bulletin d'information du CNI 1, Les Échos de l'AEEMCI 1), but **zero for the three largest** (Islam Info 695, An-Nasr Vendredi 318, Islam Hebdo 122) and five other small series.
 - `query` (natural language, any language), `country` (optional), `limit` (default 10, max 50)
@@ -162,7 +173,12 @@ Full archival-document detail (metadata, AI description, capped OCR).
 ### get_audiovisual
 Full audiovisual metadata.
 - `audiovisual_id` (int)
-- Returns id, identifier, title, creator, publisher, country, date, media_url, medium, duration (`extent`), subject, spatial, language, source, and IWAC URL.
+- Returns id, identifier, title, creator, publisher, country, date, media_url, medium, duration (`extent`), subject, spatial, language, source, `transcription` (where one exists — 4/47), and IWAC URL.
+
+### get_image *(new July 2026)*
+Full photograph record.
+- `image_id` (int)
+- Returns id, identifier, title, type, creator, date, country, spatial, coordinates, subject, caption (`description`, rarely present), rights, `iiif_manifest`, `thumbnail`, full-resolution `image_url`, and IWAC URL. URLs only — the server never returns image bytes.
 
 ### get_index_entry
 Detailed index entry. **Raw dataset columns, French names** (Titre, Titre alternatif, Type, Description, Prénom, Nom, Coordonnées, frequency, first/last_occurrence, countries…).

TODO.md

@@ -61,6 +61,11 @@
   already implemented, no resources (error-code change N/A), no roots/sampling/
   logging capabilities (deprecations N/A), static bearer auth (OAuth hardening
   N/A), single-backend proxy (routing headers N/A), read-only tools (MRTR N/A).
+  **One gap that audit did not cover:** the stateless core removes the
+  `initialize` handshake in favour of `server/discover`, and the handshake
+  `instructions` string is this server's ENTIRE guidance floor for skill-less
+  clients (ChatGPT). Confirm where `instructions` surface under `server/discover`
+  — and that prompts (`src/prompts.ts`) still list — before migrating.
 
 ## Data Enrichment (Track 2 — runs in the IWAC-Hugging-Face pipeline, not here)
 
@@ -102,20 +107,33 @@
   table/subset, so users can search *within* periodicals without loading a
   full (up to ~278k-token) OCR blob.
 
-### Audiovisual (45 rows)
+### Audiovisual (47 rows)
 
 - [ ] **Populate `descriptionAI`** — the column exists but is empty for all
-  45 rows (`length(trim(...)) = 0`; a bare `COUNT()` claims 45/45 because the
-  parquet stores empty strings, not NULLs). The recordings are Hausa/Arabic
-  content, so per-item AI descriptions are the only browsable surface.
+  47 rows (`length(trim(...)) = 0`; a bare `COUNT()` claims 47/47 because the
+  parquet stores empty strings, not NULLs). Partly mitigated since July 2026:
+  the subset gained an `OCR` transcription column, which the server now serves
+  as the item body (`transcription` on `get_audiovisual`, `text` on `fetch`) and
+  searches — but only **4 of 47** rows have one, so AI descriptions remain the
+  browsable surface this subset needs.
+- [ ] **Transcribe the remaining 43 recordings** — Hausa/Arabic audio; the 4
+  existing transcriptions prove the column and the server path work end to end.
+
+### Images (30 rows, new July 2026)
+
+- [ ] **Populate `description`** — 2 of 30 photographs have a caption. Discovery
+  currently leans on title/subject/place plus the multimodal `embedding_image`
+  (`semantic_search_images`), which works without captions but cannot be quoted
+  in a write-up.
 
 ### Server tools that light up once the columns land
 
 - `semantic_search_references` — new tool, needs `embedding_abstract`.
 - `semantic_search_publications` — ✅ now useful for the 17 TOC-covered series;
   corpus-wide once Islam Info / An-Nasr Vendredi / Islam Hebdo TOCs land.
 - `search_publications` — returns AI summaries once `descriptionAI` is populated
-  (add a description column to `publicationSummaryCols`).
+  (add a `descriptionAI` field to `SUBSET_FIELDS.publications` in
+  `src/tools/_shared.ts`, tagged `searchable` and in the `summary` view).
 
 ## Skill Improvements