Release v1.6.0: search both AI summaries, return one

The dataset now splits bibo:shortDescription's two @language literals into
descriptionAI (fr) + descriptionAI_en rather than pipe-joining them
(fmadore/IWAC-Hugging-Face#9). Both are added to SUBSET_FIELDS for articles,
documents and audiovisual.

The English column is `searchable: true` with `views: []` — searched, never
returned. Those two halves are each load-bearing:

- Without `searchable`, the English summaries ship to the Hub but stay out of
  TEXT_COLS/FAST_TEXT_COLS, so an English query matches nothing they contain.
  That is strictly worse for anglophone discovery than the pipe-joined column
  it replaced, which at least had the English text inside a searched field.
- Without `views: []`, every response carries both summaries. They say the same
  thing about the same item, so that roughly doubles the ~125 tokens/row an
  abstract costs to tell the reader nothing new. French is the one returned:
  source language, and the only one present on every row (the 51 non-FR/EN
  articles keep an untagged French summary with no English counterpart).

So a keyword hit can now come from a language the item is not written in. The
tool descriptions, the MCP instructions string and the research skill all say so
— an English keyword matching a French article is a discovery route, not
evidence the term is attested in the source.

Guards: a unit test asserting the English column reaches both TEXT_COLS and
FAST_TEXT_COLS wherever the French one exists, another asserting no view ever
projects it, and a fixture where "pilgrimage" appears only in descriptionAI_en
so a hit can only have arrived through the English column. Fixtures carry the
column with English text that is deliberately not a translation.

Tool definitions grow 32 tokens (0.2%), within budget.

Author: fmadore

.agents/skills/iwac-mcp/references/biases-and-limitations.md

@@ -110,7 +110,8 @@ The 4,697 index entries (persons, organizations, places, events, subjects) repre
 ## 8. Search Limitations
 
 - **Substring matching only** — no wildcards, fuzzy matching, or Boolean operators. One term per call; run variants as separate searches.
-- **Keyword scope varies by tool:** articles = title + OCR + AI abstract; publications = title + subject + table of contents + OCR; references = title + abstract. Keyword does NOT search the spatial field — use `subject` for curated tags.
+- **Keyword scope varies by tool:** articles = title + OCR + AI abstracts (**both languages**); publications = title + subject + table of contents + OCR; references = title + abstract. Keyword does NOT search the spatial field — use `subject` for curated tags.
+- **An English keyword can match a French article through its AI abstract.** Articles and documents carry two AI summaries, French and English, and keyword search matches both — so `keyword: "pilgrimage"` returns French-language articles about the hajj even though that word appears nowhere in the source. This is deliberate: it is the only English-language entry point into a francophone corpus. But it means a keyword hit does **not** imply the term is attested in the item itself. For what the sources actually say, read the OCR; for counting how often a word appears *in the press*, AI abstracts are not evidence. Only one summary (the French one) is returned in responses.
 - **Accents and case are folded** (server ≥ 0.6.0): `pelerinage`, `Bénin`, `These` all match. Spelling differences still matter.
 - **Country filters take exact names** (Benin, Burkina Faso, Côte d'Ivoire, Niger, Togo; Nigeria only in references/index/audiovisual). Partial names ("Burkina") are invalid and return an `{error, valid_values}` response — pick a valid value and retry; never read the error as a finding.
 - **Keyword search terms must be French for primary-source subsets** — formulate keyword terms in French for press, publications, documents, and index searches. Academic references are multilingual: search titles/abstracts with French and English concept terms when relevant, while keeping metadata/filter values in French. Semantic embedding queries may be in any language.

.agents/skills/iwac-mcp/references/capabilities-overview.md

@@ -36,7 +36,7 @@ Because reports are date-stamped and location-specific, the collection also supp
 
 ## Ways into the data (recommend 2-3, matched to the question)
 
-- **Keyword search** (French concept terms for press/publications; French and English title/abstract terms for scholarship) across titles, full OCR text and AI abstracts.
+- **Keyword search** (French concept terms for press/publications; French and English title/abstract terms for scholarship) across titles, full OCR text and AI abstracts. Articles and documents carry a French **and** an English AI abstract and both are searched, so an English term can reach a French item through its summary — responses still return only the French one.
 - **Curated themes** — 214 subject tags (Hadj, Laïcité, Ramadan, Éducation islamique…) for precise, pre-classified slices.
 - **People, organisations, places** — find the canonical name among the 4,700 index records, then pull every article that mentions it.
 - **Conceptual (semantic) search** in any language, when keywords would miss paraphrases — "Islamic education reform" also surfaces madrasa-modernisation pieces.

.agents/skills/iwac-mcp/references/research-domains.md

@@ -172,7 +172,7 @@ Server matching is **accent- and case-insensitive** (v0.6.0+), so `pelerinage` a
 - **Country filters take exact names:** Benin, Burkina Faso, Côte d'Ivoire, Niger, Togo (+ Nigeria for references/index/audiovisual). Partial names ("Burkina") are invalid and return an error listing the valid values.
 - **Start broad, then narrow:** begin with a general term (e.g., "madrasa"), then add country or date filters.
 - **Check the index first:** use `search_index` to find the canonical form of a person/organization name, then search articles with that exact form.
-- **Prefer `subject` over `keyword`** for known thematic categories: `keyword` searches title + OCR + AI abstract; `subject` searches the curated tags (which may use different terminology than the OCR text). Discover tags via `list_subjects`.
+- **Prefer `subject` over `keyword`** for known thematic categories: `keyword` searches title + OCR + AI abstracts (French and English); `subject` searches the curated tags (which may use different terminology than the OCR text). Discover tags via `list_subjects`.
 - **Islamic publications vs. mainstream press:** `search_publications` covers Islamic community media (Islam Info, An-Nasr Vendredi, …). Most items are entire issues — navigate by series (`list_periodicals`), subject (87% tagged), country and year; keyword also matches tables of contents (17 of 25 series have them, returned as `matching_toc_entries`); use `get_publication_fulltext` keyword excerpts to read inside an issue.
 - **State press vs. private press:** pre-1991 articles come almost entirely from state/single-party organs, and some outlets changed names across regimes (Ehuzu → La Nation; Togo-Presse ↔ La Nouvelle Marche; L'Observateur → L'Observateur Paalga) — search all name variants when following one outlet across time (see biases-and-limitations.md §6).
 - **Temporal filtering:** articles take `YYYY-MM-DD` or `YYYY`; publications/references take years.

.agents/skills/iwac-mcp/references/tools-by-phase.md

@@ -9,7 +9,7 @@ Cross-subset search for skill-less clients and quick discovery.
 - `query` (required): one concept, name, or short phrase. Tokens are AND-ed across each subset's searchable fields; use French concepts for primary-source discovery, and French/English terms for references. A query whose every word is under 2 characters is **refused** (it would otherwise return an empty result indistinguishable from a real absence).
 - `limit` (default 20, max 50)
 - Returns `results` with namespaced ids (`articles:28576`, `references:11045`, `images:12237`), `title`, `url`, `category`, plus a `ranking` note and `deep_scan`. There is no numeric relevance score; for precise filters use the granular `search_*` tools.
-- **Two passes.** Pass 1 matches curated metadata only (titles, subjects, AI abstracts, tables of contents) and answers in ~0.2 s; if that fills fewer than `limit` results, pass 2 also scans the full OCR (~2.5 s) and `deep_scan: true` says so. A common term therefore never pays for the OCR scan, and a rare one still finds everything — but treat `search` as discovery, not a census: for exhaustive counts use `search_*` or `get_temporal_distribution`.
+- **Two passes.** Pass 1 matches curated metadata only (titles, subjects, AI abstracts in both languages, tables of contents) and answers in ~0.2 s; if that fills fewer than `limit` results, pass 2 also scans the full OCR (~2.5 s) and `deep_scan: true` says so. A common term therefore never pays for the OCR scan, and a rare one still finds everything — but treat `search` as discovery, not a census: for exhaustive counts use `search_*` or `get_temporal_distribution`.
 
 ### fetch
 Fetch one item returned by `search`.
@@ -58,7 +58,7 @@ The 25 Islamic periodical/series titles in the publications subset, with issue c
 
 ### search_articles
 Primary search tool for the 12,287 newspaper articles.
-- `keyword` (optional): substring match on **title + OCR + AI abstract** (does NOT search subject/spatial — use the `subject` parameter for curated tags)
+- `keyword` (optional): substring match on **title + OCR + AI abstracts, French and English** (does NOT search subject/spatial — use the `subject` parameter for curated tags). An English term can therefore match a French article via its English summary
 - `country` (optional): exact name — Benin | Burkina Faso | Côte d'Ivoire | Niger | Togo
 - `newspaper` (optional): substring match
 - `subject` (optional): substring match on the pipe-separated curated tags

CITATION.cff

@@ -26,8 +26,8 @@ identifiers:
     value: 10.5281/zenodo.21805837
     description: "Concept DOI for all versions of the software."
 license: MIT
-version: 1.5.1
-date-released: "2026-08-05"
+version: 1.6.0
+date-released: "2026-08-07"
 keywords:
   - Model Context Protocol
   - MCP server

README.md

@@ -193,15 +193,15 @@ Machine-readable metadata lives in [CITATION.cff](CITATION.cff) — GitHub's
 **Cite this repository** button (sidebar) renders it as APA or BibTeX with the
 current version filled in. In text:
 
-> Madore, F. (2026). *IWAC MCP Server* (Version 1.5.1) [Computer software].
+> Madore, F. (2026). *IWAC MCP Server* (Version 1.6.0) [Computer software].
 > Zenodo. https://doi.org/10.5281/zenodo.21805837
 
 ```bibtex
 @software{madore_iwac_mcp_server,
   author    = {Madore, Frédérick},
   title     = {{IWAC MCP Server}},
   year      = {2026},
-  version   = {1.5.1},
+  version   = {1.6.0},
   publisher = {Zenodo},
   doi       = {10.5281/zenodo.21805837},
   url       = {https://github.com/fmadore/iwac-mcp-server},

mcpb/manifest.json

@@ -2,7 +2,7 @@
   "manifest_version": "0.3",
   "name": "iwac-mcp-server",
   "display_name": "Islam West Africa Collection (IWAC)",
-  "version": "1.5.1",
+  "version": "1.6.0",
   "description": "Read-only access to the Islam West Africa Collection via Hugging Face datasets.",
   "long_description": "Search and analyse the Islam West Africa Collection (IWAC): ~12,000 newspaper articles, 1,500 Islamic publications (searchable by keyword, subject, series, and table of contents), 4,700 index entries, plus audiovisual recordings, fieldwork photographs, archival documents, and academic references. Includes AI sentiment analysis (polarity/centrality/subjectivity, scored independently by gemini-3-flash-preview, gpt-5-mini and ministral-14b-2512), per-article AI abstracts, and per-year coverage timelines. All matching is accent- and case-insensitive.\n\n**No API key is required for the core tools** (cross-subset search/fetch, keyword search, filtering, statistics, timelines, item details). Three optional semantic-search tools use Gemini embeddings — over all articles, the publication series with tables of contents, and the photographs (cross-modal: describe what an image shows) — and need a free Google/Gemini API key; these are disabled by default.\n\nOn first use the server downloads ~250 MB of parquet data from Hugging Face into a local cache.",
   "author": {

mcpb/package-lock.json

@@ -1,6 +1,6 @@
 {
   "name": "iwac-mcp-server-bundle",
-  "version": "1.5.1",
+  "version": "1.6.0",
   "description": "Node MCP server for the Islam West Africa Collection, packaged as a Claude Desktop extension (.mcpb).",
   "private": true,
   "type": "module",

mcpb/package.json

@@ -22,6 +22,29 @@ const IWAC = "https://islam.zmo.de/s/afrique_ouest/item/";
 /** Subsets that carry an `OCR` column, and therefore the per-row public flag. */
 const OCR_SUBSETS = ["articles", "publications", "documents", "audiovisual"];
 
+/**
+ * Subsets carrying the AI summary. Since the summariser went bilingual the
+ * dataset stores the two `@language` literals as `descriptionAI` (fr) and
+ * `descriptionAI_en` rather than pipe-joining them, so the fixture must hold
+ * both — an English-only term has to be findable, which is the whole point of
+ * tagging the English column `searchable`.
+ */
+const DESC_AI_SUBSETS = ["articles", "documents", "audiovisual"];
+
+/**
+ * English summaries, keyed by `o:id`. Deliberately NOT translations of the
+ * French: each uses vocabulary ("pilgrimage", "fasting") that appears nowhere
+ * in the French text or the OCR, so a test matching one proves the English
+ * column is really in the search surface rather than riding on a shared token.
+ * Audiovisual is absent on purpose — 0/47 filled in the real subset.
+ */
+const DESC_AI_EN = {
+  101: "Report on the pilgrimage of Beninese faithful to Mecca.",
+  102: "The fasting month as lived in the capital's mosques.",
+  103: "Celebrations marking the end of the fasting month in Abidjan.",
+  501: "Annual report of a Burkinabè Muslim students' association.",
+};
+
 /**
  * Subsets the pipeline writes Hijri date columns to
  * (post-processing/calculate_hijri_dates.py). `references` is deliberately
@@ -277,6 +300,19 @@ async function main() {
       await conn.run(`ALTER TABLE ${table} ADD COLUMN "OCR_is_public" BOOLEAN`);
       await conn.run(`UPDATE ${table} SET "OCR_is_public" = length(trim(coalesce("OCR", ''))) > 0`);
     }
+    // The English half of the bilingual AI summary. Empty string, not NULL,
+    // for rows without one — the real parquet stores empty strings here, and a
+    // NULL would let a `COUNT()` claim coverage the subset does not have.
+    if (DESC_AI_SUBSETS.includes(subset)) {
+      await conn.run(`ALTER TABLE ${table} ADD COLUMN "descriptionAI_en" VARCHAR`);
+      await conn.run(`UPDATE ${table} SET "descriptionAI_en" = ''`);
+      for (const [oid, text] of Object.entries(DESC_AI_EN)) {
+        await conn.run(
+          `UPDATE ${table} SET "descriptionAI_en" = '${text.replace(/'/g, "''")}'
+           WHERE "o:id" = '${oid}'`,
+        );
+      }
+    }
     // Hijri columns, keyed off pub_date exactly as the pipeline does. Rows whose
     // date is not a complete YYYY-MM-DD (year-only, or a `1981-04/1981-06`
     // range) are left NULL — an imprecise date has no lunar day, and that