Release v3.5.1: read the lunar columns as numbers, not as text

The Hijri columns are DOUBLE in the parquet, and every code path that rendered
them as text cast straight to VARCHAR. CAST(3.0 AS VARCHAR) is '3.0', and
DuckDB's lpad TRUNCATES a string already longer than the target width instead of
leaving it alone, so lpad(CAST(hijri_month AS VARCHAR), 2, '0') produced '3.'
rather than '03'. Months 10, 11 and 12 came out '10', '11', '12' by accident,
which is exactly why the fault survived: three of the twelve buckets looked
right.

get_temporal_distribution(granularity=lunar_month) therefore returned nine keys
no month_labels entry matches, and the charts.html lunar view drew three bars out
of twelve — 4,563 of the 13,261 dated articles. Nothing errored anywhere. The
chart's annotation then divided that truncated total by twelve and reported "Dhu
al-Hijja leads at 1854 (+388% vs an even split)", a figure produced entirely by
the missing bars: Ramadan actually leads at 1918, and the real spread is +74%.
The hijri month and year branches were malformed the same way, as '1420.0-3.'
and '1420.0'.

HIJRI_DATE_EXPR carried the identical bug with a wider blast radius. It builds
the hijri_date field on every article and publication row, and because the year
part is unpadded there was no accidental-pass case at all: all 13,261 values were
malformed, reading 1440.0-3.-29 instead of 1440-03-29. That is the one users saw
most and the one nothing was checking.

Both now go through hijriPart(), which carries the DOUBLE -> INTEGER cast in one
place with the reason attached, so the two call sites cannot drift apart again.
It is also robust to the columns changing type: CAST('03' AS INTEGER) is still 3.

The reason the suite never caught any of this is the fixture, not the assertions.
make-fixtures declared the three columns BIGINT while the real parquet stores
DOUBLE, and CAST(3 AS VARCHAR) is '3' — so the fixtures rendered every bucket and
every date correctly no matter how the SQL was written, and could not reproduce
this class of bug at all. Typing them DOUBLE reproduces all four failures on
unmodified source. That change is worth more than the fix it caught.

On top of it, the lunar_month check now asserts the two key sets as a contract:
month_labels is exactly 01..12, and every distribution key is present in it. A
bucket key that misses is not an error anywhere in the stack — it is a bar that
silently never draws — so it has to be asserted rather than inferred from counts.
The weekly live smoke test pins counts but never cross-checked the two sets,
which is why its pins stayed green throughout.

The README and skill figures are re-measured against the corrected output:
Ramadan +74%, Dhu al-Hijja +68%, Shawwal +42% over 13,261 fully-dated articles,
with the six ordinary months 24-32% below an even split. Rabi' I holds at -5%, so
the Maouloud finding is unchanged. The stated conversion rate moves 98.9% -> 99%
with the current dataset; that one is drift, not this bug.

Author: fmadore

.agents/skills/iwac-mcp/SKILL.md

@@ -247,4 +247,4 @@ only items that happen to mention it.
 7. **Publications are mostly entire issues.** Individual articles within an issue are not separated; use the table of contents where one exists (17 of 25 series) and `get_publication_fulltext` keyword excerpts to localise content inside an issue.
 8. **Mind the 1990-91 press-system break.** Pre-1991 articles (~11% of the corpus) come almost entirely from state or single-party organs; the private press only emerges with political liberalisation. Temporal comparisons crossing 1990 compare two different press systems (see biases-and-limitations.md §6).
 9. **Full text is masked per item, and the newest items are not yet enriched.** The server reads the *public* dataset, where OCR ships only for items whose content is public on islam.zmo.de — about **56% of articles** (7,546/13,397) and **86% of publications** (1,298/1,501). Titles and subjects cover every item; AI abstracts cover every item the enrichment pass has reached. Ingestion runs ahead of that pass, so the most recent arrivals (~1,050 articles at the 2026-08 refresh) carry metadata only — no OCR, no `description_ai`, no sentiment, no topic — and because results come back newest-first, they occupy page 1 of an unfiltered `search_articles`. Nothing is invisible to discovery, but a triage pass built on `description_ai` should bound its dates rather than assume every row carries one. Read `fulltext_coverage` from `get_collection_stats`, treat keyword totals as a **floor rather than a census**, and say so whenever a count carries an argument.
-10. **A lunar-month count is not seasonality.** `granularity="lunar_month"` pools every Hijri year, so it deliberately mixes Gregorian seasons: a Ramadan peak is an observance effect, never a weather or school-year one. It also needs a complete `YYYY-MM-DD` — items dated only to a year or month land in `imprecise_date_count` and are **absent from the bars, not zero** (98.9% of articles and 82.9% of publications convert). Lunar dates do not exist for `references`.
+10. **A lunar-month count is not seasonality.** `granularity="lunar_month"` pools every Hijri year, so it deliberately mixes Gregorian seasons: a Ramadan peak is an observance effect, never a weather or school-year one. It also needs a complete `YYYY-MM-DD` — items dated only to a year or month land in `imprecise_date_count` and are **absent from the bars, not zero** (99% of articles and 82.9% of publications convert). Lunar dates do not exist for `references`.

.agents/skills/iwac-mcp/references/tools-by-phase.md

@@ -63,7 +63,7 @@ Primary search tool for the 12,287 newspaper articles.
 - `newspaper` (optional): substring match
 - `subject` (optional): substring match on the pipe-separated curated tags
 - `date_from` / `date_to` (optional): `YYYY-MM-DD` or `YYYY` (day precision)
-- `hijri_month` / `hijri_year` (optional) *(v1.3.0+)*: Islamic (Umm al-Qura) lunar date. `hijri_month` takes 1–12 or a name in either transliteration (`Ramadan`, `Chaabane`, `Chawwal`, `Dhou al-hijja`), accent- and case-folded; a misspelling errors with `valid_values`. Matches only articles with a complete `YYYY-MM-DD` (98.9%). This is how you read the items behind a `granularity="lunar_month"` peak — and it beats keyword-searching an observance name, which finds items *mentioning* it rather than items *published during* it.
+- `hijri_month` / `hijri_year` (optional) *(v1.3.0+)*: Islamic (Umm al-Qura) lunar date. `hijri_month` takes 1–12 or a name in either transliteration (`Ramadan`, `Chaabane`, `Chawwal`, `Dhou al-hijja`), accent- and case-folded; a misspelling errors with `valid_values`. Matches only articles with a complete `YYYY-MM-DD` (99%). This is how you read the items behind a `granularity="lunar_month"` peak — and it beats keyword-searching an observance name, which finds items *mentioning* it rather than items *published during* it.
 - `with_description` (optional, boolean): include each article's ~500-char AI abstract (`description_ai`) — ~125 tokens/row, pair with limit ≤ 10
 - `limit` (default 20, max 100 — 10 and 25 with `with_description`, since 100 rows carrying abstracts overrun the client's tool-result cap), `offset`
 - Returns: id, title, author, newspaper, country, date, **hijri_date** (`1440-09-15`, v1.3.0+), subject, spatial, language, **polarity**, **centrality**, **subjectivity**, url
@@ -215,12 +215,12 @@ Counts of matching items per year (or month) — one call replaces paging throug
 
 #### The Islamic calendar *(v1.3.0+)*
 
-**Reach for `granularity="lunar_month"` for any observance question.** It pools every year into the twelve lunar months, and it is the only bucket a Gregorian axis structurally cannot produce: the Hijri year drifts ~11 days annually, so across 1961–2025 each observance smears over all twelve Gregorian months and disappears. Measured over the 12,220 fully-dated articles, the archive's rhythm is unmistakable — **Ramadan +72 %, Dhu al-Hijja +70 %** (hajj and Tabaski), **Shawwal +44 %** (Korité, 1 Shawwal) against an even split, with the six ordinary months 24–35 % below it. **Rabi' I is flat (−5 %)**, so Maouloud is *not* treated as a news event the way the others are — a finding in its own right.
+**Reach for `granularity="lunar_month"` for any observance question.** It pools every year into the twelve lunar months, and it is the only bucket a Gregorian axis structurally cannot produce: the Hijri year drifts ~11 days annually, so across 1961–2025 each observance smears over all twelve Gregorian months and disappears. Measured over the 13,261 fully-dated articles, the archive's rhythm is unmistakable — **Ramadan +74 %, Dhu al-Hijja +68 %** (hajj and Tabaski), **Shawwal +42 %** (Korité, 1 Shawwal) against an even split, with the six ordinary months 24–32 % below it. **Rabi' I is flat (−5 %)**, so Maouloud is *not* treated as a news event the way the others are — a finding in its own right.
 
 - `lunar_month` implies `calendar="hijri"`; you do not have to pass both (and `lunar_month` + `calendar="gregorian"` is refused as incoherent).
 - `calendar="hijri"` with `granularity="year"` or `"month"` gives a Hijri *time series* instead (`1440`, `1440-09`).
 - Keys are zero-padded month numbers (`"01"`…`"12"`) so they sort; `month_labels` maps them to names — use it rather than hard-coding a transliteration.
-- **Precision.** Lunar dates need a complete `YYYY-MM-DD`. Items dated only to a year or month (or a `1981-04/1981-06` range) appear in `imprecise_date_count` and are **absent from the distribution, not zero** — 98.9 % of articles and 82.9 % of publications convert, so the gap is small but must be disclosed when a count carries an argument.
+- **Precision.** Lunar dates need a complete `YYYY-MM-DD`. Items dated only to a year or month (or a `1981-04/1981-06` range) appear in `imprecise_date_count` and are **absent from the distribution, not zero** — 99 % of articles and 82.9 % of publications convert, so the gap is small but must be disclosed when a count carries an argument.
 - **Not seasonality.** Pooling by lunar month deliberately mixes Gregorian seasons; a Ramadan peak is an observance effect, never a weather or school-year one.
 - **Not available on `references`** — an academic imprint date has no meaningful lunar reading, so asking returns `{error, note}` naming the subsets that do carry lunar dates (articles, publications, documents, audiovisual, images).
 - **Converter.** Umm al-Qura, precomputed in the dataset pipeline with `hijridate` — the same converter (and therefore the same buckets) as the on-this-day block on islam.zmo.de. This matters: ICU/`Intl` disagrees with it on **75 % of this collection's pre-2000 dates**, though on only 0.86 % of the *month* assignments, so month-level aggregates are robust while day-level labels are not.

CITATION.cff

@@ -26,8 +26,8 @@ identifiers:
     value: 10.5281/zenodo.21805837
     description: "Concept DOI for all versions of the software."
 license: MIT
-version: 3.5.0
-date-released: "2026-08-25"
+version: 3.5.1
+date-released: "2026-09-03"
 keywords:
   - Model Context Protocol
   - MCP server

README.md

@@ -150,8 +150,8 @@ rather than JSON.
 `granularity="lunar_month"` it pools every year into the twelve lunar months —
 the one bucket a Gregorian axis structurally cannot produce, because the Hijri
 year drifts ~11 days annually and so smears each observance across all twelve
-Gregorian months. Over the 12,220 fully-dated articles the archive's rhythm is
-plain: Ramadan +72%, Dhu al-Hijja +70% (hajj and Tabaski) and Shawwal +44%
+Gregorian months. Over the 13,261 fully-dated articles the archive's rhythm is
+plain: Ramadan +74%, Dhu al-Hijja +68% (hajj and Tabaski) and Shawwal +42%
 (Korité) against an even split, while Rabi' I — Maouloud — sits flat. `search_articles`
 and `search_publications` take `hijri_month` (1–12 or a name in either
 transliteration) and `hijri_year` to read the items behind a peak. The lunar
@@ -242,15 +242,15 @@ Machine-readable metadata lives in [CITATION.cff](CITATION.cff) — GitHub's
 **Cite this repository** button (sidebar) renders it as APA or BibTeX with the
 current version filled in. In text:
 
-> Madore, F. (2026). *IWAC MCP Server* (Version 3.5.0) [Computer software].
+> Madore, F. (2026). *IWAC MCP Server* (Version 3.5.1) [Computer software].
 > Zenodo. https://doi.org/10.5281/zenodo.21805837
 
 ```bibtex
 @software{madore_iwac_mcp_server,
   author    = {Madore, Frédérick},
   title     = {{IWAC MCP Server}},
   year      = {2026},
-  version   = {3.5.0},
+  version   = {3.5.1},
   publisher = {Zenodo},
   doi       = {10.5281/zenodo.21805837},
   url       = {https://github.com/fmadore/iwac-mcp-server},

mcpb/manifest.json

@@ -2,7 +2,7 @@
   "manifest_version": "0.3",
   "name": "iwac-mcp-server",
   "display_name": "Islam West Africa Collection (IWAC)",
-  "version": "3.5.0",
+  "version": "3.5.1",
   "description": "Read-only access to the Islam West Africa Collection via Hugging Face datasets.",
   "long_description": "Search and analyse the Islam West Africa Collection (IWAC): ~12,000 newspaper articles, 1,500 Islamic publications (searchable by keyword, subject, series, and table of contents), 4,700 index entries, plus audiovisual recordings, fieldwork photographs, archival documents, and academic references. Includes AI sentiment analysis (polarity/centrality/subjectivity, scored independently by gpt-5-6-luna, mistral-small-2603, deepseek-v4-flash-0731, gemma-4-31b-it and qwen3-8-27b), per-article AI abstracts, and per-year coverage timelines. All matching is accent- and case-insensitive.\n\n**No API key is required for the core tools** (cross-subset search/fetch, keyword search, filtering, statistics, timelines, item details). Three optional semantic-search tools use Gemini embeddings — over all articles, the publication series with tables of contents, and the photographs (cross-modal: describe what an image shows) — and need a free Google/Gemini API key; these are disabled by default.\n\nOn first use the server downloads ~250 MB of parquet data from Hugging Face into a local cache.",
   "author": {

mcpb/package.json

@@ -1,6 +1,6 @@
 {
   "name": "iwac-mcp-server-bundle",
-  "version": "3.5.0",
+  "version": "3.5.1",
   "description": "Node MCP server for the Islam West Africa Collection, packaged as a Claude Desktop extension (.mcpb).",
   "private": true,
   "type": "module",

mcpb/scripts/make-fixtures.mjs

@@ -425,8 +425,14 @@ async function main() {
     // range) are left NULL — an imprecise date has no lunar day, and that
     // absence is a case the tools have to report rather than plot.
     if (HIJRI_SUBSETS.includes(subset)) {
+      // DOUBLE, not BIGINT, because that is what the real parquet stores — and
+      // the difference is not cosmetic. `CAST(3 AS VARCHAR)` is '3' but
+      // `CAST(3.0 AS VARCHAR)` is '3.0', so a fixture typed BIGINT renders
+      // every Hijri bucket and date correctly no matter how the SQL is written,
+      // and cannot reproduce a whole class of formatting bug that production
+      // hits on every row. It hid exactly that once already.
       for (const col of ["hijri_year", "hijri_month", "hijri_day"]) {
-        await conn.run(`ALTER TABLE ${table} ADD COLUMN "${col}" BIGINT`);
+        await conn.run(`ALTER TABLE ${table} ADD COLUMN "${col}" DOUBLE`);
       }
       for (const [greg, [hy, hm, hd]] of Object.entries(HIJRI)) {
         await conn.run(

mcpb/src/tools/_shared.ts

@@ -619,15 +619,32 @@ export function resolveHijriMonth(value: string | undefined): { n?: number; err?
 /** The three precomputed columns, as a `requires` guard for column descriptors. */
 export const HIJRI_COLS = ["hijri_year", "hijri_month", "hijri_day"];
 
+/**
+ * Render one Hijri column as a bucket/date part.
+ *
+ * The three columns are stored as DOUBLE in the parquet, so the INTEGER cast is
+ * load-bearing, not defensive: `CAST(3.0 AS VARCHAR)` is `'3.0'`, and DuckDB's
+ * `lpad` TRUNCATES a string that is already longer than the target width rather
+ * than leaving it alone — so padding it to 2 yielded `'3.'`, and an unpadded
+ * year yielded `'1440.0'`. Both read as plausible until you compare them with
+ * something: a month bucket matched no `month_labels` key, and every single
+ * `hijri_date` was malformed. Anything reading these columns as text must go
+ * through here.
+ */
+export const hijriPart = (column: string, pad = 2): string => {
+  const int = `CAST(CAST(${q(column)} AS INTEGER) AS VARCHAR)`;
+  return pad ? `lpad(${int}, ${pad}, '0')` : int;
+};
+
 /**
  * The lunar date as one `1440-09-15` string, mirroring how `pub_date` reads.
  * Numeric parts rather than a month name: the name would have to be a 12-branch
  * CASE in SQL that then drifts from HIJRI_MONTHS, and the model already has that
  * table from `month_labels`.
  */
 export const HIJRI_DATE_EXPR =
-  `CASE WHEN "hijri_year" IS NULL THEN NULL ELSE CAST("hijri_year" AS VARCHAR) || '-' || ` +
-  `lpad(CAST("hijri_month" AS VARCHAR), 2, '0') || '-' || lpad(CAST("hijri_day" AS VARCHAR), 2, '0') END`;
+  `CASE WHEN "hijri_year" IS NULL THEN NULL ELSE ${hijriPart("hijri_year", 0)} || '-' || ` +
+  `${hijriPart("hijri_month")} || '-' || ${hijriPart("hijri_day")} END`;
 
 /**
  * Guard for every Hijri-aware code path: the columns are written by the

mcpb/src/tools/stats.ts

@@ -9,6 +9,7 @@ import {
   DEFAULT_SENTIMENT_MODEL,
   errorResult,
   HIJRI_MONTHS,
+  hijriPart,
   keywordFilter,
   likeFilterIfExists,
   pipeValueFilterIfExists,
@@ -467,15 +468,20 @@ export function registerStatsTools(server: Server): void {
       // Gregorian buckets slice the ISO string; Hijri buckets read the
       // precomputed columns (post-processing/calculate_hijri_dates.py), which
       // are NULL for any date too imprecise to carry a lunar day.
+      //
+      // Both parts go through `hijriPart`, which carries the DOUBLE->INTEGER
+      // cast these columns need before they can be read as text (see there).
+      const hijriYear = hijriPart("hijri_year", 0);
+      const hijriMonth = hijriPart("hijri_month");
       let bucketExpr: string;
       if (!hijri) {
         bucketExpr = `NULLIF(substr(CAST(pub_date AS VARCHAR), 1, ${granularity === "month" ? 7 : 4}), '')`;
       } else if (granularity === "lunar_month") {
-        bucketExpr = `lpad(CAST("hijri_month" AS VARCHAR), 2, '0')`;
+        bucketExpr = hijriMonth;
       } else if (granularity === "month") {
-        bucketExpr = `CAST("hijri_year" AS VARCHAR) || '-' || lpad(CAST("hijri_month" AS VARCHAR), 2, '0')`;
+        bucketExpr = `${hijriYear} || '-' || ${hijriMonth}`;
       } else {
-        bucketExpr = `CAST("hijri_year" AS VARCHAR)`;
+        bucketExpr = hijriYear;
       }
       const groupSel = groupBy ? `, ${q(groupBy)} AS grp` : "";
       // `c_dated` splits the NULL-bucket rows: on the Hijri calendar a missing

mcpb/test/fixture-server.test.mjs

@@ -492,6 +492,19 @@ await call("get_temporal_distribution", { granularity: "lunar_month" }, {
     // Keys are zero-padded so they sort; names ride along so the model and the
     // chart never hard-code a transliteration.
     if (p.month_labels?.["09"] !== "Ramadan") return `month_labels wrong: ${JSON.stringify(p.month_labels)}`;
+    // The two key sets are a CONTRACT, not a coincidence: the chart looks each
+    // bucket up in month_labels, so a bucket key that misses is not an error
+    // anywhere — it is a bar that silently never draws. Assert the label set is
+    // exactly the twelve canonical keys, and that every bucket is one of them.
+    // (Shipped once as '3.' — DOUBLE columns cast straight to VARCHAR — which
+    // dropped nine of twelve months from the lunar chart without failing a
+    // single test.)
+    const canonical = Array.from({ length: 12 }, (_, i) => String(i + 1).padStart(2, "0"));
+    const labelKeys = Object.keys(p.month_labels ?? {}).sort();
+    if (labelKeys.length !== 12 || labelKeys.some((k, i) => k !== canonical[i]))
+      return `month_labels keys must be exactly 01..12, got ${JSON.stringify(labelKeys)}`;
+    const strays = Object.keys(d).filter((k) => !(k in (p.month_labels ?? {})));
+    if (strays.length) return `lunar buckets absent from month_labels (they would not render): ${JSON.stringify(strays)}`;
     if (!String(p.note ?? "").includes("pooled across all Hijri years"))
       return "lunar_month must disclose that it pools years and is not seasonality";
     return null;