Skip to content

Commit d131b82

Browse files
committed
Release v3.4.0: serve Qwen's sentiment, and stop implying the panel is evenly covered
qwen3-8-27b joined the generation-2 annotation campaign on 2026-08-25, written from a self-hosted vLLM run rather than a vendor API, and its columns reached the public dataset the same morning. Adding it to SENTIMENT_MODELS is three lines. The rest of this commit exists because Qwen is the first panel member that does not cover the corpus. It scores 12,098 articles where the other four score 12,298. The 200-article gap is deliberate and final rather than a run to repair: each was attempted four times and then retired. It is also not missing at random. Measured against Luna's centrality, 6.15% of Marginal articles are unscored against 0.96% of Central and 0.99% of Tres central ones, because the model declines to place a subjectivity label where Islam is peripheral while the prompt licenses declining only where Islam is absent. Two things in the server assumed uniform coverage without ever saying so. agreement.pairwise is measured on the base of articles every listed model scored, which is what makes the pairs comparable with each other. Adding a fifth model therefore changed every pre-existing pairwise number, not only Qwen's: the base fell from 12,298 to 12,098 and unanimity on polarity from 36.5% to 32.5%. The common base is kept, because per-pair bases would not be comparable, and it is disclosed instead. agreement.base states what it counts and agreement.base_caveats names the short member and why. rowsToMap drops the empty key, so a model that answered 200 fewer articles produced a distribution indistinguishable from one that answered all of them. Every model block now carries coverage {polarity, centrality, subjectivity, matched_articles}, summed from the distributions already computed rather than from fresh SQL. Describing a model stays the registry's job. SentimentModel gains an optional caveat, so no call site hardcodes "qwen" or "200". It travels into the payload as model_caveat beside that model's own numbers, and into the agreement block for any model that shrinks its base. The chart said "All N scored the same articles", which was now false. The comparison view captions only a ring falling short of the panel's best coverage, five identical denominators being noise, and swaps that note for one naming the short members and the reduced base. Every figure in the docs was re-measured against the live parquet rather than carried forward. Subjectivity kappa runs 0.16 to 0.52 across the five, where the ceiling is one pair, luna and qwen; centrality 0.45 to 0.74; and on polarity Qwen is now the closest pair to Luna at 0.54, ahead of Gemma's 0.50. Mistral remains the systematic outlier on all three scales and Qwen joined the non-Mistral cluster, so a majority on centrality is still the others outvoting Mistral. Qwen is also the panel's most reluctant to reach for an extreme: 176 Tres positif and 5 Tres negatif against Luna's 425 and 45. gpt-5-6-luna stays the default, and Qwen made that the closest call so far by agreeing with Luna more than any other pair on two of the three scales. The 200-row hole is the answer: promoting it would put a silent gap in every inline polarity this server returns. Both qwen and alibaba resolve. qwen resolves where gemini is still refused, and the difference is the point of the registry: the Qwen line scored generation 2, the Gemini line never did. The fixture leaves Qwen entirely empty on row 104, the low-centrality one, so scored_by_all (5) is smaller than the matched set (6). A fixture where the new model is scored everywhere could not fail when the shrinking base is broken. The live smoke test asserts the shortfall against the real columns, since only live data can prove the server discloses it. Token footprint moves 14,880 to 15,071 of a 16,000 ceiling. 112 of that 191 is v3.3.0's instructions growth, which was never re-baselined; this change costs 79 on get_sentiment_distribution. The baseline is re-stamped at the current figure. The combined scoping tool and multi-term keyword probing that v3.3.0's note assigned to v3.4.0 remain open.
1 parent 3539885 commit d131b82

17 files changed

Lines changed: 335 additions & 109 deletions

.agents/skills/iwac-mcp/SKILL.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -180,7 +180,7 @@ Thirteen tools carry their own interactive chart on hosts that support MCP Apps
180180
**Actions:**
181181
1. Cross-reference MCP findings across subsets (articles vs. publications vs. references vs. documents vs. index)
182182
2. Use `get_sentiment_distribution` with `subject` filter to compare topic-specific sentiment against the collection baseline (e.g., `subject="Laïcité", country="Burkina Faso"` vs. the whole BF corpus)
183-
3. Before any sentiment claim carries weight, re-run it with `model="all"`. All four models agree unanimously on polarity for only **36%** of the corpus (4,488 of the 12,298 articles they all scored); where your slice diverges further, say so rather than quoting one model
183+
3. Before any sentiment claim carries weight, re-run it with `model="all"`. All five models agree unanimously on polarity for only **32%** of the corpus (3,929 of the 12,098 articles they all scored); where your slice diverges further, say so rather than quoting one model
184184
4. Use `search_articles` results (which include sentiment inline) to build topic-specific sentiment tables without extra calls
185185
5. Use `get_temporal_distribution` (optionally `group_by=country|newspaper`) to verify a claimed trend over time and compare trajectories across countries or outlets without paging. For an observance claim, `granularity="lunar_month"` is the test: it pools every year into the twelve lunar months, so a Ramadan or Dhu al-Hijja effect either stands above the even split or it does not
186186
6. Test whether a theme you have named is really distinct: `get_cooccurrence(field="subject")` shows what it is always discussed alongside, and `get_similar_items` on a key article shows whether your "finding" is one story reprinted across several outlets (scores ≥0.85)
@@ -215,7 +215,7 @@ Thirteen tools carry their own interactive chart on hosts that support MCP Apps
215215

216216
**For null results:** `Search for [term] in [tool] with [parameters] returned 0 results.`
217217

218-
**For AI sentiment findings:** Four models scored the corpus independently — `gpt-5-6-luna` (the default), `mistral-small-2603`, `deepseek-v4-flash-0731` and `gemma-4-31b-it`; the result keys are `polarity`, `centrality`, `subjectivity`. **Name the exact model in any reported figure** — the tools echo the model id precisely so a number can be attributed, and the generation-1 models these replaced are refused rather than substituted (`gemini` included: Google's generation-2 member is Gemma, a different model line, so the handle errors instead of resolving). Treat `subjectivity` as the ordinal **label** it is (Très objectif → Très subjectif), never as a percentage, and prefer not to report it at all: it is much less reliable than the other two scales. When comparing topics or countries, use `get_sentiment_distribution` with a `subject` filter for aggregate data, or tabulate the sentiment columns from `search_articles` results. See `references/biases-and-limitations.md` §4 before any sentiment claim carries an argument.
218+
**For AI sentiment findings:** Five models scored the corpus independently — `gpt-5-6-luna` (the default), `mistral-small-2603`, `deepseek-v4-flash-0731`, `gemma-4-31b-it` and `qwen3-8-27b`; the result keys are `polarity`, `centrality`, `subjectivity`. They do **not** all cover the same articles — `qwen3-8-27b` scores 12,098 where the rest score 12,298, a deliberate gap concentrated on articles peripheral to Islam — so read each model's `coverage` before comparing counts, and never present Qwen's shortfall as a defect. **Name the exact model in any reported figure** — the tools echo the model id precisely so a number can be attributed, and the generation-1 models these replaced are refused rather than substituted (`gemini` included: Google's generation-2 member is Gemma, a different model line, so the handle errors instead of resolving). Treat `subjectivity` as the ordinal **label** it is (Très objectif → Très subjectif), never as a percentage, and prefer not to report it at all: it is much less reliable than the other two scales. When comparing topics or countries, use `get_sentiment_distribution` with a `subject` filter for aggregate data, or tabulate the sentiment columns from `search_articles` results. See `references/biases-and-limitations.md` §4 before any sentiment claim carries an argument.
219219

220220
**For the evidence-base ledger (open every synthesis with one):** Report three tiers separately — items **read in full**, items **triaged on an AI abstract/snippet only**, and matches **surveyed by count only**. Example: `Evidence base: 4 articles + 1 document read in full; 18 articles triaged on description_ai; 1,909 keyword matches surveyed by count.` This keeps full-text evidence visibly distinct from AI-derived (abstract/snippet) evidence, which carries weaker evidential status.
221221

@@ -242,7 +242,7 @@ only items that happen to mention it.
242242
2. **Always disclose the francophone bias.** ~96% French-language sources specifically reflect Western-educated Muslim perspectives (those who followed French-speaking, secular, or Christian school curricula). *Arabisants* -- leaders trained in madrasas who use Arabic or national languages -- are underrepresented as direct voices, though the French press regularly reports on their activities.
243243
3. **Always disclose the Niger/Nigeria gap.** Niger has thin coverage (one newspaper, 2018+) with inconsistent subject tagging. Nigeria has no press articles at all (audiovisual only). These gaps must be stated in any cross-country analysis.
244244
4. **Always distinguish source types.** MCP tool outputs, AI sentiment labels, AI abstracts (`description_ai`), and OCR text have different evidential status.
245-
5. **AI sentiment is interpretive, not factual.** The labels are analytical signals, not ground truth — four models scored the same articles and agree unanimously on polarity for only ~36% of them. Name the model behind any figure, and run `get_sentiment_distribution(model="all")` before a divergence-sensitive claim. Use topic-specific sentiment (via `subject` filter) rather than whole-corpus baselines when comparing themes.
245+
5. **AI sentiment is interpretive, not factual.** The labels are analytical signals, not ground truth — five models scored the corpus and agree unanimously on polarity for only ~32% of the articles all five reached. Name the model behind any figure, and run `get_sentiment_distribution(model="all")` before a divergence-sensitive claim. Use topic-specific sentiment (via `subject` filter) rather than whole-corpus baselines when comparing themes.
246246
6. **Search incrementally, but call in parallel.** Keep limits low, vary one dimension at a time, and avoid retrieving full OCR text unless needed. Incremental describes what each query asks, not the pace of asking: independent calls go out in a single batch (see **Call Budget**).
247247
7. **Publications are mostly entire issues.** Individual articles within an issue are not separated; use the table of contents where one exists (17 of 25 series) and `get_publication_fulltext` keyword excerpts to localise content inside an issue.
248248
8. **Mind the 1990-91 press-system break.** Pre-1991 articles (~11% of the corpus) come almost entirely from state or single-party organs; the private press only emerges with political liberalisation. Temporal comparisons crossing 1990 compare two different press systems (see biases-and-limitations.md §6).

0 commit comments

Comments
 (0)