Evaluate Qwen3.8-27B for the generation-2 sentiment panel

[fmadore]

Proposal

Evaluate Qwen3.8-27B (Qwen/Qwen3.8-27B,
Apache-2.0, released 2026-08-05) as a member of the generation-2 sentiment panel
in AI_sentiment_analysis/sentiment_core.py (PANEL).

The panel is currently Gemma 4 31B, GPT-5.6 Luna, Mistral Small 4 and
DeepSeek V4 Flash 0731.

Blocked today. Qwen3.8-27B is not on OpenRouter as of 2026-08-14. The
catalog carries only qwen/qwen3.8-2.4t-a95b and qwen/qwen3.8-max from this
generation — both license: other, both at $2.00/$6.00 per 1M, i.e. 10–20× the
panel's cost band and disqualified on the cost-ladder grounds that moved the
panel off gemini-3.6-flash. Since open-weights members must route through
OpenRouter under OPENROUTER_PROVIDER_PREFS (data_collection: "deny"), there
is nothing to pilot until the 27B is listed. This issue is a standing note to
re-check, not a work item ready to start.

Motivation

1. It would be the panel's second genuine middle reasoning level

This is the headline, and it is the one thing no current candidate offers.

PANEL_REASONING asks every member for medium. Only GPT-5.6 Luna actually has
one; Mistral Small 4 (none|high), DeepSeek V4 Flash 0731 (low|high|max) and
Gemma 4 31B (MINIMAL|HIGH) are all rounded up to high by
clamp_thinking_level(). sentiment_core.py calls that out as
"a real limit on comparability [that] belongs in any write-up of the panel
results", and the Gemma swap on 2026-08-14 made it worse — from an even 2/2 split
to 3 of 4 rounded up.

Qwen3.8-27B's model card documents reasoning_effort at low / medium /
xhigh (xhigh is the default), plus an enable_thinking: False toggle. A
member that sits at the requested depth rather than above it moves the panel back
to 2 of 5 at a genuine middle — the first candidate since Gemini 3.5 Flash-Lite
left the panel that improves this number instead of degrading it.

Caveat that could kill the whole motivation: those levels are the upstream
contract. Gemma's graduated thinking turned out to be effectively on/off through
OpenRouter, because the request fans across third-party backends that disagree on
what an effort means (sentiment_core.py:336-347 — medium and high
indistinguishable in latency and reasoning length; one backend reporting
reasoning_tokens: 1 while emitting 3.7k characters of it). The middle level must
be measured through the actual route, not read off the model card. If it
collapses to on/off the way Gemma's did, the main reason to run this pilot is gone.

2. Open weights, and a fifth vendor

Apache-2.0, so the annotations stay re-runnable from archivable weights — with
Mistral Small 4, DeepSeek V4 Flash 0731 and Gemma 4 31B that would be 4 of 5.

Alibaba is also a lab not otherwise in the panel, so no repeat of the
Gemini/Gemma correlated-annotator-error problem that forced Gemma in as a
replacement. Weaker version of the same concern still applies between Qwen and
DeepSeek (two Chinese labs, likely overlapping pretraining corpora); worth a
sentence in any write-up, not a blocker.

3. Size parity holds

27.8B dense = 27.8B active, against the panel's 6.5B (Mistral Small 4) / 13B
(DeepSeek) / 31B (Gemma, dense). It lands next to Gemma rather than widening the
spread — unlike Gemma's own addition, which took the range from a factor of two
to a factor of five.

Verified specs (2026-08-14)

Property	Value	Source
Parameters	27,781.4M, dense	HF repo metadata
Architecture	qwen3_5, 64 layers, hybrid 16 × (3 × (Gated DeltaNet → FFN) → 1 × (Gated Attention → FFN)), hidden 5,120	model card
Context	262,144 native, ~1M with YaRN	model card
Reasoning	reasoning_effort low/medium/xhigh (default xhigh); enable_thinking: False; preserve_thinking on by default	model card
Modality	native vision-language (images + video), task tag image-text-to-text	HF repo metadata
Licence	Apache-2.0	HF repo metadata
Released	2026-08-05 (FP8 variant 2026-08-13)	HF repo metadata
Benchmarks	GPQA Diamond 89.2, Terminal Bench 2.1 73.0, QwenSWEBench 79.0, OSWorld-Verified 84.3	model card

No pricing is quoted here on purpose. It is not on OpenRouter, so there is no
rate to quote, and per AI_sentiment_analysis/README.md:308 cost gets measured
against the credits endpoint rather than inferred. For scale, the entry it would
supersede — qwen3.5-dense / qwen/qwen3.5-27b, already in MODEL_REGISTRY and
TEXT_FULL_MODELS but never piloted — sits at $0.195/$1.56 per 1M, inside the
panel band on input and at the top of it on output.

Note that this is a VLM, not a text model: none of the benchmark strengths
above (coding, agentic computer use, STEM) speak to the task, and part of the
pretraining budget went to vision. Separately, that makes it worth a look for the
multimodal pipelines — but that is a different issue, not this one.

Add as a fifth member, or replace one?

Unlike the Gemma case there is no correlated-vendor argument forcing a
replacement, so adding is the cleaner default:

• No mixed column, no generation-boundary question, no retirement of existing
  properties. Generation 2 gains a voice rather than changing definition.
• Costs one full 12,305-article pass for the new member alone, plus a permanent
  +25% on the cost and wall clock of every future run (analyze_with_all_models
  fans out to all members concurrently, so the slowest member sets the pace —
  see the throughput table; Gemma at ~72 s median already does).

If the panel should stay at four, the honest swap target is DeepSeek V4 Flash
0731 — same open-weights property, and its 31.5 h pass would be the one Qwen
has to beat. But it is also the only member whose medium-rounded-to-high
behaviour is already documented against a full corpus, so replacing it discards a
measured baseline. Decide before implementing, as with #11.

Pre-flight checks (blocking)

1. Is it on OpenRouter at all, under an Apache-2.0-priced endpoint? Currently
   no. Re-run the catalog check before anything else:
   curl -fsS https://openrouter.ai/api/v1/models | grep qwen3.8
2. Endpoint count under require_parameters. The Qwen3.5 122B-A10B lesson
   (sentiment_core.py:284-294, README:99-107): 5 endpoints, 4 after
   structured-output filtering, 104 s median, dropped before annotating anything.
   Gemma cleared this at 16 of 19. A brand-new dense VLM may well land thin.
3. structured_outputs in supported_parameters. generate_structured()
   hard-depends on it, and require_parameters filters on it. The two listed
   Qwen3.8 models both carry it, which is encouraging but not evidence for the 27B.
4. Measure whether medium is real through the route (see caveat above) —
   latency and reasoning length at low / medium / xhigh on the same article,
   several calls. If they are indistinguishable, close this issue.
5. Vendor temperature. The qwen3.5-* entries carry default_temperature=0.7.
   Verify Qwen3.8's own default rather than copying 3.5's forward — per CLAUDE.md
   the value is vendor-owned and a wrong low value is a documented cause of
   looping.

Implementation sketch

• common/llm_registry.py: new ModelOption following the qwen3.5-dense
  pattern — PROVIDER_OPENROUTER, supported_reasoning_efforts set to what the
  endpoint actually accepts, default_temperature from check 5. Add to
  TEXT_OPEN_MODELS; consider whether it supersedes qwen3.5-dense there.
• common/iwac_config.py: new AI_MODEL_ITEMS entry, which requires creating
  the Omeka authority item first (class 244, template 3, item set 267,
  dcterms:type → "Notice d'autorité").
• AI_sentiment_analysis/sentiment_core.py: new PanelMember — HF column key
  qwen3_8_27b, property prefix e.g. qwen3827b. Add a
  PANEL_REASONING_OVERRIDES entry only if rounding is needed; if medium
  holds, it is the first member since Luna to need none, and
  PANEL_REASONING_EFFECTIVE should say so.
• AI_sentiment_analysis/00_setup_properties.py --verify before any upload.
• README updates: the generation-2 panel table, "Why these four", the reasoning
  table, the open-weights/licence table, and both throughput tables.

Acceptance criteria

• Pre-flight checks 1–5 pass (1 and 4 are the likely stopping points)
• 02_pilot_new_panel.py run on a sample of already-annotated articles
• 03_pilot_report.py reviewed — κ(Qwen, each existing member) per dimension,
  plus self-consistency across repeats
• French justification quality spot-checked by hand; the corpus is
  francophone West African press and sentiment_prompt.md requires French
  justifications, so this is the same unproven-register risk as Gemma
• Median call latency measured and compared against the panel (Gemma ~72 s is
  the current ceiling; a fifth member slower than that sets the pace for every
  future run)
• Add-vs-replace decided
• Cost measured against the OpenRouter credits endpoint, not projected

Related

• Evaluate Gemma 4 31B as the sentiment panel's replacement for Gemini 3.5 Flash-Lite #11 — Gemma 4 31B for the Google slot (merged 2026-08-14); its pre-flight
  structure is reused here
• The Qwen3.5 122B-A10B rejection (2026-08-05, sentiment_core.py:284-294) — a
  serving decision, not a quality one, and the reason check 2 is blocking