Skip to content

Commit 71b5e5d

Browse files
committed
docs(sentiment): Qwen3.8 full-corpus results — 89.6% first pass, one recurring fault
All 12,251 eligible articles annotated 2026-08-17/18 at medium, prompt #d14ace9ac192, offline on Festus across three shards (one L40S, two H100). 10,975 valid on the first pass; 1,276 rejected, ~99% of them the same cross-field fault (null subjectivite beside a non-null centralite). The rate held at 10.6/9.9/10.8% across three independent slices and two hardware configurations, so it is the model on this task rather than the slice or the GPU. Timeouts were ~1% on both card types, which retires the earlier guess that the H100's shorter latency tail would buy coverage; it buys wall clock and nothing else. Two findings for the promotion decision. Polarite is barely negative — 218 of 10,975 (2.0%) — which wants checking against the live panel on the same articles. And the null subjectivite count matches Non aborde exactly (276), so the cross-field rule is obeyed perfectly where it is obeyed at all.
1 parent a8a5987 commit 71b5e5d

1 file changed

Lines changed: 41 additions & 33 deletions

File tree

AI_sentiment_analysis/README.md

Lines changed: 41 additions & 33 deletions
Original file line numberDiff line numberDiff line change
@@ -364,42 +364,50 @@ the panel's requested `medium` is a rung the model has rather than one it gets
364364
rounded up to. Weights load in ~150 s; guided decoding held, so no response
365365
needed unfencing.
366366

367-
**The open question is not reasoning depth but validity**, and a 200-article
368-
offline pass on 2026-08-17 measured it (2× L40, `medium`, prompt `#d14ace9ac192`,
369-
70 min, 172 articles/hour):
367+
**The open question is not reasoning depth but validity.** A full-corpus pass
368+
ran 2026-08-17/18 — all 12,251 eligible articles, `medium`, prompt
369+
`#d14ace9ac192`, offline on Festus across three shards (one L40S, two H100).
370370

371-
| Pass | Annotated | Failed |
372-
|---|---|---|
373-
| First | 175 / 200 | 25 (12.5%) — 24 rule-invalid, 1 timeout |
374-
| After one retry | 192 / 200 | **8 (4.0%)**, all rule-invalid |
371+
| | |
372+
|---|---|
373+
| Annotated, first pass | **10,975 / 12,251 (89.6%)** |
374+
| Rejected | 1,276 (10.4%) — ~99% one fault, ~1% timeouts |
375+
| Throughput | 361 articles/h on 2× L40S, 540–576 on one H100 |
376+
| Wall clock | ~7 h per 4,084-article shard on H100 |
375377

376-
The failure is always the same: a null `subjectivite_score` beside a non-null
378+
The fault is always the same: a null `subjectivite_score` beside a non-null
377379
centralité, which the schema's cross-field validator rejects. Guided decoding
378-
constrains shape, never logic. **Most of it is transient** — 17 of 25 cleared on
379-
a second pass, so the operational cost is a retry pass, not lost articles. When
380-
a call fails this way the whole result is replaced by `ERREUR_ANALYSE`
381-
placeholders, so a rejected article yields nothing at all, not a partial record.
382-
383-
The valid annotations use the scales properly — centralité spread `Très central`
384-
110 / `Marginal` 23 / `Central` 19 / `Secondaire` 17 / `Non abordé` 6, and the
385-
six null subjectivité scores line up exactly with the six `Non abordé` articles,
386-
which is the rule being obeyed rather than broken. One thing for any write-up:
387-
polarité came out heavily positive-or-neutral (80 / 78) with only 6 `Négatif`
388-
out of 175, which should be checked against what the live panel assigned on the
389-
same articles before Qwen is trusted on that dimension.
390-
391-
**Do not "fix" the residual by dropping to `low`.** Run on the 8 persistently
392-
failing articles, `low` returned valid output for 6 of them — but all six got
393-
*identical* `subjectivite_score` (`Très objectif`) and *identical* `polarite`
394-
(`Neutre`), where across 175 articles at `medium` those labels occur 11% and 45%
395-
of the time. Centralité did vary, so the model is not simply emitting a constant
396-
record; it is the two contested fields that flatten. On six articles that is
397-
suggestive rather than conclusive, but the shape of it is the failure this
398-
project has met before: an unusable answer that is indistinguishable from a real
399-
one once stored, which is why the 2026-07 `ocr_quality` column was reverted. A
400-
visible 4% loss is worth more than an invisible 4% of default answers. Mixing
401-
depths would also forfeit the comparability that made this candidate
402-
interesting, since a genuine `medium` rung was its main argument.
380+
constrains shape, never logic. The rate held at **10.6% / 9.9% / 10.8%** across
381+
three independent slices and two hardware configurations, so it is a property of
382+
the model on this task, not of the slice or the GPU. Most of it is transient —
383+
on the 200-article pilot one retry pass recovered two thirds, 12.5% to 4.0%.
384+
385+
Distributions over the 10,975 valid annotations:
386+
387+
```
388+
centralité Très central 7,137 · Central 1,613 · Marginal 1,011 · Secondaire 938 · Non abordé 276
389+
polarité Neutre 5,350 · Positif 4,963 · Non applicable 277 · Négatif 214 · Très positif 167 · Très négatif 4
390+
subjectivité Plutôt objectif 6,980 · Plutôt subjectif 1,803 · Très objectif 900 · Très subjectif 734 · Mixte 282 · null 276
391+
```
392+
393+
Two things to weigh before promotion. **Polarité is barely negative** — 218 of
394+
10,975 (2.0%) across a press corpus spanning decades of contested public
395+
argument — and that needs checking against what the live panel assigned on the
396+
same articles before Qwen is trusted on the dimension. In its favour, the null
397+
subjectivité count (276) matches `Non abordé` (276) exactly: the cross-field
398+
rule obeyed perfectly where it *is* obeyed.
399+
400+
**Do not "fix" the residual by dropping to `low`.** Run on the 8 articles that
401+
stayed stuck through two passes at `medium`, `low` returned valid output for 6 —
402+
but all six got *identical* `subjectivite_score` (`Très objectif`) and
403+
`polarite` (`Neutre`), labels occurring 8% and 49% of the time at `medium`.
404+
Centralité still varied, so it is the two contested fields flattening rather
405+
than a constant record. On six articles that is suggestive, not conclusive, but
406+
the shape is the failure this project has met before: an unusable answer
407+
indistinguishable from a real one once stored, which is why the 2026-07
408+
`ocr_quality` column was reverted. A visible residual beats invisible defaults,
409+
and mixing depths would forfeit the comparability that made a genuine `medium`
410+
rung this candidate's main argument.
403411

404412
Note also that `xhigh` exceeded the 300 s request timeout on one call in two on
405413
L40s, and that per-call latency at `medium` under `--concurrency 6` has a long

0 commit comments

Comments
 (0)