fix: stop encoding detection from returning mojibake

[BetterAndBetterII]

Summary

detect_file_encoding() accepted a high-confidence charset_normalizer.detect() result even when that codec was outside COMMON_ENCODINGS. Those guesses (johab, windows-1250) still decoded, so the UnprocessableEntityError guard never fired and the mojibake became element text. That hits everything that reads text through read_txt_file() (partition_text, partition_md, partition_html).

A second path in the same function walked COMMON_ENCODINGS and took the first codec that decoded. iso_8859_1 maps every byte, so later CJK codecs on that list were unreachable.

Fix

• Reject a detected codec that fails validate_encoding() (already present, previously unused).
• When detection is inconclusive or out of set, rank candidates with from_bytes(..., cp_isolation=COMMON_ENCODINGS) instead of first-success decode.

Tests

• detect_file_encoding recovers the issue's shift_jis and iso_8859_1 samples via filename and file.
• Fallback path with mocked inconclusive detect() reaches shift_jis instead of stopping at iso_8859_1.
• partition_text recovers the same three samples.

Fixes #4466

[huuufu]

Thanks for picking this up so quickly. I had prototyped the same approach while drafting #4466, so I already had a corpus and harness on hand and ran this branch against it: 28 short/medium plain-text samples in their real-world legacy encodings, f879f683 vs main (d68ab0fd), charset_normalizer 3.5.1, Python 3.12.

The fixes are real: the issue's shift_jis and latin-1 samples come back correct, and so do Spanish latin-1, Hebrew cp1255 and Arabic iso-8859-6 samples that are mojibake on main today. Raising on actual binary junk instead of returning iso-8859-1 noise is also an improvement. But these rows, all correct on main, come back worse:

sample (correct on main today)	on this branch
ru cp1251	raises UnprocessableEntityError
el cp1253	raises
th cp874	raises
ar cp1256	raises
pl cp1250 / cs cp1250	raises
en cp1252 (curly quotes, em dash)	raises
ko cp949 (UHC extension syllables)	raises, or gb18030 mojibake, depending on the sample
ja cp932 (IBM extension kanji 髙, 﨑)	mojibake: 郄粼様と粼山様…
tr cp1254 (longer sample)	mojibake: Iðdýr ovasý…

The mechanism is the same in every row: detect() returns an encoding that decodes the file correctly (confidence 1.0 everywhere except the cp932 row's 0.875), but that codec is not in COMMON_ENCODINGS, so the gate rejects the right answer. From there it either raises — nothing in-set survives the cp_isolation pass, so .best() returns None — or a wrong in-set codec wins: shift_jis_2004 claims the cp932 bytes and silently swaps the extension kanji (髙→郄, 﨑→粼).

import os, tempfile
from unstructured.partition.text import partition_text

samples = [
    ("ru cp1251", "Привет, как дела? Это тестовое сообщение для проверки.", "cp1251"),
    ("th cp874 ", "สวัสดีครับ ยินดีต้อนรับสู่ห้องประชุมของเรา", "cp874"),
    ("ja cp932 ", "髙﨑様と﨑山様が来社されました。", "cp932"),
    ("ko cp949 ", "점심 메뉴는 똠얌꿍으로 예약했습니다.", "cp949"),
]
for name, text, enc in samples:
    fd, path = tempfile.mkstemp(suffix=".txt"); os.close(fd)
    with open(path, "wb") as f:
        f.write(text.encode(enc))
    try:
        got = partition_text(filename=path)[0].text
        print(name, "->", "ok" if got == text else f"MOJIBAKE {got!r}")
    except Exception as e:
        print(name, "->", f"RAISED {type(e).__name__}")
    finally:
        os.unlink(path)

On this branch:

ru cp1251 -> RAISED UnprocessableEntityError
th cp874  -> RAISED UnprocessableEntityError
ja cp932  -> MOJIBAKE '郄粼様と粼山様が来社されました。'
ko cp949  -> RAISED UnprocessableEntityError

On main all four print ok.

One measurement that may help: changing only the .best() is None branch to fall back to the detected encoding when detection was conclusive — raising only when detection failed too, which still covers binary junk — fixed all seven raise rows on my corpus and did not un-fix anything this branch fixes. None from the constrained pass turns out to be a decent signal that the file's encoding family just isn't in the supported set, and at that point the unconstrained detection is the best hypothesis available.

The mojibake rows are harder and I don't have a rule that solves them; on my corpus the same codec name at the same confidence can be the right answer for one input and the wrong one for another, so no membership check on the name separates them (windows-1250 at 1.0 is correct for the pl/cs rows and incorrect for latin-1 French; cp932 at 1.0 is correct for a shift_jis sample and incorrect for a big5 one — and koi8-r comes back as shift_jis_2004 at 1.0, which is in the set, so the gate passes it on both trees). I also tried deciding these on chaos/coherence margins, and the numbers order the wrong way: wrong cp775-for-Spanish beats right latin-1 by a larger coherence margin (+0.107) than right cp1254-for-Turkish beats wrong latin-1 (+0.077). That residue looks like the policy half of #4466 rather than something this patch has to settle — the raise rows are the part that seems clearly fixable. Happy to share the full corpus and harness if useful.