fix: tolerate non-UTF-8 soffice output in doc/ppt conversion

[huuufu]

Summary

convert_office_doc() — shared by partition_doc() and partition_ppt() — decoded soffice stdout and stderr with a strict UTF-8 .decode() at three sites, used only for logging and to check whether stdout was empty (which drives the retry loop and the final failure check). LibreOffice echoes the input path in the console encoding, which on Windows is the locale codepage rather than UTF-8, so a name or path with multi-byte characters makes the decode raise and aborts a conversion that had already succeeded:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8a in position 32: invalid start byte

The traceback in #3652 is on the first stdout decode. The stderr site is reachable and broken the same way when a conversion fails, and sits outside the surrounding try entirely.

Fixing the child process's output encoding is not an option — @Snowman-s confirmed on the issue that PYTHONIOENCODING=utf-8:surrogateescape and sys.stdout.reconfigure() both leave the exception unchanged.

Fix

All three sites go through one _decode_soffice_output() helper decoding with errors="backslashreplace".

@scanny weighed three options on the issue and voted for option 1, try/except UnicodeDecodeError falling back to str(bytes). A codec error handler gets the same tolerance in one expression with no rarely-exercised second branch, and escapes only the offending bytes rather than turning a perfectly good line into a repr. Option 2 — detect Windows and decode with the locale encoding — would render the filename correctly when the guess is right, but the console output codepage need not match locale.getpreferredencoding() — anything that ran chcp upstream moves one and not the other — and a wrong guess is silent mojibake rather than a visible escape. It layers cleanly on top of this later if you want it.

backslashreplace rather than replace because U+FFFD is not encodable in the very codepages this bug is about: a logging.FileHandler whose stream uses cp932 or cp936 cannot encode it and drops the whole record, so replace would stop the abort and then throw away the diagnostic. backslashreplace keeps the line pure ASCII and shows the actual bytes — closer to what option 1 was reaching for.

One behavior change worth flagging: stdout consisting entirely of undecodable bytes now yields a non-empty string, so with returncode == 0 it no longer trips the empty-stdout heuristic the code uses to detect a silent failure. Real soffice output carries ASCII scaffolding around the path, so this needs a degenerate payload, but it is a real narrowing of that heuristic — one option 1 shares, since str() of non-empty bytes is never empty either. The b"" that the heuristic actually watches for still decodes to "".

Testing

No LibreOffice needed — stub subprocess.run with the bytes soffice emits on a Japanese console:

from unittest import mock
from unstructured.partition.common import common

out = mock.Mock(returncode=0, stdout="convert 文章.doc -> 文章.docx".encode("cp932"), stderr=b"")
with mock.patch.object(common.subprocess, "run", return_value=out):
    common.convert_office_doc("文章.doc", "fake-directory", target_format="docx")

Raises UnicodeDecodeError on main; returns normally here.

test_convert_office_doc_survives_non_utf8_soffice_output is parametrized over stdout and stderr so both decode paths are covered, and asserts the escaped bytes reach the log and that the record is ASCII-encodable — it fails on main, and also fails if the helper is switched to errors="replace".

test_common.py: 63 passed / 2 failed here, 61 passed / 4 failed with the same file on main — the extra two are the new test's stdout and stderr cases. The other two fail identically on both (FileNotFoundError: soffice command was not found; no LibreOffice on this machine). ruff check . passes repo-wide and ruff format --check passes on both changed files (repo-wide it exits non-zero on the example-docs/umlauts-non-utf8.md fixture, identically on main).

Not verified against a real soffice: the root cause is from @Snowman-s's and @scanny's analysis on the issue, and the CP932 payload is my own reconstruction of it.

Resolves #3652

[huuufu]

Prior art: #3830 patched the first of these three sites in 2024; the open question there was a 1-page .doc that reproduces this in a unittest. As far as I can determine, no checked-in fixture can: the undecodable bytes are not in the document, they are the input path as soffice echoes it back in the console's encoding. The failing files in #3652 were empty docs and renaming to an ASCII name clears the error; the failing file shared in #2449 partitioned fine on a Mac; and under a UTF-8 locale on Ubuntu 24.04 (LibreOffice 24.2.7, what CI installs) I verified soffice echoes a multi-byte path as valid UTF-8, so a committed fixture converts cleanly with or without this fix. The test here instead stubs subprocess.run with the CP932 bytes a Japanese Windows console emits, exercising both the stdout and stderr sites; it fails on main unpatched. Happy to also add a multi-byte-named .doc through partition_doc() as a live-soffice smoke test if that coverage is wanted — it just can't fail on this particular bug.