Fix UnicodeEncodeError that kills the CLI after a successful run - #1282
Merged
Merged
Conversation
On a Windows console the default code page is a legacy one (cp1252 in
Western Europe). Generating a video in French produces U+202F, the narrow
no-break space French typography places before ':' and '!', and Loguru's
own progress lines carry circled digits such as U+2464.
Either character raises UnicodeEncodeError when the CLI prints its result,
which happens *after* the pipeline has finished writing the video:
SUCCESS | task ... finished, generated 1 videos
Traceback (most recent call last):
File "cli.py", line 1495, in run_cli
print(json.dumps({"task_id": task_id, "result": result}, ...))
UnicodeEncodeError: 'charmap' codec can't encode character ' '
The video is fine on disk, but the process exits non-zero and never prints
where the file is, so the run looks like a failure and the path has to be
hunted for under storage/tasks/.
Reconfigure stdout and stderr to UTF-8 at the entry point, before anything
is written. Streams that do not support reconfigure() -- a wrapper, a pipe
under some runners -- are left alone rather than crashing on the fix.
Reproduced and verified on Windows 11, Python 3.13, with
--video-language fr-FR.
Owner
|
Thank you for the clear report and the focused fix. I reproduced the issue with a cp1252 output stream and confirmed that the CLI now prints the Unicode result correctly and exits successfully. This has been merged. Thanks again for your contribution! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What happens
On a Windows console the default code page is a legacy one (cp1252 in Western Europe). Generating a video in French produces
U+202F, the narrow no-break space French typography places before:and!, and Loguru's own progress lines carry circled digits such asU+2464.Either character raises
UnicodeEncodeErrorwhen the CLI prints its result — which happens after the pipeline has finished writing the video:The video is intact on disk. But the process exits non-zero and never prints where the file is, so the run reads as a failure and the path has to be hunted for under
storage/tasks/.The same thing happens inside Loguru's handler during the run:
Reproducing
python cli.py --video-subject "Les 18 heures de Pompéi" --video-language fr-FR --stop-at scriptWindows 11, Python 3.13,
chcp1252. Any non-ASCII target language reaches it; French does so reliably because of the narrow no-break space.The change
Reconfigure
stdoutandstderrto UTF-8 at the entry point, before anything is written. Streams that do not supportreconfigure()— a wrapper, a pipe under some runners — are skipped rather than made to crash on the fix itself.22 lines,
cli.pyonly. No behaviour change on platforms that already default to UTF-8.Verified
Same command after the patch prints the JSON result intact, with
’,°Cand the narrow no-break space preserved, and exits 0.🤖 Generated with Claude Code