[0969] 修复 LLM 插件输出 emoji 乱码:herk_to_utf8 补 4 字节 UTF-8 分支 - #4536
Merged
Conversation
append_utf8_code 缺少 U+10000 以上码点的 4 字节编码分支,herk_to_utf8 还原 <#1F60A> 等逃逸串时按 3 字节截断,输出非法 UTF-8(FF 98 8A), 经聊天显示与 .tmu 保存逐字节 Latin-1 再编码后即用户所见的乱码。 补齐分支并与 encode_as_utf8 行为对齐,附 herk_4byte_codepoints 回归测试。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
修复 LLM 插件输出 emoji 乱码(任务 0969)
问题
llm 商业版中 LLM 回复里的 emoji(如 😊)显示为乱码,在 mogan 自带的 llm
假插件(Insert → Session → LLM)可稳定复现。复现样本
你是谁.tmu中,原应为😊 的位置存储的是
C3 BF C2 98 C2 8A(U+00FF U+0098 U+008A),界面显示为ÿ加两个缺字形方块。根因
会话输出链路(插件 stdout →
texmacs_input_rep::utf8_flush→tree_utf8_to_herk→ 文档内部 herk 编码)中,utf8_to_herk把映射不到单字节的码点正确逃逸为
<#XXXX>(😊 U+1F60A →<#1F60A>)。但反向
herk_to_utf8的辅助函数append_utf8_code(lolly/lolly/data/herk.cpp)只有 1/2/3 字节 UTF-8 分支。对 ≥ U+10000 的码点按 3 字节形式截断编码:
U+1F60A 输出非法字节
FF 98 8A,下游(聊天显示、.tmu 保存)按 Latin-1 逐字节再编码即
C3 BF C2 98 C2 8A,与样本文件逐字节吻合。cork 侧同功能的
cork_to_utf8走encode_as_utf8,有完整 4 字节分支,不受影响;全库仅此一处缺该分支。
修复
lolly/lolly/data/herk.cpp:append_utf8_code补code < 0x10000(3 字节)与
code <= 0x1FFFFF(4 字节)分支,超出 Unicode 上限不再编码,与encode_as_utf8行为对齐。测试与验证
herk_4byte_codepoints回归用例(U+1F60A / U+1F642 / U+20000 /U+10FFFF 的
<#XXXX>↔ UTF-8 往返,及混合文本还原)。xmake test "lolly_tests/herk_test"(仓库根目录):修复前新用例失败(复现 bug),修复后 38 用例 635 断言全部通过。
xmake b stem与xmake install --yes stem均通过。手工验证步骤(待验证)
ÿ加方块;xxd检查对应位置应为f0 9f 98 8a。涉及文件
lolly/lolly/data/herk.cpplolly/tests/lolly/data/herk_test.cppdevel/0969.md