[1216] 优化 lolly tokenize 性能 - #4369
Merged
Merged
Conversation
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
接续 #4366(replace 优化),用相同手法优化
tokenize (string s, string sep):改动
s[i] == sep[0]才进入test()完整比较N(sep)提到循环外,循环上界收紧为i + k <= ntest对空模式恒真且i+=0不前进,现按整串唯一 token 处理)实测(同进程交错,releasedbg,88KB 文本)
tokenize 有 73 处 C++ 调用点(TeX 导入导出、排版核心、字体系统、URL 解析)+ 35 处 scheme 调用。
验证
xmake test lolly_tests/analyze_test:全部通过(含新增边界用例与大输入冒烟)xmake r analyze_bench:tokenize microbench(多命中 / 无命中 / 首字符干扰)gf fmt --changed-since=main:已格式化🤖 Generated with Claude Code