Skip to content

[1216] 优化 lolly tokenize 性能 - #4369

Merged
da-liii merged 2 commits into
MoganLab:mainfrom
da-liii:da/1216/analyze2
Aug 18, 2026
Merged

[1216] 优化 lolly tokenize 性能#4369
da-liii merged 2 commits into
MoganLab:mainfrom
da-liii:da/1216/analyze2

Conversation

@da-liii

@da-liii da-liii commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

接续 #4366(replace 优化),用相同手法优化 tokenize (string s, string sep)

改动

  1. 首字符快速筛选:仅 s[i] == sep[0] 才进入 test() 完整比较
  2. N(sep) 提到循环外,循环上界收紧为 i + k <= n
  3. 修复空分隔符死循环(原实现 test 对空模式恒真且 i+=0 不前进,现按整串唯一 token 处理)

实测(同进程交错,releasedbg,88KB 文本)

负载 旧 → 新 ns/op 提速
每行多命中 894k → 638k 1.4×
无命中 218k → 61k 3.6×
首字符干扰 321k → 78k 4.1×

tokenize 有 73 处 C++ 调用点(TeX 导入导出、排版核心、字体系统、URL 解析)+ 35 处 scheme 调用。

验证

  • xmake test lolly_tests/analyze_test:全部通过(含新增边界用例与大输入冒烟)
  • xmake r analyze_bench:tokenize microbench(多命中 / 无命中 / 首字符干扰)
  • gf fmt --changed-since=main:已格式化

🤖 Generated with Claude Code

da-liii and others added 2 commits August 18, 2026 16:12
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
@da-liii
da-liii merged commit 174840e into MoganLab:main Aug 18, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant