Repository navigation
fix(api): make the embed token scan linear - #7831
Conversation
|
No actionable comments were generated in the recent review.
|
| Layer / File(s) | Summary |
|---|---|
Prefix scanning and validation api/oss/src/core/embeds/utils.py, api/oss/tests/pytest/unit/embeds/test_utils.py |
_find_embed_tokens searches for token prefixes and returns complete tokens with nested brackets. Tests cover absent, multiple, nested, and incomplete tokens, and check runtime on a large input. |
Priority:
Estimated code review effort: 2 (Simple) | ~8 minutes
Merge Risk: 
Minimal · up to b2ef5
-
No actionable merge-blocking risk remains.
Pre-merge checks: 
4 · 
1
-
Failed checks: 1 warning Passed checks: 4 passed -
Finishing Touches: 💡 1
-
📝 Generate docstrings: 💡
-
- Commit to this branch
- Create a new PR
🧪 Generate unit tests (beta)
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands.
|
🤖 The AI agent says: I timed the old and new
|
Railway Preview Environment
PR closed or converted to draft. Environment absence verified; retained storage requires separate provider accounting. Post |
Context
Agent sends waited about 15 s in
/api/workflows/revisions/resolvewhen the agent had large inline skills. While it ran, the API event loop stalled, and Redis 0.5 s socket timeouts fired. On a real revision (670 KB JSON, 7 embedded skills, one skill 1.39 MB), one embed scan took 2.9 s on the revision and 7.8 s on the skills.The cause is
_find_embed_tokens. At every character it rantext[i:].startswith("@ag.embed["). The slice copies the rest of the string each time, so the scan is O(n^2) and holds the event loop.Changes
_find_embed_tokensnow jumps between candidates withtext.find("@ag.embed[", i)and does not slice. Matching is unchanged: balanced brackets, incomplete tokens skipped, same order in the returned list. Astr.findscan of the same inputs takes under 1 ms.I checked the rest of
embeds/utils.py._find_snippet_tokensslices only 2 to 3 characters per step, so it is linear. I left it alone.Tests
TestFindEmbedTokensinapi/oss/tests/pytest/unit/embeds/test_utils.py: no token, one token, nested brackets, two tokens, incomplete token at the end, and a 2 MB input that must finish in under 1 s.pytest oss/tests/pytest/unit/embeds: 106 passed.ruff formatandruff checkare clean.