There was an error while loading. Please reload this page.
1 parent 226a775 commit eb97c90Copy full SHA for eb97c90
1 file changed
tivars/tokenizer/state.py
@@ -87,6 +87,7 @@ def next(self, token: TIToken, remainder: str) -> list[list['TokenizerState']]:
87
Determines the next tokenizer timelines given a token
88
89
Each timeline that will branch from this one is returned as a list of states to add to a copy of the stack.
90
+
91
If a list of states in a timeline is...
92
- empty, then the timeline is exiting the current state.
93
- length one, then the timeline's current state is being replaced by a new state.
0 commit comments