`IndentationLexer.lookAhead()` emits the final token twice when the text has no trailing newline, duplicating it in semantic tokens and diagnostics

[Favixx]

What happens

Inside the loop of lookAhead() every token that is read is pushed onto this.tokens
(the last else branch). After the loop, when the final token was not an EOL, it is
pushed a second time:

eo-lsp-server/src/IndentationLexer.ts

Lines 113 to 145 in b637f88

	private lookAhead(): void {
	let current: Token | null = null;
	let next = this.wrapped.nextToken();
	while (next.type !== Token.EOF) {
	if ((current === null || current.type !== EoLexer.EOL) && next.type === EoLexer.EOL) {
	this.spaces.push(IndentationLexer.textSpaces(next.text || ""));
	this.tokens.push(next);
	} else if (current !== null && current.type === EoLexer.EOL && next.type === EoLexer.EOL) {
	this.spaces.push(IndentationLexer.textSpaces(next.text || ""));
	this.handleTabs(Math.floor(this.spaces[this.spaces.length - 1].length / 2), next);
	} else if (current !== null && current.type === EoLexer.EOL && next.type !== EoLexer.EOL) {
	const spaceText = this.spaces.pop() || "";
	this.handleTabs(Math.floor(spaceText.length / 2), next);
	} else {
	this.tokens.push(next);
	}
	current = next;
	next = this.wrapped.nextToken();
	}
	if (current !== null) {
	if (current.type === EoLexer.EOL) {
	const spaceText = this.spaces.pop() || "";
	this.handleTabs(Math.floor(spaceText.length / 2), next);
	} else {
	this.tokens.push(current);
	}
	}
	while (this.indent.length > 1) {
	this.indent.pop();
	this.emitDedent(1);
	}
	this.tokens.push(next);
	}

        } else {
            this.tokens.push(next);
        }
        current = next;
        next = this.wrapped.nextToken();
    }
    if (current !== null) {
        if (current.type === EoLexer.EOL) {
            const spaceText = this.spaces.pop() || "";
            this.handleTabs(Math.floor(spaceText.length / 2), next);
        } else {
            this.tokens.push(current);
        }
    }

current at that point is the token that the last iteration already pushed, so the
token stream that tokenize() returns contains it twice. Any document whose text does
not end in a newline reaches this branch, which is the normal state of a file while it
is being typed. The duplicate reaches the client through two request handlers:
textDocument/semanticTokens/full, where it produces a second token with
deltaLine = 0, deltaStart = 0 on top of the first, and textDocument/publishDiagnostics,
where the ANTLR message quotes the doubled token text.

Steps to reproduce

Add this to test/, then run npx jest:

import { getParserErrors, tokenize, antlrTypeNumToString } from "../src/parser";

test("duplicate last token", () => {
    const text = "# Fibonacci number.\n[] > fib";
    console.log(tokenize(text)
        .map(t => `${antlrTypeNumToString(t.type)} ${t.line}:${t.column} ${JSON.stringify(t.text)}`)
        .join("\n"));
    console.log("diagnostic:", JSON.stringify(getParserErrors(text)));
});

Over the wire: initialize with textDocument.semanticTokens.tokenTypes containing
comment, keyword, variable and the rest, initialized, then textDocument/didOpen
with the text above, then textDocument/semanticTokens/full.

Observed

From the test:

COMMENTARY 1:0 "# Fibonacci number."
EOL 1:19 "\n"
LSQ 2:0 "["
RSQ 2:1 "]"
SPACE 2:2 " "
ARROW 2:3 ">"
SPACE 2:4 " "
NAME 2:5 "fib"
NAME 2:5 "fib"
undefined 2:8 "<EOF>"
diagnostic: [{"line":2,"column":5,"msg":"no viable alternative at input '# Fibonacci number.\\n[] > fibfib'"}]

From the running server:

{"jsonrpc":"2.0","method":"textDocument/publishDiagnostics","params":{"uri":"file:///nonl.eo","diagnostics":[{"severity":1,"range":{"start":{"line":1,"character":5},"end":{"line":1,"character":5}},"message":"no viable alternative at input '# Fibonacci number.\\n[] > fibfib' (EO 0.61.3)","source":"eo"}]}}
{"jsonrpc":"2.0","id":2,"result":{"resultId":"1786180352762","data":[0,0,19,0,0,1,3,1,6,0,0,2,3,18,0,0,0,3,18,0]}}

The last five numbers of data repeat the fib token at the same position, and the
diagnostic shown to the user reads fibfib for a document that contains fib once.

Expected behavior

IndentationLexer should emit each source token exactly once, so the token stream for a
document without a trailing newline matches the one for the same document with a trailing
newline, apart from the final EOL.