Skip to content

Commit fbdc9ba

Browse files
JordanBoltonMNJordan BoltonCopilot
authored
Replace per-line lexing with fail-fast document scanning (#424)
Preserve function-based readers, remove incremental APIs and line-error aggregation, simplify snapshots and directives, and bump the package to 3.0.0. Co-authored-by: Jordan Bolton <jobolton@microsoft.com> Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 570042df-2a22-47f8-8749-5ddd98932499
1 parent 1e49dd1 commit fbdc9ba

108 files changed

Lines changed: 997 additions & 8452 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎README.md‎

Lines changed: 25 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,31 @@ A parser for the [Power Query/M](https://docs.microsoft.com/en-us/power-query/)
66

77
## How to use
88

9-
The most common way to consume the project is to interact with the helper functions found in [taskUtils.ts](src/powerquery-parser/task/taskUtils.ts). There are all-in-one functions, such as `tryLexParse`, which does a full pass on a given document. There are also incremental functions, such as `tryLex` and `tryParse`, which perform one step at a time. Minimal code samples can be found in [example.ts](src/example.ts).
9+
The most common way to consume the project is to interact with the helper functions found in [taskUtils.ts](src/powerquery-parser/task/taskUtils.ts). There are all-in-one functions, such as `tryLexParse`, which does a full pass on a given document. There are also separate lexing and parsing steps, `tryLex` and `tryParse`. Minimal code samples can be found in [example.ts](src/example.ts).
10+
11+
### Lexing
12+
13+
`Lexer.tryLex(settings, text)` lexes the complete document and returns `Result<LexerSnapshot, LexError.TLexError>`. It stops at the first lexical error, including unterminated text, quoted identifiers, and multiline comments. Errors contain one localized diagnostic, not a map of line errors. Diagnostic positions include the absolute UTF-16 offset, zero-based line offset, and grapheme column.
14+
15+
The lexer retains its function-based token readers. Tokens and comments are emitted directly; line information is used only for source positions and type directives. `TaskUtils.tryLex` and `TaskUtils.tryLexParse` use the same lexer, and parser inputs remain `LexerSnapshot` values.
16+
17+
**Breaking change:** the incremental APIs (`tryAppendLine`, `tryUpdateLine`, `tryUpdateRange`, `tryDeleteLine`), editable `State`/line/range types, line-token kinds, line-error maps and predicates, and `trySnapshot` are removed. Apply edits to the document text and call `Lexer.tryLex` again. There is no intermediate line state or separate snapshot-validation step.
18+
19+
Snapshots derive directive associations from their comments; the constructor no longer accepts a separate directive map. Grapheme-position queries index and cache the token's starting line, including for multiline tokens.
20+
21+
```typescript
22+
import { DefaultSettings, Lexer, ResultUtils } from "@microsoft/powerquery-parser";
23+
24+
const updatedDocumentText = "let x = 2 in x";
25+
const result = Lexer.tryLex(DefaultSettings, updatedDocumentText);
26+
if (ResultUtils.isOk(result)) {
27+
console.log(result.value.tokens, result.value.comments);
28+
} else {
29+
console.error(result.error.message);
30+
}
31+
```
32+
33+
Cancellation still returns a `CommonError` wrapping `CancellationError`; polling occurs at entry and during scanning, rather than at incremental update or snapshot stages.
1034

1135
## Related projects
1236

‎package-lock.json‎

Lines changed: 2 additions & 2 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎package.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
{
22
"name": "@microsoft/powerquery-parser",
3-
"version": "2.0.0",
3+
"version": "3.0.0",
44
"description": "A parser for the Power Query/M formula language.",
55
"author": "Microsoft",
66
"license": "MIT",

‎src/example.ts‎

Lines changed: 6 additions & 74 deletions
Original file line numberDiff line numberDiff line change
@@ -36,84 +36,16 @@ async function parseText(text: string): Promise<void> {
3636

3737
// @ts-ignore
3838
function lexText(text: string): void {
39-
// Notice that the Lexer.State variable is declared using let instead of const.
40-
// This is because calling Lexer functions return a new state object.
41-
42-
// An error state is returned if the argument can't be lexed.
43-
// Use either the typeguard Lexer.isErrorState to discover if it's an error state,
44-
// or Lexer.errorLineMap to quickly get all lines with errors.
45-
// Note: At this point all errors are isolated to a single line.
46-
// Checks for multiline errors, such as an unterminated string, have not been processed.
47-
let triedLex: Lexer.TriedLex = Lexer.tryLex(DefaultSettings, text);
39+
// Apply edits to the text before calling the lexer; every call scans the complete document.
40+
const triedLex: Lexer.TriedLex = Lexer.tryLex(DefaultSettings, text);
4841

4942
if (ResultUtils.isError(triedLex)) {
50-
console.log(`An error occured while lexing: ${triedLex.error.message}`);
51-
52-
return;
53-
}
54-
55-
let lexerState: Lexer.State = triedLex.value;
56-
57-
// The lexer state might have an error.
58-
// To be sure either use the typeguard Lexer.isErrorState,
59-
// or Lexer.errorLineMap to get an option containing a map of all lines with errors.
60-
const errorLineMap: Lexer.ErrorLineMap | undefined = Lexer.errorLineMap(lexerState);
61-
62-
if (errorLineMap !== undefined) {
63-
for (const [lineNumber, errorLine] of errorLineMap.entries()) {
64-
console.log(`lineNumber ${lineNumber} has the following error: ${errorLine.error.message}`);
65-
}
66-
67-
return;
68-
}
69-
70-
// Appending a line is easy.
71-
// Be aware that this is a Result due to the potential of errors such as
72-
// a cancellation request from the CancellationToken.
73-
triedLex = Lexer.tryAppendLine(lexerState, "// hello world", "\n");
74-
ResultUtils.assertIsOk(triedLex);
75-
lexerState = triedLex.value;
76-
77-
// Updating a line number is also easy.
78-
// Be aware that this is a Result due the potential of errors such as
79-
// invalid line numbers or a cancellation request from the CancellationToken.
80-
// For fine-grained control there is also the method Lexer.tryUpdateRange,
81-
// which is how Lexer.tryUpdateLine is implemented.
82-
const triedUpdate: Lexer.TriedLex = Lexer.tryUpdateLine(
83-
lexerState,
84-
lexerState.lines.length - 1,
85-
"// goodbye world",
86-
);
87-
88-
if (ResultUtils.isError(triedUpdate)) {
89-
console.log("Failed to update line");
43+
console.log(`An error occurred while lexing: ${triedLex.error.message}`);
9044

9145
return;
9246
}
9347

94-
lexerState = triedUpdate.value;
95-
96-
// Once no more changes will occur a LexerSnapshot should be created, which is an immutable copy that:
97-
// * combines multiline tokens together
98-
// (eg. TextLiteralStart + TextLiteralContent + TextLiteralEnd)
99-
// * checks for multiline errors
100-
// (eg. unterminated string error)
101-
102-
// If you intend to only lex the document once (perform no append/updates),
103-
// then use the `jobs.tryLex` helper function to perform both actions in one go.
104-
105-
// Creating a LexerSnapshot is a Result due to potential multiline token errors.
106-
const triedLexerSnapshot: Lexer.TriedLexerSnapshot = Lexer.trySnapshot(lexerState);
107-
108-
if (ResultUtils.isOk(triedLexerSnapshot)) {
109-
const snapshot: Lexer.LexerSnapshot = triedLexerSnapshot.value;
110-
console.log(`numTokens: ${snapshot.tokens}`);
111-
console.log(`numComments: ${snapshot.comments}`);
112-
}
113-
// A multiline token validation error was thrown.
114-
else {
115-
const error: Lexer.LexError.LexError = triedLexerSnapshot.error;
116-
console.log(error.innerError.message);
117-
console.log(JSON.stringify(error.innerError, undefined, 4));
118-
}
48+
const snapshot: Lexer.LexerSnapshot = triedLex.value;
49+
console.log(`numTokens: ${snapshot.tokens.length}`);
50+
console.log(`numComments: ${snapshot.comments.length}`);
11951
}

‎src/powerquery-parser/language/token.ts‎

Lines changed: 0 additions & 93 deletions
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,6 @@
11
// Copyright (c) Microsoft Corporation.
22
// Licensed under the MIT license.
33

4-
export enum LineTokenKindAdditions {
5-
LineComment = "LineComment",
6-
MultilineComment = "MultilineComment",
7-
MultilineCommentContent = "MultilineCommentContent",
8-
MultilineCommentEnd = "MultilineCommentEnd",
9-
MultilineCommentStart = "MultilineCommentStart",
10-
TextLiteralContent = "TextLiteralContent",
11-
TextLiteralEnd = "TextLiteralEnd",
12-
TextLiteralStart = "TextLiteralStart",
13-
QuotedIdentifierContent = "QuotedIdentifierContent",
14-
QuotedIdentifierEnd = "QuotedIdentifierEnd",
15-
QuotedIdentifierStart = "QuotedIdentifierStart",
16-
}
17-
184
export enum TokenKind {
195
Ampersand = "Ampersand",
206
Asterisk = "Asterisk",
@@ -80,83 +66,6 @@ export enum TokenKind {
8066
TextLiteral = "TextLiteral",
8167
}
8268

83-
export enum LineTokenKind {
84-
Ampersand = TokenKind.Ampersand,
85-
Asterisk = TokenKind.Asterisk,
86-
AtSign = TokenKind.AtSign,
87-
Bang = TokenKind.Bang,
88-
Comma = TokenKind.Comma,
89-
Division = TokenKind.Division,
90-
DotDot = TokenKind.DotDot,
91-
Ellipsis = TokenKind.Ellipsis,
92-
Equal = TokenKind.Equal,
93-
FatArrow = TokenKind.FatArrow,
94-
GreaterThan = TokenKind.GreaterThan,
95-
GreaterThanEqualTo = TokenKind.GreaterThanEqualTo,
96-
HexLiteral = TokenKind.HexLiteral,
97-
Identifier = TokenKind.Identifier,
98-
KeywordAnd = TokenKind.KeywordAnd,
99-
KeywordAs = TokenKind.KeywordAs,
100-
KeywordEach = TokenKind.KeywordEach,
101-
KeywordElse = TokenKind.KeywordElse,
102-
KeywordError = TokenKind.KeywordError,
103-
KeywordFalse = TokenKind.KeywordFalse,
104-
KeywordHashBinary = TokenKind.KeywordHashBinary,
105-
KeywordHashDate = TokenKind.KeywordHashDate,
106-
KeywordHashDateTime = TokenKind.KeywordHashDateTime,
107-
KeywordHashDateTimeZone = TokenKind.KeywordHashDateTimeZone,
108-
KeywordHashDuration = TokenKind.KeywordHashDuration,
109-
KeywordHashInfinity = TokenKind.KeywordHashInfinity,
110-
KeywordHashNan = TokenKind.KeywordHashNan,
111-
KeywordHashSections = TokenKind.KeywordHashSections,
112-
KeywordHashShared = TokenKind.KeywordHashShared,
113-
KeywordHashTable = TokenKind.KeywordHashTable,
114-
KeywordHashTime = TokenKind.KeywordHashTime,
115-
KeywordIf = TokenKind.KeywordIf,
116-
KeywordIn = TokenKind.KeywordIn,
117-
KeywordIs = TokenKind.KeywordIs,
118-
KeywordLet = TokenKind.KeywordLet,
119-
KeywordMeta = TokenKind.KeywordMeta,
120-
KeywordNot = TokenKind.KeywordNot,
121-
KeywordOr = TokenKind.KeywordOr,
122-
KeywordOtherwise = TokenKind.KeywordOtherwise,
123-
KeywordSection = TokenKind.KeywordSection,
124-
KeywordShared = TokenKind.KeywordShared,
125-
KeywordThen = TokenKind.KeywordThen,
126-
KeywordTrue = TokenKind.KeywordTrue,
127-
KeywordTry = TokenKind.KeywordTry,
128-
KeywordType = TokenKind.KeywordType,
129-
LeftBrace = TokenKind.LeftBrace,
130-
LeftBracket = TokenKind.LeftBracket,
131-
LeftParenthesis = TokenKind.LeftParenthesis,
132-
LessThan = TokenKind.LessThan,
133-
LessThanEqualTo = TokenKind.LessThanEqualTo,
134-
Minus = TokenKind.Minus,
135-
NotEqual = TokenKind.NotEqual,
136-
NullCoalescingOperator = TokenKind.NullCoalescingOperator,
137-
NullLiteral = TokenKind.NullLiteral,
138-
NumericLiteral = TokenKind.NumericLiteral,
139-
Plus = TokenKind.Plus,
140-
QuestionMark = TokenKind.QuestionMark,
141-
RightBrace = TokenKind.RightBrace,
142-
RightBracket = TokenKind.RightBracket,
143-
RightParenthesis = TokenKind.RightParenthesis,
144-
Semicolon = TokenKind.Semicolon,
145-
TextLiteral = TokenKind.TextLiteral,
146-
147-
LineComment = LineTokenKindAdditions.LineComment,
148-
MultilineComment = LineTokenKindAdditions.MultilineComment,
149-
MultilineCommentContent = LineTokenKindAdditions.MultilineCommentContent,
150-
MultilineCommentEnd = LineTokenKindAdditions.MultilineCommentEnd,
151-
MultilineCommentStart = LineTokenKindAdditions.MultilineCommentStart,
152-
TextLiteralContent = LineTokenKindAdditions.TextLiteralContent,
153-
TextLiteralEnd = LineTokenKindAdditions.TextLiteralEnd,
154-
TextLiteralStart = LineTokenKindAdditions.TextLiteralStart,
155-
QuotedIdentifierContent = LineTokenKindAdditions.QuotedIdentifierContent,
156-
QuotedIdentifierEnd = LineTokenKindAdditions.QuotedIdentifierEnd,
157-
QuotedIdentifierStart = LineTokenKindAdditions.QuotedIdentifierStart,
158-
}
159-
16069
export interface IToken<Kind, Position> {
16170
readonly kind: Kind;
16271
// range is [start, end)
@@ -165,8 +74,6 @@ export interface IToken<Kind, Position> {
16574
readonly data: string;
16675
}
16776

168-
export type LineToken = IToken<LineTokenKind, number>;
169-
17077
export type Token = IToken<TokenKind, TokenPosition>;
17178

17279
// 0-indexed

‎src/powerquery-parser/lexer/error.ts‎

Lines changed: 2 additions & 84 deletions
Original file line numberDiff line numberDiff line change
@@ -3,37 +3,10 @@
33

44
import { CommonError, StringUtils } from "../common";
55
import { Localization, LocalizationUtils } from "../localization";
6-
import { Lexer } from "..";
76

87
export type TLexError = CommonError.CommonError | LexError;
98

10-
export type TInnerLexError =
11-
| BadLineNumberError
12-
| BadRangeError
13-
| BadStateError
14-
| EndOfStreamError
15-
| ErrorLineMapError
16-
| ExpectedError
17-
| UnexpectedEofError
18-
| UnexpectedReadError
19-
| UnterminatedMultilineTokenError;
20-
21-
export enum BadLineNumberKind {
22-
LessThanZero = "LessThanZero",
23-
GreaterThanNumLines = "GreaterThanNumLines",
24-
}
25-
26-
// do not these sort variants,
27-
// they are in order that logical checks are made, which in turn help create logical variants
28-
export enum BadRangeKind {
29-
SameLine_LineCodeUnitStart_Higher = "SameLine_LineCodeUnitStart_Higher",
30-
LineNumberStart_GreaterThan_LineNumberEnd = "LineNumberStart_GreaterThan_LineNumberEnd",
31-
LineNumberStart_LessThan_Zero = "LineNumberStart_LessThan_Zero",
32-
LineNumberStart_GreaterThan_NumLines = "LineNumberStart_GreaterThan_NumLines",
33-
LineNumberEnd_GreaterThan_NumLines = "LineNumberEnd_GreaterThan_NumLines",
34-
LineCodeUnitStart_GreaterThan_LineLength = "LineCodeUnitStart_GreaterThan_LineLength",
35-
LineCodeUnitEnd_GreaterThan_LineLength = "LineCodeUnitEnd_GreaterThan_LineLength",
36-
}
9+
export type TInnerLexError = ExpectedError | UnexpectedEofError | UnexpectedReadError | UnterminatedMultilineTokenError;
3710

3811
export enum ExpectedKind {
3912
HexLiteral = "HexLiteral",
@@ -53,57 +26,7 @@ export class LexError extends Error {
5326
constructor(innerError: TInnerLexError) {
5427
super(innerError.message);
5528
this.innerError = innerError;
56-
Object.setPrototypeOf(this, LexError);
57-
}
58-
}
59-
60-
export class BadLineNumberError extends Error {
61-
constructor(
62-
readonly kind: BadLineNumberKind,
63-
readonly lineNumber: number,
64-
readonly numLines: number,
65-
locale: string,
66-
) {
67-
super(Localization.error_lex_badLineNumber(LocalizationUtils.getLocalizationTemplates(locale), kind));
68-
Object.setPrototypeOf(this, BadLineNumberError.prototype);
69-
}
70-
}
71-
72-
export class BadRangeError extends Error {
73-
constructor(
74-
readonly range: Lexer.Range,
75-
readonly kind: BadRangeKind,
76-
locale: string,
77-
) {
78-
super(Localization.error_lex_badRange(LocalizationUtils.getLocalizationTemplates(locale), kind));
79-
Object.setPrototypeOf(this, BadRangeError.prototype);
80-
}
81-
}
82-
83-
export class BadStateError extends Error {
84-
constructor(
85-
readonly innerError: TLexError,
86-
locale: string,
87-
) {
88-
super(Localization.error_lex_badState(LocalizationUtils.getLocalizationTemplates(locale)));
89-
Object.setPrototypeOf(this, BadStateError.prototype);
90-
}
91-
}
92-
93-
export class ErrorLineMapError extends Error {
94-
constructor(
95-
readonly errorLineMap: Lexer.ErrorLineMap,
96-
locale: string,
97-
) {
98-
super(Localization.error_lex_lineMap(LocalizationUtils.getLocalizationTemplates(locale), errorLineMap));
99-
Object.setPrototypeOf(this, ErrorLineMapError.prototype);
100-
}
101-
}
102-
103-
export class EndOfStreamError extends Error {
104-
constructor(locale: string) {
105-
super(Localization.error_lex_endOfStream(LocalizationUtils.getLocalizationTemplates(locale)));
106-
Object.setPrototypeOf(this, EndOfStreamError.prototype);
29+
Object.setPrototypeOf(this, LexError.prototype);
10730
}
10831
}
10932

@@ -162,11 +85,6 @@ export function isTLexError(error: unknown): error is TLexError {
16285

16386
export function isTInnerLexError(error: unknown): error is TInnerLexError {
16487
return (
165-
error instanceof BadLineNumberError ||
166-
error instanceof BadRangeError ||
167-
error instanceof BadStateError ||
168-
error instanceof EndOfStreamError ||
169-
error instanceof ErrorLineMapError ||
17088
error instanceof ExpectedError ||
17189
error instanceof UnexpectedEofError ||
17290
error instanceof UnexpectedReadError ||

0 commit comments

Comments
 (0)