Repository navigation
fix(grammar): parse real-world VB.NET - #1
Merged
Merged
Conversation
Parsing the SCrawler and FFmpegFreeUI corpora (520 files) with 0.2.0 left
312/320 and 200/200 files carrying a parse error. The causes, in rough order
of how many files they cost:
- A file that ends right after `End Class`, with no trailing newline, was
missing a statement terminator. Nearly every SCrawler file ends that way.
- Keywords outranked longer identifiers, because lexical precedence is
weighed before match length: `Domains` lexed as `Do` + `mains`. A `word`
token now drives keyword extraction, and the precedence hack is gone.
- `REM` had no word boundary, so `RemoveAll(x)` became a comment to the end
of the line.
- `Imports` and `Option` were only accepted as the very first node, so a
leading comment or blank line broke the next one.
- Identifiers were ASCII-only; VB.NET follows the same Unicode classes as
C#. A leading BOM is skipped as whitespace.
- `#Region` / `#End Region` and other directives were only allowed inside a
procedure body, and nested types were not allowed inside a type at all.
- Type parameter lists had no parentheses (`Sub Bar(Of T)()`), and an
invocation could not carry type arguments. `(Of` now lexes as one token:
with a bare `(` the parser cannot tell a type argument list from a
constructor argument list until it has read a second token.
- `Handles` and `Implements` clauses on members were unsupported.
- `New Point(0, 0)` read its arguments as an array rank specifier.
The same pass adds the constructs the corpora leaned on: implicit line
continuations after an operator, a comma, `=`, an open paren or a trailing
dot; single-line `If ... Then ... Else ...`; LINQ query expressions; event
wiring, operator and Custom Event declarations; `Await`, `TypeOf ... Is`,
nullable and tuple types, `From {}` initialisers, omitted arguments, array
bounds in a declarator, `Async` lambdas and generic constraint lists.
XML literals remain a known gap.
Claude-Session: https://claude.ai/code/session_016BpMYrS81wxdJznMPXTTue
One corpus case per class of failure found in the SCrawler and FFmpegFreeUI corpora, including a file that ends without a newline. Claude-Session: https://claude.ai/code/session_016BpMYrS81wxdJznMPXTTue
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Parsing two real VB.NET repositories with 0.2.0 showed almost every file
carrying a parse error: 312 of 320 files in SCrawler and 200 of 200 in
FFmpegFreeUI. This fixes the grammar against those corpora.
Imports/Optiononly accepted as the first node#Region/#End Regionand other directives outside a procedureSub Bar(Of T)(),Foo.Bar(Of Integer)())Handles(andImplements) clauses on membersFrom {}initialisers, omitted argumentsThree further causes turned up while measuring, each costing more files than
any of the above:
End Classwith no trailing newline wasmissing a statement terminator. Nearly every SCrawler file ends that way.
weighed before match length, so
Domainslexed asDo+mains. Awordtoken now drives keyword extraction.
REMhad no word boundary, soRemoveAll(x)became a comment.The same pass adds what the corpora leaned on: implicit line continuations,
single-line
If ... Then ... Else, LINQ queries,Await,TypeOf ... Is,nullable and tuple types, operator and Custom Event declarations, event
wiring statements, array bounds in a declarator and
Asynclambdas.Files with any parse error, over both corpora:
No existing node type or field was renamed. Shape changes worth knowing:
option_statementsnow wraps oneOptionline and may repeat, instead ofone node covering every leading
Optionline.not wrapped in
type_declaration(which already carriesdelegate_declarationand would clash with it).handles_clause,query_expression,query_range,query_clause,tuple_type,tuple_element,operator_declaration,type_of_expression,await_expression,with_member_access,array_bounds,inline_statements,inline_statement, and theevent-wiring statements.
https://claude.ai/code/session_016BpMYrS81wxdJznMPXTTue