Repository navigation
Commit dab9187
committed
fix(extract): capture complete doc comments across languages
extract_docstring() took only the single comment node directly before a
definition. Multi-line `//` and `///` docs therefore kept just their last
line (in C# usually `/// </summary>`); docs above attributes, exports and
wrapper declarations were lost; fields, variables and modules never got a
doc; Perl POD was never stored; and every doc was cut at 500 bytes.
Capture the full documentation comment instead:
- contiguous same-style comment runs; Go comment groups with go/ast
directive lines (`//go:`, `//line`, `//export`, ...) stripped
- docs above Rust attributes, JS/TS export statements and wrapper
declarations (arrow and function expressions, class fields,
assignments, ambient declarations) and C# top-level statements
- docs for fields, variables, constants and enum members, found with a
memoised linear walk
- Go package comments and Rust `//!` docs on the File node
- Perl POD sections that name a sub (`=head`/`=item <name>`)
- Rust impl methods, Kotlin KDoc folded into package_header or sibling
gaps, C `typedef struct/enum/union` declarations
- a doc comment wins over an intervening plain `//`; runs detached by a
blank line and runs made only of divider lines are not docs
- no 500-byte cut; the properties buffer grows with the docstring, so a
long doc no longer pushes other properties out
Measured on nine real-world corpora (Go, C#, Java, TypeScript, Rust, C,
PHP, Perl, Kotlin): full-doc capture Go 35% -> 99.999%, C# 10% -> 99.8%,
Java 74% -> 99.99%, Rust 7% -> 99.9%, C 67% -> 99.2%, Kotlin 70% ->
99.96%, TypeScript sources 42% -> 100%, named-sub Perl POD 0% -> 100%.
Node sets and all non-semantic edges are unchanged; index time stays
within run-to-run noise.
Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>1 parent c61b380 commit dab9187
7 files changed
Lines changed: 1946 additions & 63 deletions
File tree
- internal/cbm
- src/pipeline
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
606 | 606 | | |
607 | 607 | | |
608 | 608 | | |
| 609 | + | |
| 610 | + | |
| 611 | + | |
| 612 | + | |
| 613 | + | |
609 | 614 | | |
610 | 615 | | |
611 | 616 | | |
| |||
692 | 697 | | |
693 | 698 | | |
694 | 699 | | |
| 700 | + | |
| 701 | + | |
| 702 | + | |
| 703 | + | |
| 704 | + | |
695 | 705 | | |
696 | 706 | | |
697 | 707 | | |
| |||
0 commit comments