Skip to content

Commit dab9187

Browse files
committed
fix(extract): capture complete doc comments across languages
extract_docstring() took only the single comment node directly before a definition. Multi-line `//` and `///` docs therefore kept just their last line (in C# usually `/// </summary>`); docs above attributes, exports and wrapper declarations were lost; fields, variables and modules never got a doc; Perl POD was never stored; and every doc was cut at 500 bytes. Capture the full documentation comment instead: - contiguous same-style comment runs; Go comment groups with go/ast directive lines (`//go:`, `//line`, `//export`, ...) stripped - docs above Rust attributes, JS/TS export statements and wrapper declarations (arrow and function expressions, class fields, assignments, ambient declarations) and C# top-level statements - docs for fields, variables, constants and enum members, found with a memoised linear walk - Go package comments and Rust `//!` docs on the File node - Perl POD sections that name a sub (`=head`/`=item <name>`) - Rust impl methods, Kotlin KDoc folded into package_header or sibling gaps, C `typedef struct/enum/union` declarations - a doc comment wins over an intervening plain `//`; runs detached by a blank line and runs made only of divider lines are not docs - no 500-byte cut; the properties buffer grows with the docstring, so a long doc no longer pushes other properties out Measured on nine real-world corpora (Go, C#, Java, TypeScript, Rust, C, PHP, Perl, Kotlin): full-doc capture Go 35% -> 99.999%, C# 10% -> 99.8%, Java 74% -> 99.99%, Rust 7% -> 99.9%, C 67% -> 99.2%, Kotlin 70% -> 99.96%, TypeScript sources 42% -> 100%, named-sub Perl POD 0% -> 100%. Node sets and all non-semantic edges are unchanged; index time stays within run-to-run noise. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
1 parent c61b380 commit dab9187

7 files changed

Lines changed: 1946 additions & 63 deletions

File tree

‎internal/cbm/cbm.h‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -606,6 +606,11 @@ typedef struct CBMFileResult {
606606
// by cbm_free_result(); ordinary single-file results leave these zeroed.
607607
struct CBMFileResult **owned_results;
608608
int owned_result_count;
609+
610+
/* The file's own doc, set on its File node: the Go package comment or
611+
* the Rust inner docs (//!). NULL for other languages and undocumented
612+
* files. */
613+
const char *module_doc;
609614
} CBMFileResult;
610615

611616
// --- Enclosing function cache ---
@@ -692,6 +697,11 @@ typedef struct {
692697
/* How many nodes the unified walk actually visited (whether or not it ran
693698
* out of budget) — the measurement the budget has to be expressed in. */
694699
uint32_t walk_nodes_visited;
700+
/* Doc-comment lookup state (extract_defs.c), NULL until first used and
701+
* allocated in `scratch`: the memo of parents' child arrays and the Perl
702+
* POD section index. */
703+
void *doc_memo;
704+
void *doc_pod_index;
695705
} CBMExtractCtx;
696706

697707
// --- Public API ---

0 commit comments

Comments
 (0)