design: evaluate splitting the AST/classification layer into a sub-crate

[dekobon]

Question

What would it take to move the AST / parsing layer out of the root crate into its own crate, with the root crate depending on it? This is a design investigation, not a request to do it; the findings below are from reading the current tree (main @ 97c9ac1).

Finding 1: the natural "parser" seam is upside-down-coupled

Parser<T: ParserTrait> (src/parser.rs) is the parser type, but
ParserTrait (src/traits.rs:53) carries 15 associated types, 13 of
which are metric traits (Cognitive, Halstead, Loc, …) plus
Checker / Getter / Alterator. So the parsing layer depends on the
metrics layer, the opposite of what a split needs. mk_langs!
(src/macros/mod.rs:768, one invocation at src/langs.rs:24) emits
both halves — the LANG enum / *Code tags / extension lookup and
the implement_metric_trait! fan-out — in a single expansion.

Everything else on the parse side is clean:

Candidate to move	Lines	Coupling
src/languages/* (generated kind enums)	~17k	none — pure data from enums/
src/langs.rs	836	PreprocResults, *Code tags
src/node.rs	1943	imports Checker (count_specific_ancestors, line 392) and Search / LanguageInfo; no metric or spaces types
src/parser.rs, src/traits.rs	~450	the blocker above
src/preproc.rs, c_macro.rs, comment_rm.rs, c_declarator.rs	~2k	called by parser.rs during parse and by spaces.rs / cognitive.rs after
vendored tree-sitter-* crates	—	already separate; unchanged

Finding 2: a bare-parser crate is the wrong cut

A crate holding only tree-sitter + kind enums + LANG detection offers
little over tree-sitter itself, and the part with real value —
Checker / Getter classification (is_func, is_call,
get_op_type, get_space_kind) — cannot move without the
ParserTrait inversion anyway. The compile-time win is also small: the
expensive part is the 22 grammar C builds, which are already separate
cached crates.

The better line is the classification layer, where
.claude/rules/grammar-dispatch.md already draws it:

• new crate: languages/, langs.rs, node.rs, traits.rs (minus
  metric associated types), checker/, getter/, alterator.rs,
  preproc.rs, c_macro.rs, c_declarator.rs, comment_rm.rs,
  macros/kind_sets.rs, c_langs_macros/
• root keeps: spaces/, metrics/, ops.rs, output/, wire.rs,
  suppression.rs, vcs/, tools.rs, concurrent_files.rs

Finding 3: a workspace sub-crate makes packaging nearly free, not the refactor

As a default-members entry with a path dependency:

• publish = false makes utils/check-publish-metadata.py skip it
  (it filters on publish != []).
• Language features forward mechanically
  (bash = ["big-code-analysis-ast/bash"]); grammar =-pins are
  inherited via workspace = true, so check-excluded-manifests.py
  gains nothing to gate.
• enums/, recreate-grammars.sh, check-grammar-marker-sync.py
  need a one-path change (-o ./src/languages → the new crate).
• .bca-baseline.toml and .rustfmt-bail-baseline.txt are path-keyed
  and rewrite once (make self-scan-write-baseline-headroom,
  check-rustfmt-bail.py --update).
• make pre-commit, worktrees, CI shape: unchanged.

What does not get cheaper:

1. ParserTrait split. The AST crate keeps Checker / Getter /
   Alterator / LanguageInfo / Search; a second trait in the root
   carries the 13 metric associated types, keyed on the AST crate's
   *Code tags. mk_langs! splits into two macros, one per crate, and
   mk_action! / AstInner dispatch moves with the metric half.
2. pub(crate) → pub. LanguageInfo, ParserTrait, Search,
   Ancestors, Checker, Getter, Alterator, PreprocResults, and
   every *Code tag cross a crate boundary. There is no
   pub(workspace); the only lever is #[doc(hidden)] pub mod __internal with no re-export from the root, and missing_docs in
   [workspace.lints] still demands docs on each. Anything the root
   does re-export becomes STABILITY.md surface.
3. Tests. src/checker/* and src/getter/* unit tests go through
   test_support.rs → analyze(Source::new(..)), a root function. Either
   the AST crate grows a parse-only helper or those ~50 test modules
   stay in the root as integration tests against the AST crate's public
   surface (which doubles as a forcing function for item 2).
4. PyO3. big-code-analysis-py/src/node.rs's unsafe soundness doc
   and make check-safety-doc-pin retarget to the new crate; same
   commit.

Finding 4: publish = false cannot survive a release

A published crate cannot depend on an unpublished path dependency, so
cargo publish of the root fails at the next tag. Before the move,
decide between:

• publish it — joins the =X.Y.Z internal pin chain in
  utils/check-versions.py (INTERNAL_PIN_MANIFESTS) and RELEASING.md,
  which already cannot cargo publish --dry-run the pinned crates
  before the tag; and item 2 above becomes a real API design; or
• keep it private and fold back before release — which makes the
  sub-crate a refactoring scaffold, not a deliverable.

Suggested sequence, if pursued

1. Split mk_langs! in-tree, both halves still in one crate, suite
   green. This isolates the only risky refactor.
2. Decide Finding 4.
3. git mv into the sub-crate — mostly path and visibility churn.
4. make pre-commit; refresh the path-keyed baselines; retarget the
   safety-doc pin and the enum generator output path.

Recommendation

Not worth doing for its own sake: the cost is a two-macro
ParserTrait inversion plus ~10 newly public traits/types, and the
payoff is an unpublishable-as-is internal crate. It becomes worth it
only if a second consumer of the classification layer (a linter, a
refactoring tool, a language server) appears — at which point cut at
the checker/getter line, not the bare parser.