Skip to content

fix(pipeline): keep both nodes when a file and a directory share a stem - #2624

Merged
DeusData merged 1 commit into
mainfrom
fix/stem-folder-collision
Oct 13, 2026
Merged

DeusData merged 1 commit into
mainfrom
fix/stem-folder-collision

Conversation

@DeusData

Copy link
Copy Markdown
Owner

What

A file and the directory beside it with the same stem got one qualified name:

path node QN
src/frontend/genproto.sh Module proj.src.frontend.genproto
src/frontend/genproto/ Folder proj.src.frontend.genproto

The graph buffer keeps the structural node of a QN on purpose (Java/Go modules ARE directories), so the file's Module was dropped, together with every edge that should have pointed at it. On 17 infrastructure repositories that lost 27 Modules: shell scripts, YAML and JSON manifests, a Makefile, Rust, Elixir, Python, Terraform.

Fix

One side yields. The choice is made once per run from the discovered files (cbm_stem_marks_t, src/pipeline/pipeline.c):

  • The Folder is spelled proj.pkg.foo/. No module QN ends in a slash.
  • Unless the directory is a Java/Go package. That Folder IS the package's module, and Go imports resolve to its plain QN. So it keeps the QN, and the file's Module yields instead: proj.pkg.foo#sh (the file's extension).
  • Symbol QNs do not change. Rust foo.rs beside foo/ keeps proj.foo, and the C foo.h/foo.c stem pair is untouched. A dotted stem (foo.test.ts beside foo/test/) collides by QN and is handled the same way.

Around it:

  • Incremental: the spelling depends on the file set. The closure-repair planner therefore declines (stem_mark_changed, full rebuild) when a deletion touches a collision on either side. Adding a file already declines. The repair route builds the same set, so a re-extracted file keeps its spelling.
  • Kustomize: a directory entry (dashboards) beside a same-stem manifest (dashboards.yaml) must not bind to the manifest's now-visible Module. The resolver accepts a Module only for the entry's own file, and otherwise looks up the Folder in either spelling.
  • CBM_SEMANTIC_INDEX_VERSION 7 → 8 (history entry), so existing indexes rebuild once.

Evidence (17 infra corpora, main 56700bd vs this branch, fresh index roots)

main branch
colliding non-Go/Java files without a Module 27 0
marked Folders (…/), each with its Module 0 19
#ext Modules beside a Go package 0 8 (#sh ×5, #mk, #json ×2)
Folder QN equal to a Module QN — 0

Graph changes, identity diff:

  • Folder renames, with CONTAINS edges following them.
  • DEFINES from the file now lands on its Module instead of the Folder.
  • Script top-level CALLS, USAGE, SPAWNS, DEPLOYS and WRITES are now sourced at the recovered Module rather than the File fallback, as for every other script.
  • deckhouse: 12 source shell_lib.sh imports now land on the script, not on the shell_lib/ directory.
  • k8s.io: 3 more kustomize entries resolve (dashboards.yaml, unresolved on main).
  • Go imports of the genproto packages are unchanged.
  • gardener: SEMANTICALLY_RELATED +22/−41, a vocabulary shift from the 7 new Modules. Both binaries re-run identically, so this is deterministic.

Tests (each RED without its part of the change)

  • pipeline:pipeline_stem_folder_collision_keeps_both_nodes covers every collision shape: Folder side, Go-package side, Rust, the C stem pair, the dotted stem, and a Go import onto the package Folder. It runs once small (sequential extraction) and once past MIN_FILES_FOR_PARALLEL (parallel workers), since the two insert Modules on separate paths.
  • pipeline:pipeline_kustomize_entry_beside_same_stem_file: a directory entry lands on the marked Folder, a file entry on the file's Module.
  • pipeline_semantic_manifest_repro:incremental_stem_mark_deletion_rebuilds:
    • deleting the script that marked a Folder rebuilds, and the Folder is plain again;
    • deleting a Go package's only Go file rebuilds, and the script beside it gets its plain Module back while its Folder is marked;
    • a deletion away from any collision stays a closure repair.

Revert checks, each part removed on its own (macOS, ASan test runner):

reverted red
Folder marking 20 checks (both runs), the kustomize directory entry, the deletion test's setup
Java/Go exemption Go Folder keeps its QN, Go import onto it (both runs)
#ext in process_def (sequential) api.sh Module + DEFINES (small run)
#ext in insert_def_into_gbuf (parallel) api.sh Module + DEFINES (parallel run)
#ext lookup in register_and_link_def api.sh DEFINES (parallel run)
planner decline genproto.sh deletion takes the closure repair
kustomize file check the directory entry binds to the manifest

Known limits

  • Something that imports a #ext Module by its computed QN still lands on the Go Folder, exactly as on main.
  • Python foo.py beside a foo/ package: the file wins for import pkg.foo, as JS/TS resolution does. Python itself prefers the package; the layout is rare and is itself a bug in the user's code.

Local legs

Run locally:

  • the proof on the 17 corpora;
  • 34 affected suites on macOS (ASan): the pipeline family, fqn, parallel, incremental, graph_buffer, registry, kustomize probes, complexity, integration, the doc-link and doc-mention closure suites. 1568 pass. The one red, incremental_deleted_spawner_drops_its_process (closure declined python_namespace_changed), is pre-existing on main 56700bd and fixed by fix: never qsort an empty k8s vector; split the spawner deletion test #2617;
  • clang-format (Homebrew LLVM), cppcheck 2.20 (lint image), the memory-core ratchet (pipeline.c baseline lowered 134 → 132), the NOLINT check and the security audit.

Skipped, with PR CI as the gate: the Linux and Windows legs. The change is platform-neutral C with no new I/O, process or memory-lifetime risk beyond the per-run arena owned by run_extraction_phase / run_closure_delta.

A file and the directory beside it with the same stem (genproto.sh next
to genproto/) got one qualified name: the file's Module QN equals the
directory's Folder QN, and the graph buffer keeps the structural node, so
the file's Module was dropped, and with it the edges that pointed at it.
On 17 infrastructure repositories this lost 27 Modules (shell scripts,
YAML and JSON manifests, a Makefile, Rust, Elixir, Python, Terraform).

One side now yields, decided once per run from the discovered files:
- the Folder is spelled proj.pkg.foo/ (no module QN ends in a slash);
- where the directory is a Java/Go package it IS that package's module
  and Go imports resolve to it, so the Folder keeps its QN and the file's
  Module is spelled proj.pkg.foo#sh instead.
Symbol QNs do not change.

The closure-repair planner declines when a deletion touches a collision
on either side, since the spelling depends on the file set. The kustomize
resolver binds a directory entry to its Folder even when a same-stem
manifest sits beside it. CBM_SEMANTIC_INDEX_VERSION 7 -> 8, so existing
indexes rebuild once.

Measured on main 56700bd over the 17 corpora: colliding files without a
Module 27 -> 0 (19 marked Folders, 8 #ext Modules). k8s.io resolves 3
more kustomize entries, and deckhouse's `source shell_lib.sh` imports
land on the script instead of the directory.

Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
@DeusData
DeusData force-pushed the fix/stem-folder-collision branch from d39ff62 to f96001f Compare October 13, 2026 17:05
@DeusData
DeusData merged commit 8aa5259 into main Oct 13, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant