Skip to content

Commit 2aeeed5

Browse files
authored
fix(welcome-scan): bound the specialization walk and end it with its scan (#810)
The specialization probe took its contracts from sorted(start.rglob(...))[:8], which reads the whole cwd tree before it can sort. Under a directory holding many repos that was 279 s and 7,816 hits. #800 made the scan return at its 1.0 s cap, but Python cannot kill a thread, so the walker kept running at about half a core, and the mode picker, the interview signals and the welcome panel each start a scan: up to three walkers overlapped. The probe now takes its contracts from _find_contracts: - depth-first with each directory's entries in name order, which yields exactly sorted(rglob(...)). The walk stops at the 8th contract and the 8 are the ones the old code tallied, wherever the bounds below do not cut; - it does not enter the directories on the repo's list (upstream_discovery IGNORED_DIRS | BUILD_OUTPUT_DIRS), lists at most 6 levels below the start and reads at most 10,000 entries; - symlinks are not followed and only a regular file counts as a contract (rglob opens a symlink to /dev/zero, or a FIFO named like a contract, and blocks); - run_welcome_scan sets a threading.Event when it returns. The walk ends at its next directory, and the probe stops parsing on it too; - a process-wide lock runs one walk at a time. A walker stuck in a filesystem call (a dead mount) cannot be stopped; the walks of later scans wait behind it and leave with their own scan (a caller with no scan gives up after one scan budget), so a process holds at most one. Measured from ~/Documents/open-source-community, three scans in one process after build_parser(): walker threads alive 3 -> 0, CPU over a 3 s idle 1.42 s -> 0.00 s, scan wall time 1.0 s -> 0.25-0.67 s (it no longer waits out the budget for the walk). tests/test_welcome_scan_walk.py pins the order against rglob, the skip list, the caps, symlinks, the stop flag, the wait for the gate and the tally rule. tests/test_welcome_scan_budget.py pins that the walker ends within a second of its scan, that consecutive scans never have two walks reading at once, and that three scans over a hung filesystem leave one walker.
1 parent 85174c0 commit 2aeeed5

5 files changed

Lines changed: 594 additions & 20 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -262,6 +262,20 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
262262

263263
### Fixed
264264

265+
- **The `fluid forge` welcome scan kept walking the whole directory tree after it had returned.**
266+
Its type-suggestion probe listed contracts with `sorted(cwd.rglob("contract.fluid.yaml"))[:8]`,
267+
which reads every file under the current directory (279 s under a folder holding many repos) and
268+
goes into `.venv`, `.git` and `node_modules`. The scan returns at its 1.0 s budget, but Python
269+
cannot kill a thread, so the walker went on at about half a core, and the mode picker, the
270+
interview and the welcome panel each start a scan: up to three overlapped. The walk is now
271+
bounded and ends with its scan. It skips the directories `util/upstream_discovery` already skips
272+
(`.git`, `.venv`, `site-packages`, `node_modules`, `dist`, `build`, ...), lists at most 6 levels
273+
below the current directory, reads at most 10,000 directory entries, and stops at its next
274+
directory once the scan has returned; one walk runs at a time per process. It tallies the same
275+
first 8 contracts as before wherever those limits do not cut the walk. A symlink named
276+
`contract.fluid.yaml`, or one that is not a regular file (a FIFO, a device), is no longer
277+
opened.
278+
265279
- **`fluid apply --mode amend-and-build` provisioned a product, then refused its id.** The build
266280
phase keeps run records, cursors and watermarks under `.fluid/runs/<contract.id>/<build.id>/`,
267281
so its ids must start with a letter or underscore, use only letters, digits, `_`, `.` and `-`,

‎CLAUDE.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -420,10 +420,11 @@ hand-rolling the protocol.
420420

421421
`cli/_welcome_scan.py::run_welcome_scan` now has a budget that means what it says:
422422

423-
- **One wall-clock cap.** `_TOTAL_TIMEOUT_S` (1.0 s) is the only scan budget; the old per-probe `_PROBE_TIMEOUT_S` is gone. Probes run on daemon threads (`_spawn_probe`: one `Future` each, deliberately not a `ThreadPoolExecutor`, whose non-daemon workers are joined at interpreter exit and would hold the CLI open for an abandoned probe). The scan returns at the deadline and drops stragglers. Python cannot kill a thread, so an abandoned probe keeps running in the background until it finishes; the one that can run long is `_probe_specialization_suggestion` (unbounded `rglob`), and the picker, the interview signals and the welcome panel each start a scan, so up to three walkers can overlap on a huge cwd. Probes must not write to stdout/stderr (a daemon thread caught mid-write at interpreter exit can abort the process).
423+
- **One wall-clock cap.** `_TOTAL_TIMEOUT_S` (1.0 s) is the only scan budget; the old per-probe `_PROBE_TIMEOUT_S` is gone. Probes run on daemon threads (`_spawn_probe`: one `Future` each, deliberately not a `ThreadPoolExecutor`, whose non-daemon workers are joined at interpreter exit and would hold the CLI open for an abandoned probe). The scan returns at the deadline and drops stragglers. Python cannot kill a thread, so an abandoned probe keeps running until it finishes (the one that used to run for minutes is the tree walk, below). Probes must not write to stdout/stderr (a daemon thread caught mid-write at interpreter exit can abort the process).
424424
- **Cold imports are off the clock.** `_warm_probe_imports` imports `_WARM_MODULES` (the AI-config pydantic chain, the dbt runner) from the main thread while the probes run, then the budget starts. Probe imports stay function-local: if a probe starts importing something new, list it in `_WARM_MODULES` or `tests/test_welcome_scan_budget.py::test_probes_import_nothing_new_after_the_warm_up` fails. NB `build_parser()` already imports `ai_setup`, the dbt runner and pydantic, so in a real CLI process only `copilot.unified_config` is still cold; the warm-up matters for tests and library callers.
425425
- **dbt has its own cap.** `_DBT_VERSION_TIMEOUT_S` (0.32 s) sizes the `dbt --version` subprocess and is not derived from the scan budget and must stay below it, so the child is killed by its own timeout instead of being orphaned. With Python dbt-core on PATH it usually waits out that cap, which is the scan's floor (~0.35 s).
426-
- `scan_duration_ms` is the real wall-clock of the call, imports included. Pins: `tests/test_welcome_scan_budget.py`.
426+
- **The specialization walk is bounded and ends with its scan.** `_probe_specialization_suggestion` used to `rglob` the whole cwd tree (279 s under a directory holding many repos), and the picker, the interview signals and the welcome panel each start a scan, so up to three abandoned walkers overlapped (~0.5 core). Its contracts now come from `_find_contracts`: depth-first in name order, so the first 8 are exactly `sorted(rglob(...))[:8]` wherever the bounds do not cut. It skips `util/upstream_discovery`'s `IGNORED_DIRS | BUILD_OUTPUT_DIRS`, lists at most `_SPECIALIZATION_MAX_DEPTH` (6) levels, reads at most `_SPECIALIZATION_MAX_ENTRIES` (10,000) entries, and does not follow symlinks or take anything but a regular file. `run_welcome_scan` sets a `threading.Event` when it returns and the walk ends at its next directory; `_WALK_GATE` runs one walk at a time, so a walker stuck in a filesystem call (a dead mount) is the only one a process can hold. `tests/fragments/test_no_raw_readers.py` registers `_probe_specialization_suggestion` as an accepted raw reader. The scanner finds it through the word `contract` in that function (the `contract.fluid.yaml` in its docstring, the `_find_contracts` call); the registry only checks that the function exists, so losing both silently orphans the entry.
427+
- `scan_duration_ms` is the real wall-clock of the call, imports included. Pins: `tests/test_welcome_scan_budget.py`, `tests/test_welcome_scan_walk.py`.
427428

428429
## Working style
429430

‎fluid_build/cli/_welcome_scan.py‎

Lines changed: 119 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -31,8 +31,10 @@
3131
3232
Every probe is best-effort and never raises. Budget: once the one-time
3333
module imports are done, the probes share ``_TOTAL_TIMEOUT_S`` of wall-clock
34-
and the scan returns by then, dropping any probe still running. The dbt probe
35-
sets the floor: with Python dbt-core on PATH it usually waits out
34+
and the scan returns by then, dropping any probe still running. A dropped probe
35+
is abandoned, not killed (Python cannot kill a thread); the one that walks the
36+
directory tree bounds itself and ends when the scan returns (``_find_contracts``).
37+
The dbt probe sets the floor: with Python dbt-core on PATH it usually waits out
3638
``_DBT_VERSION_TIMEOUT_S`` and reports nothing. ``scan_duration_ms`` is the
3739
measured wall-clock of the whole call, imports included.
3840
"""
@@ -48,6 +50,7 @@
4850
import time
4951
from concurrent.futures import Future, wait
5052
from dataclasses import asdict, dataclass, field
53+
from itertools import islice
5154
from pathlib import Path
5255
from typing import Any, Callable, Dict, List, Optional, Tuple
5356

@@ -73,11 +76,30 @@
7376
# the AI probe's cold cost (~0.1s CPU, several times that on a loaded host).
7477
"fluid_build.copilot.unified_config",
7578
"fluid_build.build_runners.dbt.runner", # _probe_dbt_engine
79+
# ``_walk_contracts`` takes the contract filename from ``cli.artifact_paths``, which
80+
# ``workspace_config`` above already imports.
81+
"fluid_build.util.upstream_discovery", # _walk_contracts (directories to skip)
7682
)
7783
# Repeat-user threshold — at this many forges we stop showing the panel.
7884
_REPEAT_USER_THRESHOLD = 5
7985
# File extensions we recognise as candidate sample data.
8086
_SAMPLE_EXTENSIONS = (".csv", ".parquet", ".json", ".jsonl", ".tsv", ".duckdb")
87+
# The specialization probe tallies the productType of the first contracts under the
88+
# start directory. Its tree walk (``_find_contracts``) is bounded three ways: contracts
89+
# taken, directory levels below the start, and directory entries read. At the caps an
90+
# ordinary tree costs ~0.1s (a pathological one, 10,000 one-entry directories, ~0.6s of
91+
# CPU), and the scan's stop flag ends the walk early either way.
92+
_SPECIALIZATION_SAMPLE = 8
93+
_SPECIALIZATION_MAX_DEPTH = 6
94+
_SPECIALIZATION_MAX_ENTRIES = 10_000
95+
# One specialization walk at a time in the process. A walk the scan gave up on ends at
96+
# its next directory once the scan's stop flag is set, but one stuck in a filesystem
97+
# call (a hung network mount) never does, and each later scan would add another. Behind
98+
# this lock the later walks wait and leave when their own scan returns (a caller with no
99+
# scan gives up after one scan budget), so a process holds at most one stuck walker.
100+
_WALK_GATE = threading.Lock()
101+
# How often a walk waiting for the gate looks at its scan's stop flag.
102+
_WALK_GATE_POLL_S = 0.05
81103
# CLI binaries we report on.
82104
_CLI_BINARIES = (
83105
"dbt",
@@ -361,24 +383,99 @@ def _probe_return_user() -> Dict[str, Any]:
361383
return {"return_user": False, "forge_count": 0}
362384

363385

364-
def _probe_specialization_suggestion(start: Path) -> Dict[str, Any]:
365-
"""Suggest a productType based on recent forge history.
386+
def _walk_contracts(start: Path, stop: Optional[threading.Event]) -> List[Path]:
387+
"""The bounded walk behind :func:`_find_contracts`: depth-first, entries in name order."""
388+
from fluid_build.cli.artifact_paths import CONTRACT_FILENAME
389+
from fluid_build.util.upstream_discovery import BUILD_OUTPUT_DIRS, IGNORED_DIRS
366390

367-
Reads the last 5 ``.fluid/forge-receipt.json`` entries under the
368-
workspace and proposes the dominant type. Only fires when ≥4/5
369-
are the same type.
391+
skip = IGNORED_DIRS | BUILD_OUTPUT_DIRS
392+
found: List[Path] = []
393+
unread = _SPECIALIZATION_MAX_ENTRIES
394+
395+
def visit(directory: str, depth: int) -> bool:
396+
"""Read ``directory``, then its subdirectories; True once the whole walk is over."""
397+
nonlocal unread
398+
if unread <= 0 or (stop is not None and stop.is_set()):
399+
return True
400+
try:
401+
with os.scandir(directory) as listing:
402+
entries = list(islice(listing, unread))
403+
except OSError:
404+
return False
405+
unread -= len(entries)
406+
# Names compare as path components do, so a contract file falls among the
407+
# subdirectories exactly where ``sorted(rglob(...))`` would put it.
408+
for entry in sorted(entries, key=lambda e: e.name):
409+
try:
410+
if entry.is_dir(follow_symlinks=False):
411+
if depth < _SPECIALIZATION_MAX_DEPTH and entry.name not in skip:
412+
if visit(entry.path, depth + 1):
413+
return True
414+
elif entry.name == CONTRACT_FILENAME and entry.is_file(follow_symlinks=False):
415+
found.append(Path(entry.path))
416+
if len(found) >= _SPECIALIZATION_SAMPLE:
417+
return True
418+
except OSError:
419+
continue
420+
return False
421+
422+
visit(str(start), 0)
423+
return found
424+
425+
426+
def _find_contracts(start: Path, stop: Optional[threading.Event] = None) -> List[Path]:
427+
"""The first ``_SPECIALIZATION_SAMPLE`` contracts under ``start``, in ``rglob`` order.
428+
429+
Depth-first with each directory's entries in name order, which lists the contracts
430+
exactly as ``sorted(start.rglob("contract.fluid.yaml"))`` does, so the walk can end
431+
at the last one it needs instead of reading the whole tree first. It also ends on
432+
its own bounds, and only the contracts it reached are returned:
433+
434+
* directories on the repo's VCS / dependency / build-output list
435+
(``upstream_discovery.IGNORED_DIRS`` and ``BUILD_OUTPUT_DIRS``: ``.git``,
436+
``.venv``, ``site-packages``, ``node_modules``, ``dist``, ...) are not entered;
437+
* nothing deeper than ``_SPECIALIZATION_MAX_DEPTH`` levels below ``start`` is
438+
listed, and at most ``_SPECIALIZATION_MAX_ENTRIES`` entries are read in all (a
439+
directory larger than what is left of that is read in part);
440+
* symlinks are neither followed nor tallied, and only a regular file counts as a
441+
contract: ``rglob`` opens a symlink to ``/dev/zero`` or a FIFO named like one;
442+
* ``stop`` is looked at before every directory. Python cannot kill a thread, so
443+
this is how a scan ends the walk it abandoned.
444+
445+
One walk runs at a time per process (``_WALK_GATE``); a walk that is made to wait
446+
leaves with ``[]`` as soon as its ``stop`` is set, or after one scan budget if it
447+
has none.
448+
"""
449+
deadline = time.monotonic() + _TOTAL_TIMEOUT_S
450+
while not _WALK_GATE.acquire(timeout=_WALK_GATE_POLL_S):
451+
if (stop is not None and stop.is_set()) or time.monotonic() >= deadline:
452+
return []
453+
try:
454+
return _walk_contracts(start, stop)
455+
finally:
456+
_WALK_GATE.release()
457+
458+
459+
def _probe_specialization_suggestion(
460+
start: Path, stop: Optional[threading.Event] = None
461+
) -> Dict[str, Any]:
462+
"""Suggest a productType from the ``contract.fluid.yaml`` files under ``start``.
463+
464+
Tallies ``metadata.productType`` over the first ``_SPECIALIZATION_SAMPLE``
465+
contracts ``_find_contracts`` returns and proposes the dominant type when at
466+
least 4 of at least 5 typed contracts agree. ``stop`` is the scan's flag for a
467+
probe it has given up on: the walk and the parse loop both end on it.
370468
"""
371469
try:
372-
# Cheapest possible: glob for nearby contracts and tally their
373-
# productType. Limited to a small radius so the scan stays
374-
# under budget.
375-
contracts = sorted(start.rglob("contract.fluid.yaml"))[:8]
470+
contracts = _find_contracts(start, stop)
376471
if not contracts:
377472
return {}
378473
import yaml as _yaml
379474

380475
types: List[str] = []
381476
for c in contracts:
477+
if stop is not None and stop.is_set():
478+
return {}
382479
try:
383480
doc = _yaml.safe_load(c.read_text(encoding="utf-8")) or {}
384481
pt = (doc.get("metadata") or {}).get("productType")
@@ -456,14 +553,17 @@ def run_welcome_scan(*, start: Optional[Path] = None) -> WelcomeFindings:
456553
the dbt subprocess included); the probes then get ``_TOTAL_TIMEOUT_S`` of
457554
wall-clock. A probe still running at the deadline is dropped — its fields
458555
stay at their defaults — and abandoned, so a hung probe delays neither the
459-
prompt nor the process exit.
556+
prompt nor the process exit. The probe that walks the directory tree is told
557+
to stop when the scan returns and ends at its next directory; the others run
558+
on to their own end.
460559
461560
Wall-clock is the cold imports plus at most ``_TOTAL_TIMEOUT_S``;
462561
``scan_duration_ms`` reports all of it.
463562
"""
464563
target = (start or Path.cwd()).resolve()
465564
findings = WelcomeFindings()
466565
started = time.monotonic()
566+
stop = threading.Event() # set when the scan returns, see ``_find_contracts``
467567

468568
probes: Dict[str, Tuple[Callable[..., Dict[str, Any]], Tuple[Any, ...]]] = {
469569
"workspace": (_probe_workspace, (target,)),
@@ -474,12 +574,15 @@ def run_welcome_scan(*, start: Optional[Path] = None) -> WelcomeFindings:
474574
"samples": (_probe_sample_data, (target,)),
475575
"cloud": (_probe_cloud_creds, ()),
476576
"return_user": (_probe_return_user, ()),
477-
"specialization": (_probe_specialization_suggestion, (target,)),
577+
"specialization": (_probe_specialization_suggestion, (target, stop)),
478578
}
479579

480-
futures = {name: _spawn_probe(name, fn, args) for name, (fn, args) in probes.items()}
481-
_warm_probe_imports()
482-
done, _ = wait(futures.values(), timeout=_TOTAL_TIMEOUT_S)
580+
try:
581+
futures = {name: _spawn_probe(name, fn, args) for name, (fn, args) in probes.items()}
582+
_warm_probe_imports()
583+
done, _ = wait(futures.values(), timeout=_TOTAL_TIMEOUT_S)
584+
finally:
585+
stop.set()
483586
for name, future in futures.items():
484587
if future not in done:
485588
LOG.debug("welcome_scan_probe_timeout: %s", name)

0 commit comments

Comments
 (0)