You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 2aeeed5
Browse filesBrowse the repository at this point in the historyBrowse files
fix(welcome-scan): bound the specialization walk and end it with its scan (#810)
The specialization probe took its contracts from sorted(start.rglob(...))[:8],
which reads the whole cwd tree before it can sort. Under a directory holding
many repos that was 279 s and 7,816 hits. #800 made the scan return at its
1.0 s cap, but Python cannot kill a thread, so the walker kept running at about
half a core, and the mode picker, the interview signals and the welcome panel
each start a scan: up to three walkers overlapped.
The probe now takes its contracts from _find_contracts:
- depth-first with each directory's entries in name order, which yields
exactly sorted(rglob(...)). The walk stops at the 8th contract and the 8 are
the ones the old code tallied, wherever the bounds below do not cut;
- it does not enter the directories on the repo's list
(upstream_discovery IGNORED_DIRS | BUILD_OUTPUT_DIRS), lists at most 6 levels
below the start and reads at most 10,000 entries;
- symlinks are not followed and only a regular file counts as a contract
(rglob opens a symlink to /dev/zero, or a FIFO named like a contract, and
blocks);
- run_welcome_scan sets a threading.Event when it returns. The walk ends at its
next directory, and the probe stops parsing on it too;
- a process-wide lock runs one walk at a time. A walker stuck in a filesystem
call (a dead mount) cannot be stopped; the walks of later scans wait behind it
and leave with their own scan (a caller with no scan gives up after one scan
budget), so a process holds at most one.
Measured from ~/Documents/open-source-community, three scans in one process
after build_parser(): walker threads alive 3 -> 0, CPU over a 3 s idle
1.42 s -> 0.00 s, scan wall time 1.0 s -> 0.25-0.67 s (it no longer waits out
the budget for the walk).
tests/test_welcome_scan_walk.py pins the order against rglob, the skip list, the
caps, symlinks, the stop flag, the wait for the gate and the tally rule.
tests/test_welcome_scan_budget.py pins that the walker ends within a second of
its scan, that consecutive scans never have two walks reading at once, and that
three scans over a hung filesystem leave one walker.
Copy file name to clipboardExpand all lines: CLAUDE.md
+3-2Lines changed: 3 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -420,10 +420,11 @@ hand-rolling the protocol.
420
420
421
421
`cli/_welcome_scan.py::run_welcome_scan` now has a budget that means what it says:
422
422
423
-
-**One wall-clock cap.**`_TOTAL_TIMEOUT_S` (1.0 s) is the only scan budget; the old per-probe `_PROBE_TIMEOUT_S` is gone. Probes run on daemon threads (`_spawn_probe`: one `Future` each, deliberately not a `ThreadPoolExecutor`, whose non-daemon workers are joined at interpreter exit and would hold the CLI open for an abandoned probe). The scan returns at the deadline and drops stragglers. Python cannot kill a thread, so an abandoned probe keeps running in the background until it finishes; the one that can run long is `_probe_specialization_suggestion` (unbounded `rglob`), and the picker, the interview signals and the welcome panel each start a scan, so up to three walkers can overlap on a huge cwd. Probes must not write to stdout/stderr (a daemon thread caught mid-write at interpreter exit can abort the process).
423
+
-**One wall-clock cap.**`_TOTAL_TIMEOUT_S` (1.0 s) is the only scan budget; the old per-probe `_PROBE_TIMEOUT_S` is gone. Probes run on daemon threads (`_spawn_probe`: one `Future` each, deliberately not a `ThreadPoolExecutor`, whose non-daemon workers are joined at interpreter exit and would hold the CLI open for an abandoned probe). The scan returns at the deadline and drops stragglers. Python cannot kill a thread, so an abandoned probe keeps running until it finishes (the one that used to run for minutes is the tree walk, below). Probes must not write to stdout/stderr (a daemon thread caught mid-write at interpreter exit can abort the process).
424
424
-**Cold imports are off the clock.**`_warm_probe_imports` imports `_WARM_MODULES` (the AI-config pydantic chain, the dbt runner) from the main thread while the probes run, then the budget starts. Probe imports stay function-local: if a probe starts importing something new, list it in `_WARM_MODULES` or `tests/test_welcome_scan_budget.py::test_probes_import_nothing_new_after_the_warm_up` fails. NB `build_parser()` already imports `ai_setup`, the dbt runner and pydantic, so in a real CLI process only `copilot.unified_config` is still cold; the warm-up matters for tests and library callers.
425
425
-**dbt has its own cap.**`_DBT_VERSION_TIMEOUT_S` (0.32 s) sizes the `dbt --version` subprocess and is not derived from the scan budget and must stay below it, so the child is killed by its own timeout instead of being orphaned. With Python dbt-core on PATH it usually waits out that cap, which is the scan's floor (~0.35 s).
426
-
-`scan_duration_ms` is the real wall-clock of the call, imports included. Pins: `tests/test_welcome_scan_budget.py`.
426
+
- **The specialization walk is bounded and ends with its scan.** `_probe_specialization_suggestion` used to `rglob` the whole cwd tree (279 s under a directory holding many repos), and the picker, the interview signals and the welcome panel each start a scan, so up to three abandoned walkers overlapped (~0.5 core). Its contracts now come from `_find_contracts`: depth-first in name order, so the first 8 are exactly `sorted(rglob(...))[:8]` wherever the bounds do not cut. It skips `util/upstream_discovery`'s `IGNORED_DIRS | BUILD_OUTPUT_DIRS`, lists at most `_SPECIALIZATION_MAX_DEPTH` (6) levels, reads at most `_SPECIALIZATION_MAX_ENTRIES` (10,000) entries, and does not follow symlinks or take anything but a regular file. `run_welcome_scan` sets a `threading.Event` when it returns and the walk ends at its next directory; `_WALK_GATE` runs one walk at a time, so a walker stuck in a filesystem call (a dead mount) is the only one a process can hold. `tests/fragments/test_no_raw_readers.py` registers `_probe_specialization_suggestion` as an accepted raw reader. The scanner finds it through the word `contract` in that function (the `contract.fluid.yaml` in its docstring, the `_find_contracts` call); the registry only checks that the function exists, so losing both silently orphans the entry.
427
+
-`scan_duration_ms` is the real wall-clock of the call, imports included. Pins: `tests/test_welcome_scan_budget.py`, `tests/test_welcome_scan_walk.py`.
0 commit comments