Skip to content

Commit 09a34ad

Browse files
safishamsiclaude
andcommitted
fix(update,llm): retry failed extractions; surface claude-cli envelope errors; bump to 0.9.37 (Graphify-Labs#2543, Graphify-Labs#2554)
Graphify-Labs#2543 (adopts PR Graphify-Labs#2546, thanks @michaelxer): a failed extraction is no longer stamped in the incremental manifest as up-to-date, so graphify update retries it instead of skipping it forever; a manifest already poisoned by the old behavior is healed on the next run; genuinely unchanged files are not re-processed. Extended to the watch save_manifest paths too. Graphify-Labs#2554 (adopts PR Graphify-Labs#2555, thanks @annieyii): the claude-cli backend now inspects the stdout envelope for an is_error result (e.g. a rate limit returned with exit code 0) and raises it on both the zero and non-zero exit paths, instead of parsing it as an empty success and bisecting against a live rate limit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent cfc6a75 commit 09a34ad

11 files changed

Lines changed: 598 additions & 19 deletions

‎CHANGELOG.md‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,15 @@
22

33
Full release notes with details on each version: [GitHub Releases](https://github.com/safishamsi/graphify/releases)
44

5-
## 0.9.36 (unreleased)
5+
## 0.9.37 (unreleased)
6+
7+
- Fix: TypeScript member calls no longer fabricate a high-confidence `calls` edge by matching a receiver type by name alone (#2553, thanks @Earthfreedom). A member call now resolves only when the receiver's type is defined in the same file or actually imported by the caller's file, so a third-party `import type { Repo }` can no longer bind to an unrelated local `class Repo`; table-inferred receivers are tiered to INFERRED rather than EXTRACTED.
8+
- Fix: TypeScript/JavaScript calls inside a callback body passed to another call (for example `export const handler = wrapper(async (req) => { helper() })`) are no longer dropped (#2552, thanks @Earthfreedom). The callback body is now walked and its calls attributed to the declaration, through the same import-gated resolution so it cannot fabricate edges.
9+
- Fix: Kotlin imports, fully-qualified calls, and one-line type bodies (#2526, #2550, #2551, thanks @spaceBrownie, @thomasrengot-hub, and @Mustaqeem66 for #2531). The extractor now matches the bundled grammar's `import` node (imports were silently dropped) and resolves each import to the real target node; a fully-qualified call like `com.example.Foo.bar()` now produces a `calls` edge; and a file with syntax the bundled grammar cannot parse (such as a one-line `class C { val x }`) now emits a warning instead of silently extracting nothing, and declarations recovered inside an error span keep their enclosing class.
10+
- Fix: `graphify update` now retries a file whose extractor failed on a previous run instead of stamping it up-to-date forever (#2543, thanks @michaelxer). A failed extraction is no longer recorded in the manifest as processed, a manifest already poisoned by the old behavior is healed on the next run, and the fix avoids re-processing a genuinely-unchanged file.
11+
- Fix: the claude-cli backend now surfaces an API error carried in the stdout envelope (for example a rate limit returned with a zero exit code) instead of treating it as an empty success (#2554, thanks @annieyii). The error is raised on both the zero and non-zero exit paths.
12+
13+
## 0.9.36 (2026-08-07)
614

715
- Fix: four commands that failed silently while exiting 0 now surface the problem (#2534, thanks @elecnix). `cluster-only` warns when `--backend`/`--model`/`--batch-size` are ignored because saved labels are being reused; the community-label prompt no longer collides with the discard sentinel (a model echoing the key back is no longer silently dropped); `tree --root` exits non-zero when the root matches no source file instead of silently flattening the tree; and `cluster-only` stamps `built_at_commit` from the analysed graph rather than the shell's working directory. Also folds in the `cluster-only` refused-write guard from #2522 (thanks @aniJani).
816
- Fix: a Swift `extension Foo` in a different file from `Foo` no longer drops static and singleton call edges into the type (#2538, thanks @pawelo446). The extension node id is now remapped consistently so the extension merges onto its base type before call resolution, and the merge is gated so it never absorbs a same-named type from another language.

‎graphify/cli.py‎

Lines changed: 127 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -90,6 +90,7 @@ def _stamped_manifest_files(
9090
sem_result: dict,
9191
root: Path,
9292
partial_source_files: "set[str] | None" = None,
93+
failed_ast_sources: "set[str] | list[str] | None" = None,
9394
) -> dict[str, list[str]]:
9495
"""Manifest-safe files dict: only stamp semantic files that actually
9596
produced output (cache hit or fresh extraction). Files whose chunk failed
@@ -115,6 +116,10 @@ def _stamped_manifest_files(
115116
doc unstamped, so detect_incremental re-queued it on every run. The stamping
116117
condition mirrors the cache-write keying (a hyperedge carries its own
117118
``source_file``); do not derive it from member nodes.
119+
120+
``failed_ast_sources`` (#2543): code files whose AST extractor errored
121+
(missing optional extra, etc.) or returned zero nodes. They must not be
122+
stamped as up-to-date or a later install of the extra will never re-run.
118123
"""
119124
root = Path(root)
120125

@@ -134,12 +139,16 @@ def _resolve(value: str) -> Path:
134139
if sf:
135140
sem_extracted.add(_resolve(sf))
136141
partial_resolved = {_resolve(p) for p in (partial_source_files or set())}
142+
failed_ast_resolved = {_resolve(p) for p in (failed_ast_sources or [])}
137143
sem_types = {"document", "paper", "image"}
138144
return {
139145
ftype: [
140146
f for f in flist
141-
if ftype not in sem_types
142-
or (_resolve(f) in sem_extracted and _resolve(f) not in partial_resolved)
147+
if _resolve(f) not in failed_ast_resolved
148+
and (
149+
ftype not in sem_types
150+
or (_resolve(f) in sem_extracted and _resolve(f) not in partial_resolved)
151+
)
143152
]
144153
for ftype, flist in files_by_type.items()
145154
}
@@ -330,6 +339,88 @@ def _provably_excluded(c: Path) -> bool:
330339
return stale
331340

332341

342+
def _zero_node_stamped_code_sources(
343+
graph_path: Path,
344+
scan_root: Path,
345+
unchanged_code: list[str],
346+
) -> list[str]:
347+
"""Manifest-stamped code files with a registered extractor but ZERO nodes
348+
in the existing graph.json (#2543 heal).
349+
350+
The failed-source unstamping only covers failures that happen AFTER it
351+
shipped; a manifest poisoned by an earlier run (extraction failed, hashes
352+
stamped anyway) keeps reporting the file unchanged forever, and the only
353+
documented recovery was deleting graphify-out/. A stamped file that the
354+
graph has no nodes for, despite an extractor being wired up for it, is
355+
exactly that state — re-queue it as changed. Bounded by the same no-wedge
356+
property: if it fails again this run it is now left unstamped, and if it
357+
succeeds its nodes enter graph.json so the next scan stops re-queuing it.
358+
359+
Membership mirrors the ``source_file`` spellings extracts store (#1897/
360+
#1941: scan-root-relative, forward slash; absolute for out-of-root) and
361+
compares NFC-normalized (#2210/#2221).
362+
"""
363+
if not unchanged_code:
364+
return []
365+
from graphify.paths import nfc
366+
try:
367+
data = json.loads(graph_path.read_text(encoding="utf-8"))
368+
except Exception:
369+
return []
370+
if not isinstance(data, dict):
371+
return []
372+
try:
373+
root_res = scan_root.resolve()
374+
except (OSError, RuntimeError):
375+
root_res = scan_root
376+
# <out>/graphify-out/graph.json — legacy relative source_files may be
377+
# anchored here instead of the scan root (<=0.9.16, #555/#1899).
378+
out_base = graph_path.parent.parent
379+
try:
380+
out_base = out_base.resolve()
381+
except (OSError, RuntimeError):
382+
pass
383+
384+
present: set[str] = set()
385+
for n in data.get("nodes", []):
386+
if not isinstance(n, dict):
387+
continue
388+
sf = n.get("source_file")
389+
if not sf or not isinstance(sf, str):
390+
continue
391+
present.add(nfc(sf))
392+
p = Path(sf)
393+
if p.is_absolute():
394+
try:
395+
present.add(nfc(str(p.resolve())))
396+
except (OSError, RuntimeError):
397+
pass
398+
else:
399+
rel = sf.replace("\\", "/")
400+
for base in (root_res, out_base):
401+
present.add(nfc(os.path.normpath(str(base / rel))))
402+
403+
from graphify.extract import _get_extractor
404+
healed: list[str] = []
405+
for f in unchanged_code:
406+
p = Path(f)
407+
if _get_extractor(p) is None:
408+
continue # no extractor: absence from the graph is expected
409+
spellings = {nfc(str(p))}
410+
try:
411+
spellings.add(nfc(str(p.resolve())))
412+
except (OSError, RuntimeError):
413+
pass
414+
try:
415+
spellings.add(nfc(p.resolve().relative_to(root_res).as_posix()))
416+
except (ValueError, OSError, RuntimeError):
417+
pass
418+
if spellings & present:
419+
continue # the graph has this file: stamp is honest
420+
healed.append(f)
421+
return healed
422+
423+
333424
def _prune_graph_json_sources(graph_path: Path, stale_sources: list[str]) -> int:
334425
"""Drop nodes/edges/hyperedges owned by ``stale_sources`` from graph.json
335426
in place. Returns the number of nodes removed.
@@ -2991,6 +3082,24 @@ def _parse_float(name: str, raw: str) -> float:
29913082
graph_stale_sources = _stale_graph_sources(
29923083
existing_graph_path, target, _seen_files, detection=detection
29933084
)
3085+
# #2543 heal: manifests poisoned BEFORE failed-source unstamping
3086+
# existed carry live hashes for code files whose extraction failed
3087+
# (missing extra, crash) — stamped up-to-date yet absent from
3088+
# graph.json, so the incremental gate skips them forever. Treat
3089+
# such a file as changed and re-queue it; if it fails again this
3090+
# run it is now left unstamped, so this cannot wedge.
3091+
_healed_sources = _zero_node_stamped_code_sources(
3092+
existing_graph_path,
3093+
target,
3094+
detection.get("unchanged_files", {}).get("code", []),
3095+
)
3096+
if _healed_sources:
3097+
print(
3098+
f"[graphify extract] re-queuing {len(_healed_sources)} "
3099+
f"manifest-stamped code file(s) with no nodes in graph.json "
3100+
f"(prior failed extraction, #2543)"
3101+
)
3102+
code_files.extend(Path(p) for p in _healed_sources)
29943103
else:
29953104
print(f"[graphify extract] scanning {target}")
29963105
detection = _detect(
@@ -3528,8 +3637,16 @@ def _progress(idx: int, total: int, _result: dict) -> None:
35283637
# Path normalization against the scan root happens inside the helper
35293638
# (#1897) so fresh root-relative source_files match detect()'s
35303639
# absolute file lists.
3531-
_manifest_files = _stamped_manifest_files(files_by_type, sem_result, target,
3532-
partial_source_files=_partial_semantic_files)
3640+
# #2543: also drop AST sources that failed (missing optional extra /
3641+
# zero-node anomaly) so they are not frozen as up-to-date.
3642+
_failed_ast_sources = list(ast_result.get("failed_sources") or [])
3643+
_manifest_files = _stamped_manifest_files(
3644+
files_by_type,
3645+
sem_result,
3646+
target,
3647+
partial_source_files=_partial_semantic_files,
3648+
failed_ast_sources=_failed_ast_sources,
3649+
)
35333650

35343651
# Files dispatched this run but dropped by _stamped_manifest_files
35353652
# above (failed chunk, LLM omission, or any future exclusion) still
@@ -3546,6 +3663,9 @@ def _progress(idx: int, total: int, _result: dict) -> None:
35463663
f for _flist in _manifest_files.values() for f in _flist
35473664
}
35483665
_cleared_semantic = {str(p) for p in semantic_files} - _stamped_semantic
3666+
# #2543: AST failures need both hashes blanked (clear_ast), not just
3667+
# semantic_hash — otherwise a prior bad stamp keeps the file "unchanged".
3668+
_cleared_ast = set(_failed_ast_sources)
35493669

35503670
# Full-scan manifest saves prune rows for in-root files that left the
35513671
# scan corpus but still exist on disk (#1908). The corpus must be the
@@ -3607,7 +3727,7 @@ def _invalidate_file_manifest_for_db_graph() -> None:
36073727
"(--no-cluster); outputs left untouched."
36083728
)
36093729
try:
3610-
_save_manifest(_manifest_files, manifest_path=str(manifest_path), kind="both", root=target, scan_corpus=_scan_corpus, clear_semantic=_cleared_semantic)
3730+
_save_manifest(_manifest_files, manifest_path=str(manifest_path), kind="both", root=target, scan_corpus=_scan_corpus, clear_semantic=_cleared_semantic, clear_ast=_cleared_ast or None)
36113731
except Exception as exc:
36123732
print(f"[graphify extract] warning: could not write manifest: {exc}", file=sys.stderr)
36133733
stages.total()
@@ -3713,7 +3833,7 @@ def _invalidate_file_manifest_for_db_graph() -> None:
37133833
)
37143834
try:
37153835
if has_path:
3716-
_save_manifest(_manifest_files, manifest_path=str(manifest_path), kind="both", root=target, scan_corpus=_scan_corpus, clear_semantic=_cleared_semantic)
3836+
_save_manifest(_manifest_files, manifest_path=str(manifest_path), kind="both", root=target, scan_corpus=_scan_corpus, clear_semantic=_cleared_semantic, clear_ast=_cleared_ast or None)
37173837
except Exception as exc:
37183838
print(f"[graphify extract] warning: could not write manifest: {exc}", file=sys.stderr)
37193839
if global_merge:
@@ -3860,7 +3980,7 @@ def _invalidate_file_manifest_for_db_graph() -> None:
38603980
_wja(analysis_path, analysis, indent=2)
38613981
try:
38623982
if has_path:
3863-
_save_manifest(_manifest_files, manifest_path=str(manifest_path), kind="both", root=target, scan_corpus=_scan_corpus, clear_semantic=_cleared_semantic)
3983+
_save_manifest(_manifest_files, manifest_path=str(manifest_path), kind="both", root=target, scan_corpus=_scan_corpus, clear_semantic=_cleared_semantic, clear_ast=_cleared_ast or None)
38643984
except Exception as exc:
38653985
print(f"[graphify extract] warning: could not write manifest: {exc}", file=sys.stderr)
38663986

‎graphify/detect.py‎

Lines changed: 26 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1699,6 +1699,7 @@ def save_manifest(
16991699
root: Path | None = None,
17001700
scan_corpus: set[str] | list[str] | None = None,
17011701
clear_semantic: set[str] | list[str] | None = None,
1702+
clear_ast: set[str] | list[str] | None = None,
17021703
) -> None:
17031704
"""Save current file mtimes + content hashes for change detection.
17041705
@@ -1733,6 +1734,12 @@ def save_manifest(
17331734
and making detect_incremental(kind="semantic") report them unchanged.
17341735
Pass the set of such files (any path form ``scan_corpus`` accepts) to
17351736
force their seeded semantic_hash to "" instead of inheriting it.
1737+
1738+
``clear_ast`` (#2543): same idea for AST failures (missing optional extra,
1739+
zero-node anomalous extract). Blanks BOTH ``ast_hash`` and
1740+
``semantic_hash`` on the seeded row so either detect_incremental kind
1741+
re-queues the file after the failure is fixed, without deleting
1742+
graphify-out/.
17361743
"""
17371744
existing = load_manifest(manifest_path, root=root)
17381745

@@ -1749,6 +1756,7 @@ def _path_index(paths: set[str] | list[str] | None) -> set[str] | None:
17491756

17501757
scan_set = _path_index(scan_corpus)
17511758
clear_set = _path_index(clear_semantic)
1759+
clear_ast_set = _path_index(clear_ast)
17521760
try:
17531761
root_res: Path | None = Path(root).resolve() if root is not None else None
17541762
except (OSError, RuntimeError):
@@ -1764,6 +1772,8 @@ def _in_scan(path_str: str) -> bool:
17641772
return False
17651773

17661774
def _in_clear(path_str: str) -> bool:
1775+
if clear_set is None:
1776+
return False
17671777
if path_str in clear_set or _nfc(path_str) in clear_set:
17681778
return True
17691779
try:
@@ -1772,6 +1782,17 @@ def _in_clear(path_str: str) -> bool:
17721782
except (OSError, RuntimeError):
17731783
return False
17741784

1785+
def _in_clear_ast(path_str: str) -> bool:
1786+
if clear_ast_set is None:
1787+
return False
1788+
if path_str in clear_ast_set or _nfc(path_str) in clear_ast_set:
1789+
return True
1790+
try:
1791+
resolved = str(Path(path_str).resolve())
1792+
return resolved in clear_ast_set or _nfc(resolved) in clear_ast_set
1793+
except (OSError, RuntimeError):
1794+
return False
1795+
17751796
def _in_root(path_str: str) -> bool:
17761797
# Without a root we cannot tell in-root from out-of-root; fail open
17771798
# (keep the row) so out-of-root corpora are never pruned by accident.
@@ -1817,7 +1838,11 @@ def _normalise_entry(entry):
18171838
continue
18181839
if scan_set is not None and not _in_scan(f) and _in_root(f):
18191840
continue # excluded-but-alive: drop the stale row (#1908)
1820-
if clear_set is not None and _in_clear(f):
1841+
if clear_ast_set is not None and _in_clear_ast(f):
1842+
# AST failure this run (missing extra / zero nodes, #2543): blank
1843+
# both hashes so either detect_incremental kind re-queues.
1844+
normalised = {**normalised, "ast_hash": "", "semantic_hash": ""}
1845+
elif clear_set is not None and _in_clear(f):
18211846
# Dispatched-but-omitted this run: don't inherit the stale
18221847
# semantic_hash, or detect_incremental would call it unchanged (#1948).
18231848
normalised = {**normalised, "semantic_hash": ""}

‎graphify/llm.py‎

Lines changed: 36 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1373,6 +1373,30 @@ def _claude_cli_envelope(stdout: str) -> dict:
13731373
return envelope
13741374

13751375

1376+
def _claude_cli_error(stdout: str) -> str:
1377+
"""Return the CLI's own error text when the envelope flags `is_error`.
1378+
1379+
`claude -p` reports API failures (rate limits, auth) in the stdout JSON
1380+
envelope with `is_error: true` and leaves stderr EMPTY — and on a rate limit
1381+
it still exits 0. So the two obvious checks both miss it: a non-zero exit
1382+
printed a bare "exited 1: " with no cause, and a zero exit fed the error
1383+
string to the JSON parser, producing an empty graph that `_response_is_hollow`
1384+
misread as truncation and adaptive retry then bisected, re-issuing requests
1385+
that were still being refused (#2554). Best-effort: unparseable stdout is not
1386+
this function's problem, the caller's `_claude_cli_envelope` reports that.
1387+
"""
1388+
try:
1389+
envelope = _claude_cli_envelope(stdout)
1390+
except RuntimeError:
1391+
return ""
1392+
if not envelope.get("is_error"):
1393+
return ""
1394+
detail = envelope.get("result")
1395+
if isinstance(detail, str) and detail.strip():
1396+
return detail.strip()
1397+
return "unspecified error"
1398+
1399+
13761400
# A JSON Schema pinning the top-level shape graphify consumes. Passed to
13771401
# `claude -p --json-schema` (structured output) so the CLI CONSTRAINS the model
13781402
# to emit the object directly instead of relying on it CHOOSING to honour a
@@ -1538,10 +1562,12 @@ def _call_claude_cli(user_message: str, max_tokens: int = 8192, *, deep_mode: bo
15381562
check=False,
15391563
**_no_window_kwargs(),
15401564
)
1565+
cli_error = _claude_cli_error(proc.stdout)
15411566
if proc.returncode != 0:
1542-
raise RuntimeError(
1543-
f"claude -p exited {proc.returncode}: {proc.stderr.strip()[:500]}"
1544-
)
1567+
detail = proc.stderr.strip() or cli_error or "(no stderr, no error envelope)"
1568+
raise RuntimeError(f"claude -p exited {proc.returncode}: {detail[:500]}")
1569+
if cli_error:
1570+
raise RuntimeError(f"claude -p reported an error: {cli_error[:500]}")
15451571

15461572
envelope = _claude_cli_envelope(proc.stdout)
15471573

@@ -2597,8 +2623,14 @@ def _rec(inp, out) -> None:
25972623
check=False,
25982624
**_no_window_kwargs(),
25992625
)
2626+
cli_error = _claude_cli_error(proc.stdout)
26002627
if proc.returncode != 0:
2601-
raise RuntimeError(f"claude -p exited {proc.returncode}: {proc.stderr.strip()[:500]}")
2628+
detail = proc.stderr.strip() or cli_error or "(no stderr, no error envelope)"
2629+
raise RuntimeError(f"claude -p exited {proc.returncode}: {detail[:500]}")
2630+
if cli_error:
2631+
# Without this the error text is returned as the model's reply and
2632+
# the caller writes it into the graph as a community label (#2554).
2633+
raise RuntimeError(f"claude -p reported an error: {cli_error[:500]}")
26022634
envelope = _claude_cli_envelope(proc.stdout)
26032635
cli_usage = envelope.get("usage") or {}
26042636
if cli_usage:

0 commit comments

Comments
 (0)