-
Notifications
You must be signed in to change notification settings - Fork 2
Expand file tree
/
Copy pathshipgate-detect.py
More file actions
2141 lines (1952 loc) · 87.7 KB
/
Copy pathshipgate-detect.py
File metadata and controls
2141 lines (1952 loc) · 87.7 KB
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
#!/usr/bin/env python3
"""Zero-install Agents Shipgate detector.
Replicates the structural output of ``agents-shipgate detect --json`` for
the most common decision a coding agent needs to make — *is this an agent
project, and which framework(s)?* — without requiring a local install of
the ``agents-shipgate`` package. Stdlib-only, one file.
Usage::
python3 tools/shipgate-detect.py [--workspace PATH] [--json]
curl -sSL https://raw.githubusercontent.com/ThreeMoonsLab/agents-shipgate/main/tools/shipgate-detect.py \\
| python3 - --workspace . --json
Output mirrors :class:`agents_shipgate.cli.discovery.signals.DetectResult`
plus a ``script_version`` field. It is a **structural subset** of the
canonical ``agents-shipgate detect --json`` output, NOT a drop-in
replacement: the CLI also emits ``diagnostics[]`` and ``next_actions[]``
arrays (the diagnostic engine), which are intentionally out of scope for
the zero-install path. The contract test pins the verdict — ``is_agent_project``,
fired frameworks, suggested sources, excluded sources — against the CLI on
every sample in ``samples/``, so the two cannot drift on the load-bearing
fields.
Both this script and the canonical CLI silently skip common fixture corpus
directories (for example ``fixtures/``, ``testdata/``, and ``golden/``) when
those directories are below the selected workspace. Point ``--workspace``
directly at a fixture project to detect that fixture itself.
The workspace inventory matches the canonical CLI's: ``git ls-files`` when
the workspace is a repository Git can read, a contained filesystem walk
otherwise. That is not a performance detail — a ``.gitignore``d module is
invisible to ``init``, so a script that walked it anyway could name an agent
``init`` will never write. Paths that escape the workspace through a symlink
are dropped for the same reason. The bound is on Python *parses*
(``MAX_PYTHON_FILES``), never on the inventory, so an asset-heavy repository
cannot exhaust the budget before the walk reaches any source.
Like the canonical CLI, glob-matched MCP/OpenAPI candidates are
parse-probed before they are suggested: a filename is a glob match, not a
guarantee. A Cursor plugin ``mcp.json`` is an ``mcpServers``-style host
config, not an MCP tools-array export — suggesting it would make the very
next ``agents-shipgate init --write`` → ``scan`` step fail. Rejected
candidates move to ``excluded_sources[]`` (``{type, path, reason}``)
instead of ``suggested_sources``.
Intentional simplifications vs. the canonical CLI:
- No ``diagnostics[]`` / ``next_actions[]`` (the diagnostic engine is
not in scope for stdlib-only / zero-install).
- ``agent_scope`` / ``agent_scope_truncated`` / ``python_parse_truncated`` /
``agent_project_candidates[]``
are carried, and the contract test pins them against the CLI: an agent that
consults the zero-install path must not adopt a manifest scope the CLI
refuses, nor read a candidate list the cap cut short as an enumeration.
- Descriptive (not byte-identical) ``evidence`` / ``reason`` strings.
- Absolute scores may differ by ±0.5 in edge cases.
- The parse probe is **JSON-only** (stdlib has no YAML parser). A
``.json`` candidate the input adapters would reject is excluded here
too; a ``.yaml`` / ``.yml`` OpenAPI spec is kept as a suggestion
unconditionally (never wrongly dropped). The real-world miss this
guards against — ``mcpServers``-style host configs — is always JSON,
so the probe is exact where it matters.
The verdict, detected framework set, suggested/excluded source split, and
the ranked ``agent_name_candidates`` all match. The name ranking is pinned
rather than simplified: it decides which agent a manifest declares as the
reviewed identity, and a script that ranked differently from ``init`` would
send an agent to fix the wrong one.
"""
from __future__ import annotations
import argparse
import ast
import fnmatch
import json
import os
import re
import subprocess
import sys
import threading
from pathlib import Path
from typing import Any
SCRIPT_VERSION = "0.4.0"
MAX_STRUCTURED_FILE_BYTES = 10 * 1024 * 1024
# Matches ``detect_workspace``'s ``max_python_files``. The bound is on
# parses, not on the inventory: capping the inventory lets an asset-heavy
# repository exhaust the budget before the walk reaches any source.
MAX_PYTHON_FILES = 1000
# Framework signal vocabulary (mirror of cli/discovery/signals.py).
LANGCHAIN_IMPORTS = {
"langchain", "langchain.agents", "langchain.tools", "langchain_core",
"langchain_core.tools", "langchain_core.agents", "langgraph",
"langgraph.graph", "langgraph.prebuilt",
}
LANGCHAIN_DECORATOR_MODULES = {"langchain.tools", "langchain_core.tools"}
LANGCHAIN_AGENT_CALLS = {"create_agent", "create_react_agent", "AgentExecutor"}
CREWAI_IMPORTS = {"crewai", "crewai.tools", "crewai_tools"}
CREWAI_DECORATOR_MODULES = {"crewai.tools"}
CREWAI_CLASSES = {"Agent", "Crew", "Task"}
GOOGLE_ADK_CLASSES = {
"Agent", "LlmAgent", "FunctionTool", "LongRunningFunctionTool",
"OpenAPIToolset", "McpToolset", "MCPToolset",
}
ANTHROPIC_IMPORTS = {"anthropic"}
OPENAI_AGENTS_SDK_IMPORTS = {"agents", "openai_agents"}
OPENAI_AGENTS_SDK_DECORATORS = {
"function_tool", "agents.function_tool", "openai_agents.function_tool",
}
PACKAGE_HINTS: dict[str, tuple[str, ...]] = {
"langchain": ("langchain", "langchain-core", "langchain_core", "langgraph"),
"crewai": ("crewai", "crewai-tools"),
"google_adk": ("google-adk", "google_adk", "google-genai"),
"anthropic": ("anthropic",),
"openai_agents_sdk": ("openai-agents", "openai_agents", "agents"),
"n8n": ("n8n", "@n8n/n8n-nodes-langchain"),
"conductor": ("conductor-client", "conductor-server", "conductor-oss"),
"openai_api": (),
}
FRAMEWORKS = (
"langchain", "crewai", "google_adk", "anthropic",
"openai_agents_sdk", "n8n", "conductor", "openai_api",
)
OPENAPI_PATTERNS = (
"*openapi*.yaml", "*openapi*.yml", "*openapi*.json",
"*swagger*.yaml", "*swagger*.yml", "*swagger*.json",
)
MCP_PATTERNS = ("*mcp*.json", ".agents-shipgate/*.json")
ANTHROPIC_TOOL_PATTERNS = ("tools/*anthropic*tools*.json", "tools/anthropic-tools.json")
ANTHROPIC_POLICY_PATTERNS = ("policies/*anthropic*.yaml", "policies/anthropic-policy.yaml")
N8N_WORKFLOW_PATTERNS = (
"workflows/*.json", "workflows/**/*.json",
"n8n/*.json", "n8n/**/*.json",
"*workflow*.json",
)
CONDUCTOR_WORKFLOW_PATTERNS = (
"workflows/*.json", "workflows/**/*.json",
"conductor/*.json", "conductor/**/*.json",
"ai/examples/*.json", "ai/examples/**/*.json",
"*workflow*.json",
)
OPENAI_API_PATTERNS = (
("openai-config.json", "openai-config marker"),
("tools/*openai*tools*.json", "openai tool file"),
("policies/*openai*.yaml", "openai-api policy file"),
("policies/*api*.yaml", "openai-api policy file"),
("tests/*openai*cases*.json", "openai-api test cases"),
("tests/*api*cases*.json", "openai-api test cases"),
)
CONVENTIONAL_DIRS = ("prompts", "tools", ".agents-shipgate")
# Agent-name evidence vocabulary (mirror of cli/discovery/signals.py). The
# ranking below is pinned to the CLI's by tests/test_zero_install_detector.py:
# two rankings that disagree would have `init` and this script name different
# agents as the reviewed identity.
AGENT_NAME_CLASSES = {"Agent", "LlmAgent"}
APP_ROOT_CLASSES = {"App"}
# Modules an aliased agent/app constructor may legitimately come from. An
# alias is only read as a framework constructor when its module is one of
# these; otherwise `X as Agent`-style renames of unrelated classes would
# widen recognition instead of sharpening it.
AGENT_FRAMEWORK_MODULE_PREFIXES = (
"google.adk",
"agents",
"openai_agents",
"crewai",
"langchain",
"langchain_core",
"langgraph",
)
ROOT_AGENT_SYMBOL = "root_agent"
CHILD_AGENT_KEYWORDS = ("sub_agents", "handoffs")
# Origin is meant to dominate hierarchy and corroboration, so the test
# penalty is strictly greater than their whole spread (3.0 + 1.0 + 1.5).
# Conventional test module filenames that carry no `test_` prefix.
TEST_MODULE_NAMES = frozenset({"conftest.py", "test.py", "tests.py"})
ROOT_AGENT_BONUS = 3.0
SUB_AGENT_PENALTY = 1.5
CORROBORATION_BONUS = 1.0
ORIGIN_TEST_PENALTY = 6.0
QUALITY_FLOOR_PENALTY = 3.0
AGENT_NAME_MIN_LENGTH = 3
GENERIC_AGENT_NAME_VALUES = frozenset({
"agent", "agents", "bar", "baz", "changeme", "dummy", "example", "foo",
"myagent", "name", "placeholder", "qux", "sample", "temp", "test",
"tests", "tmp", "todo", "untitled",
})
# Files that mark a self-contained project root. Mirrors
# ``agents_shipgate.cli.discovery.scope.PROJECT_MARKERS`` — the canonical CLI
# and this script must agree on which directory a manifest describes, or an
# agent that consults the zero-install path adopts a scope the CLI refuses.
PROJECT_MARKERS = (
"shipgate.yaml",
"pyproject.toml",
"setup.py",
"package.json",
"go.mod",
"Cargo.toml",
"pom.xml",
"build.gradle",
"build.gradle.kts",
"composer.json",
"Gemfile",
)
# Markers that name a project only where agent evidence sits in the same
# directory. Mirrors ``scope.WEAK_PROJECT_MARKERS``.
WEAK_PROJECT_MARKERS = (
"requirements.txt",
"requirements.in",
)
SKIP_DIRS = {
".agents-private", ".cache", ".claude", ".direnv", ".git", ".hg",
".nox", ".svn", ".mypy_cache", ".next", ".pnpm-store", ".pytest_cache",
".ruff_cache", ".turbo", ".tox", ".venv", "__pycache__",
"agents-shipgate-reports", "build", "dist", "env", "node_modules",
"target", "venv", "fixtures", "_fixtures", "__fixtures__", "golden",
"goldens", "test-fixtures", "test_fixtures", "test_data", "testdata",
}
PYPROJECT_NAME_RE = re.compile(r'^\s*name\s*=\s*["\']([^"\']+)["\']', re.MULTILINE)
REQ_TOKEN_RE = re.compile(r"^\s*([A-Za-z0-9_.\-]+)", re.MULTILINE)
MAX_GIT_INVENTORY_BYTES = 16 * 1024 * 1024
# Ceiling for the non-Git fallback walk. Deliberately far above any real
# agent repository: it is a refusal threshold for pathological inputs, not
# a working limit, and exceeding it raises rather than truncating.
MAX_WALK_FILES = 200_000
def _contained(path: Path, workspace: Path) -> Path | None:
"""``path`` itself when it lives in the workspace, else ``None``.
Resolution proves containment and nothing more. Returning the resolved
path instead would *rename* the entry: with ``agent.py -> source.txt``
both inventory entries collapse onto ``source.txt``, the ``.py`` suffix
disappears, and the script reports zero Python files where canonical
detection reports an agent project. A symlink pointing outside is still
dropped — it is not part of the workspace whatever its name suggests,
and ranking a name out of one also leaks the outside absolute path.
"""
try:
resolved = path.resolve()
rel = resolved.relative_to(workspace)
except (OSError, RuntimeError, ValueError):
return None
if any(p in SKIP_DIRS or p.startswith(".venv") for p in rel.parts):
return None
try:
if not resolved.is_file():
return None
except OSError:
return None
return path
class DiscoveryError(RuntimeError):
"""The workspace inventory could not be collected safely.
Mirrors ``core.errors.DiscoveryError``. Canonical discovery *raises*
when the bounded Git inventory overruns rather than falling back to an
unbounded walk, and so does this: the fallback would do the very work
the bound exists to refuse.
"""
def _git_env() -> dict[str, str]:
env = {k: v for k, v in os.environ.items() if not k.startswith("GIT_")}
env.update({
"GIT_ATTR_NOSYSTEM": "1", "GIT_CONFIG_GLOBAL": os.devnull,
"GIT_CONFIG_NOSYSTEM": "1", "GIT_CONFIG_SYSTEM": os.devnull,
"GIT_NO_LAZY_FETCH": "1", "GIT_NO_REPLACE_OBJECTS": "1",
"GIT_OPTIONAL_LOCKS": "0", "GIT_PAGER": "cat",
"GIT_PROTOCOL_FROM_USER": "0", "GIT_TERMINAL_PROMPT": "0",
})
return env
def _git_inventory_bounded(workspace: Path, args: list[str], *,
env: dict[str, str],
max_output_bytes: int) -> bytes | None:
"""Read Git's output incrementally, never buffering more than the cap.
``capture_output=True`` would materialise the whole inventory before any
size check could reject it, which makes the cap decorative. Reading in
chunks and killing the child on overrun is what actually bounds memory.
"""
try:
process = subprocess.Popen(
["git", "--no-replace-objects", "-C", str(workspace), *args],
env=env, stdout=subprocess.PIPE, stderr=subprocess.DEVNULL,
)
except OSError:
return None
output = bytearray()
exceeded = False
failed = False
def drain() -> None:
nonlocal exceeded, failed
assert process.stdout is not None
try:
while chunk := process.stdout.read(64 * 1024):
remaining = max_output_bytes + 1 - len(output)
if remaining > 0:
output.extend(chunk[:remaining])
if len(output) > max_output_bytes:
exceeded = True
try:
process.kill()
except OSError:
pass
return
except OSError:
failed = True
reader = threading.Thread(target=drain, daemon=True)
reader.start()
try:
returncode = process.wait(timeout=10)
except subprocess.TimeoutExpired:
process.kill()
process.wait()
reader.join()
return None
reader.join()
if returncode != 0 or exceeded or failed:
return None
return bytes(output)
def _git_files(workspace: Path) -> list[Path] | None:
"""The workspace inventory as Git sees it, or ``None`` if Git cannot.
Canonical detection prefers this, and matching it is what makes the
ranking parity claim true: without it a `.gitignore`d file is invisible
to `init` but visible here, so the two would name different agents.
``None`` means "Git cannot answer" (not installed, not a repository) and
the caller falls back to a contained walk. An inventory that overruns
the bound raises instead, matching canonical discovery.
"""
env = _git_env()
try:
root = subprocess.run(
["git", "--no-replace-objects", "-C", str(workspace),
"rev-parse", "--show-toplevel"],
check=False, capture_output=True, env=env, timeout=10,
)
except (OSError, subprocess.SubprocessError):
return None
if root.returncode != 0:
return None
try:
git_root = Path(root.stdout.decode("utf-8").strip()).resolve()
except (UnicodeDecodeError, OSError, RuntimeError, ValueError):
return None
listed = _git_inventory_bounded(
workspace,
[
"-c", "core.fsmonitor=false", "-c", "submodule.recurse=false",
"-c", "core.quotePath=false",
"ls-files", "-co", "--exclude-standard", "--full-name", "-z", "--", ".",
],
env=env,
max_output_bytes=MAX_GIT_INVENTORY_BYTES,
)
if listed is None:
raise DiscoveryError(
"Git candidate-file inventory exceeded static output bounds or "
"could not be collected safely."
)
out: list[Path] = []
for raw in listed.split(b"\0"):
if not raw:
continue
try:
rel = raw.decode("utf-8")
except UnicodeDecodeError:
continue
contained = _contained(git_root / rel, workspace)
if contained is not None:
out.append(contained)
return sorted(set(out))
def _walk_files(workspace: Path) -> list[Path]:
"""Fallback inventory when Git cannot answer. Uncapped, and contained.
Never *silently* truncated: the manifest-scope verdict is computed from
where project markers sit, so dropping entries drops that verdict too —
with enough filler ahead of them two nested agent projects vanish and
this script reports one scope where the CLI reports ambiguity (#363).
The bound that shapes the work is on Python *parses*
(``MAX_PYTHON_FILES``), which is where the canonical CLI puts it.
``MAX_WALK_FILES`` is a ceiling, not a cap: a workspace past it raises
rather than returning a partial inventory, because a verdict computed
from part of a repository is a verdict about part of a repository. It
exists so a downloaded tree of millions of unrelated assets cannot
consume unbounded time and memory before detection sees any source.
"""
out: list[Path] = []
seen = 0
for root, dirs, files in os.walk(workspace):
dirs[:] = [
d for d in dirs
if d not in SKIP_DIRS and not d.startswith(".venv")
]
for fn in files:
# Counted per entry, not per directory: one directory holding
# the whole tree would otherwise sail past the ceiling.
seen += 1
if seen > MAX_WALK_FILES:
raise DiscoveryError(
f"Workspace inventory exceeds {MAX_WALK_FILES} files without "
"Git to bound it. Run inside the Git repository, or point "
"--workspace at the project directory you are adopting."
)
contained = _contained(Path(root) / fn, workspace)
if contained is not None:
out.append(contained)
return sorted(set(out))
def _inventory(workspace: Path) -> list[Path]:
"""Mirror of ``artifacts._candidate_files``: Git when it can answer,
a contained filesystem walk otherwise. Deliberately uncapped — the cap
that matters is on Python *parses* (see ``MAX_PYTHON_FILES``), and a
global file cap could exhaust itself on assets before reaching any
source at all."""
git_files = _git_files(workspace)
if git_files is not None:
return git_files
return _walk_files(workspace)
def _project_marker(directory: Path, extra: tuple[str, ...] = ()) -> str | None:
for name in (*PROJECT_MARKERS, *extra):
candidate = directory / name
# A symlink is not a marker: the verifier refuses a manifest path
# with symlink components, so accepting one here would name a
# directory whose scoped command cannot run.
if candidate.is_symlink():
continue
if candidate.is_file():
return name
return None
def _project_of(
path: Path,
workspace: Path,
evidence_dirs: frozenset[Path] = frozenset(),
) -> tuple[str, str | None] | None:
"""Nearest project root at or above ``path``, as (relative, marker)."""
directory = path if path.is_dir() else path.parent
while True:
extra = WEAK_PROJECT_MARKERS if directory in evidence_dirs else ()
marker = _project_marker(directory, extra)
if marker is not None:
rel = _rel(directory, workspace) if directory != workspace else "."
return rel, marker
if directory == workspace:
return None
parent = directory.parent
if parent == directory:
return None
directory = parent
def _agent_project_candidates(
workspace: Path,
evidence_paths: list[str],
literals_by_path: dict[str, list[str]],
) -> list[dict[str, Any]]:
"""Group agent evidence by the project each piece of it sits in.
Same rule as the canonical CLI: a workspace whose agents live in more
than one self-contained project is not one manifest's scope.
"""
names: dict[str, set[str]] = {}
markers: dict[str, str | None] = {}
evidence_dirs = frozenset(
(workspace / rel) if (workspace / rel).is_dir() else (workspace / rel).parent
for rel in evidence_paths
)
for rel in evidence_paths:
found = _project_of(workspace / rel, workspace, evidence_dirs)
project, marker = (
found
if found is not None
else (".", _project_marker(workspace, WEAK_PROJECT_MARKERS))
)
names.setdefault(project, set()).update(literals_by_path.get(rel, []))
markers.setdefault(project, marker)
return [
{"path": project, "marker": markers[project], "agent_names": sorted(found)}
for project, found in sorted(names.items())
]
def _rel(path: Path, workspace: Path) -> str:
"""Workspace-relative path, by its logical name.
The logical name is tried first on purpose: re-resolving here would
undo `_contained`'s guarantee and rename a symlinked ``agent.py`` to
its target, which is how the inventory lost its Python files.
"""
try:
return path.relative_to(workspace).as_posix()
except ValueError:
pass
try:
return path.resolve().relative_to(workspace.resolve()).as_posix()
except (OSError, RuntimeError, ValueError):
return path.as_posix()
def _matches(rel: str, basename: str, pattern: str) -> bool:
if fnmatch.fnmatch(rel, pattern):
return True
if "/" not in pattern:
return fnmatch.fnmatch(basename, pattern)
return fnmatch.fnmatch(rel, f"*/{pattern}")
def _glob(workspace: Path, files: list[Path], patterns: tuple[str, ...]) -> list[str]:
found: list[str] = []
seen: set[str] = set()
for pattern in patterns:
for p in files:
rel = _rel(p, workspace)
if rel in seen or not _matches(rel, p.name, pattern):
continue
seen.add(rel)
found.append(rel)
return sorted(found)
def _looks_like_n8n_workflow(path: Path) -> bool:
"""Match the CLI heuristic in cli/discovery/artifacts.py: a JSON file
is an n8n workflow when it (or any element in a list) is a dict with
a ``nodes`` list and ``connections`` dict, and at least one node has
a ``type`` starting with ``n8n-nodes-`` or ``@n8n/n8n-nodes-``."""
if path.suffix.lower() != ".json":
return False
try:
data = json.loads(path.read_text(encoding="utf-8"))
except (OSError, UnicodeDecodeError, json.JSONDecodeError):
return False
candidates = data if isinstance(data, list) else [data]
for item in candidates:
if not isinstance(item, dict):
continue
nodes = item.get("nodes")
connections = item.get("connections")
if not isinstance(nodes, list) or not isinstance(connections, dict):
continue
for node in nodes:
if not isinstance(node, dict):
continue
node_type = node.get("type")
if isinstance(node_type, str) and (
node_type.startswith("n8n-nodes-")
or node_type.startswith("@n8n/n8n-nodes-")
):
return True
return False
def _conductor_agent_markers(data: Any) -> set[str]:
candidates = data if isinstance(data, list) else [data]
if not candidates:
return set()
for item in candidates:
if not (
isinstance(item, dict)
and isinstance(item.get("name"), str)
and bool(item["name"].strip())
and isinstance(item.get("tasks"), list)
and bool(item["tasks"])
and item.get("schemaVersion", 2) == 2
):
return set()
markers: set[str] = set()
def walk(value: Any) -> None:
if isinstance(value, dict):
task_type = value.get("type")
if task_type in {"CALL_MCP_TOOL", "LIST_MCP_TOOLS", "LLM_CHAT_COMPLETE"}:
markers.add(str(task_type))
for nested in value.values():
walk(nested)
elif isinstance(value, list):
for nested in value:
walk(nested)
walk(data)
return markers
def _probe_suggested(workspace: Path, rel: str, kind: str) -> str | None:
"""Return ``None`` if the input adapters would accept ``rel`` as a
``kind`` tool source, else a one-line reason ``scan`` would reject it.
Stdlib mirror of
:func:`agents_shipgate.cli.discovery.artifacts.probe_suggested_source`
(which calls the real ``load_mcp_tools`` / ``load_openapi_tools``).
The probe is JSON-only — see the module docstring — so an unparseable
or YAML candidate is kept as a suggestion rather than wrongly dropped.
The MCP suggestion globs are all ``*.json`` / ``.agents-shipgate/*.json``,
so the load-bearing ``mcpServers``-host-config case is always covered.
"""
path = workspace / rel
if kind == "openapi" and path.suffix.lower() in (".yaml", ".yml"):
return None # No stdlib YAML parser — keep, never wrongly exclude.
try:
data = json.loads(path.read_text(encoding="utf-8"))
except (OSError, UnicodeDecodeError, json.JSONDecodeError):
if path.suffix.lower() == ".json":
# A .json the adapter would also fail to parse (exit 3).
return f"Unable to parse input file: {rel}"
return None # Non-JSON we can't read — conservative keep.
if kind == "mcp":
return _probe_mcp(data)
if kind == "conductor":
return None if _conductor_agent_markers(data) else (
"not a Conductor AI/MCP workflow JSON document"
)
return _probe_openapi(data)
def _probe_mcp(data: Any) -> str | None:
"""Mirror ``load_mcp_tools``'s accept rule (inputs/mcp.py)."""
if isinstance(data, list):
return None # Top-level tools array.
if not isinstance(data, dict):
return "MCP tools file must be an object or array"
if isinstance(data.get("mcpServers"), dict) or isinstance(data.get("servers"), dict):
# Host MCP *configuration* (e.g. a Cursor/Claude plugin manifest),
# which the mcp adapter never accepts as a tools export.
return (
"mcpServers-style MCP server config (host configuration), "
"not an MCP tools-array export"
)
raw_tools = data.get("tools")
if data.get("wildcard") is True or raw_tools == "*":
if isinstance(raw_tools, list) and raw_tools:
return "MCP source declares wildcard tool exposure and an explicit tools array"
return None # Wildcard exposure.
if not isinstance(raw_tools, list):
return "MCP tools file must contain a tools array"
return None
def _probe_openapi(data: Any) -> str | None:
"""Mirror ``load_openapi_tools``'s accept rule (inputs/openapi.py)."""
if not isinstance(data, dict):
return "OpenAPI file must contain an object"
if "openapi" not in data:
# Catches Swagger 2.0 (keyed ``swagger:``) and non-OpenAPI JSON.
return "OpenAPI file missing 'openapi' version"
if not isinstance(data.get("paths"), dict):
return "OpenAPI file missing paths object"
return None
def _name(node: ast.AST) -> str | None:
if isinstance(node, ast.Call):
return _name(node.func)
if isinstance(node, ast.Name):
return node.id
if isinstance(node, ast.Attribute):
prefix = _name(node.value)
return f"{prefix}.{node.attr}" if prefix else node.attr
return None
def _parse_py(path: Path) -> dict[str, Any] | None:
try:
tree = ast.parse(path.read_text(encoding="utf-8"), filename=str(path))
except (OSError, UnicodeDecodeError, SyntaxError):
return None
imports, decos, ctors, names = set(), set(), set(), []
constant_imports: dict[tuple[int, str], tuple[str, int, str]] = {}
plain_imports: dict[str, str] = {}
writes: dict[str, list[dict[str, Any]]] = {}
scopes = _walk_scoped(tree)
nodes = scopes["nodes"]
scope_parents, class_scopes = scopes["parents"], scopes["class_scopes"]
hierarchy = _new_hierarchy(scope_parents, class_scopes, writes)
attribute_writes: set[str] = set()
star_linenos: list[int] = []
agent_calls: list[tuple[ast.Call, int, bool]] = []
agent_targets: dict[int, ast.Call] = {}
write_by_node: dict[int, dict[str, Any]] = {}
global_decls: dict[int, set[str]] = {}
nonlocal_decls: dict[int, set[str]] = {}
star_import = False
pending_calls: list[tuple[ast.Call, str, int, bool]] = []
def _write(name: str, scope: int, lineno: int, conditional: bool,
kind: str = "assignment") -> dict[str, Any]:
entry = {
"scope": scope, "lineno": lineno, "conditional": conditional,
"call_id": None, "kind": kind,
}
writes.setdefault(name, []).append(entry)
return entry
for node, scope, conditional in nodes:
if isinstance(node, ast.Import):
for a in node.names:
imports.add(a.name)
bound = (a.asname or a.name).split(".")[0]
_write(bound, scope, node.lineno, conditional, "import")
# `import a.b.c` binds `a` denoting `a`; `import a.b.c as x`
# binds `x` denoting `a.b.c`.
plain_imports[bound] = (
a.name if a.asname else a.name.split(".")[0]
)
elif isinstance(node, ast.ImportFrom):
if node.module:
imports.add(node.module)
for a in node.names:
imports.add(f"{node.module}.{a.name}")
for a in node.names:
if a.name == "*":
star_import = True
star_linenos.append(node.lineno)
continue
bound = a.asname or a.name
_write(bound, scope, node.lineno, conditional, "import")
if node.module or node.level:
constant_imports[(scope, bound)] = (
node.module or "", node.level, a.name,
)
elif isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
_write(node.name, scope, node.lineno, conditional, "definition")
for d in node.decorator_list:
n = _name(d)
if n:
decos.add(n)
elif isinstance(node, ast.Name) and isinstance(node.ctx, ast.Store):
write_by_node[id(node)] = _write(node.id, scope, node.lineno, conditional)
elif isinstance(node, ast.Attribute) and isinstance(
node.ctx, (ast.Store, ast.Del)
):
dotted = _name(node)
if dotted:
attribute_writes.add(dotted)
elif isinstance(node, ast.Name) and isinstance(node.ctx, ast.Del):
_write(node.id, scope, node.lineno, conditional, "delete")
elif isinstance(node, ast.arg):
_write(node.arg, scope, node.lineno, conditional, "parameter")
elif isinstance(node, ast.ExceptHandler) and node.name:
_write(node.name, scope, node.lineno, conditional, "except")
elif isinstance(node, (ast.MatchAs, ast.MatchStar)) and node.name:
_write(node.name, scope, node.lineno, conditional, "match")
elif isinstance(node, ast.MatchMapping) and node.rest:
_write(node.rest, scope, node.lineno, conditional, "match")
elif isinstance(node, ast.Global):
global_decls.setdefault(scope, set()).update(node.names)
elif isinstance(node, ast.Nonlocal):
nonlocal_decls.setdefault(scope, set()).update(node.names)
elif isinstance(node, ast.Call):
ctor = _name(node.func)
if ctor:
ctors.add(ctor)
pending_calls.append((node, ctor, scope, conditional))
if isinstance(node, (ast.Assign, ast.AnnAssign)):
targets = (
list(node.targets) if isinstance(node, ast.Assign) else [node.target]
)
for target in targets:
if isinstance(target, ast.Name) and isinstance(node.value, ast.Call):
agent_targets[id(target)] = node.value
facts: dict[str, Any] = {
"imports": imports,
"decorators": decos,
"constructors": ctors,
"names": names,
"constant_imports": constant_imports,
"plain_imports": plain_imports,
"writes": writes,
"scope_parents": scope_parents,
"class_scopes": class_scopes,
"star_import": star_import,
"star_imports": star_linenos,
"attribute_writes": attribute_writes,
}
hierarchy["star_import"] = star_import
_apply_scope_declarations(facts, global_decls, nonlocal_decls)
roles = [
(call, _constructor_role(ctor, scope, call.lineno, facts), scope, conditional)
for call, ctor, scope, conditional in pending_calls
]
for call, role, scope, conditional in roles:
if role == "agent":
agent_calls.append((call, scope, conditional))
hierarchy["agent_call_ids"].add(id(call))
for call, role, scope, conditional in roles:
if role is not None:
_observe_call(
hierarchy, call, role, scope,
conditional or scopes["declaration_conditional"].get(scope, False),
)
for node_id, call in agent_targets.items():
entry = write_by_node.get(node_id)
if entry is not None and id(call) in hierarchy["agent_call_ids"]:
entry["call_id"] = id(call)
# Roles are assigned only once the whole module has been seen: an
# App(root_agent=…) binding can appear after the construction it names.
_resolve_references(hierarchy)
agent_calls.sort(key=lambda item: (item[0].lineno, item[0].col_offset))
unresolved_root = ""
for call, scope, _conditional in agent_calls:
evidence = _agent_name_evidence(call, scope, hierarchy)
if evidence is not None:
names.append(evidence)
elif (id(call) in hierarchy["root_calls"]
or id(call) in hierarchy["resolved_root_calls"]):
unresolved_root = "the application root's name is not a static value"
if hierarchy["unresolved_root"]:
unresolved_root = hierarchy["unresolved_root"]
facts["constants"] = _module_constants(tree, facts)
facts["unresolved_root"] = unresolved_root
return facts
_MODULE_SCOPE = 0
_SCOPE_NODES = (ast.FunctionDef, ast.AsyncFunctionDef, ast.Lambda, ast.ClassDef)
# Comprehensions have their own scope in Python 3.
_COMPREHENSION_NODES = (ast.ListComp, ast.SetComp, ast.DictComp, ast.GeneratorExp)
# Constructs whose bodies may or may not run. An assignment under one of
# these is not provably the binding a later reference sees.
_BRANCH_NODES = tuple(node for node in (
ast.If, ast.IfExp, ast.Try, getattr(ast, "TryStar", ast.Try),
ast.While, ast.For, ast.AsyncFor, ast.Match,
))
def _binding_count(facts: dict[str, Any], name: str) -> int:
return len(facts["writes"].get(name, []))
def _walk_scoped(tree: ast.AST) -> dict[str, Any]:
"""Nodes paired with their lexical scope and whether they run conditionally.
Definition *headers* — decorators, defaults, annotations, class bases and
keywords — are walked in the enclosing scope, because that is where Python
evaluates them. Comprehensions get their own scope, with the outermost
iterable evaluated outside it. `declaration_conditional` says whether the
`def`/`class` introducing a scope was itself conditional: the body is
straight-line relative to itself, but everything it claims is contingent
on that branch having run.
"""
out: list[tuple[ast.AST, int, bool]] = []
parents: dict[int, int] = {}
class_scopes: set[int] = set()
declaration_conditional: dict[int, bool] = {_MODULE_SCOPE: False}
def open_scope(node: ast.AST, scope: int, conditional: bool) -> int:
inner = id(node)
parents[inner] = scope
declaration_conditional[inner] = conditional or declaration_conditional.get(
scope, False
)
return inner
def visit(node: ast.AST, scope: int, conditional: bool) -> None:
out.append((node, scope, conditional))
if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.Lambda)):
inner = open_scope(node, scope, conditional)
for header in _definition_header(node):
visit(header, scope, conditional)
for arg in _argument_nodes(node.args):
visit(arg, inner, False)
body = node.body if isinstance(node.body, list) else [node.body]
for statement in body:
visit(statement, inner, False)
return
if isinstance(node, ast.ClassDef):
inner = open_scope(node, scope, conditional)
class_scopes.add(inner)
for header in _definition_header(node):
visit(header, scope, conditional)
for statement in node.body:
visit(statement, inner, False)
return
if isinstance(node, _COMPREHENSION_NODES):
inner = open_scope(node, scope, conditional)
generators = node.generators
if generators:
visit(generators[0].iter, scope, conditional)
for index, generator in enumerate(generators):
visit(generator.target, inner, False)
if index:
visit(generator.iter, inner, True)
for guard in generator.ifs:
visit(guard, inner, True)
for element in _comprehension_elements(node):
visit(element, inner, True)
return
inner_conditional = conditional or isinstance(node, _BRANCH_NODES)
for child in ast.iter_child_nodes(node):
visit(child, scope, inner_conditional)
visit(tree, _MODULE_SCOPE, False)
return {
"nodes": out, "parents": parents, "class_scopes": class_scopes,
"declaration_conditional": declaration_conditional,
}
def _definition_header(node: ast.AST) -> list[ast.expr]:
header: list[ast.expr] = []
header.extend(getattr(node, "decorator_list", []))
bases = getattr(node, "bases", None)
if bases:
header.extend(bases)
for keyword in getattr(node, "keywords", []) or []:
header.append(keyword.value)
returns = getattr(node, "returns", None)
if returns is not None:
header.append(returns)
args = getattr(node, "args", None)
if isinstance(args, ast.arguments):
header.extend(args.defaults)
header.extend(d for d in args.kw_defaults if d is not None)
for arg in _argument_nodes(args):
if arg.annotation is not None:
header.append(arg.annotation)
return header
def _argument_nodes(args: ast.arguments) -> list[ast.arg]:
collected = [*args.posonlyargs, *args.args, *args.kwonlyargs]
if args.vararg is not None:
collected.append(args.vararg)
if args.kwarg is not None:
collected.append(args.kwarg)
return collected
def _comprehension_elements(node: ast.AST) -> list[ast.expr]:
if isinstance(node, ast.DictComp):
return [node.key, node.value]
return [node.elt]
def _new_hierarchy(scope_parents: dict[int, int], class_scopes: set[int],
writes: dict[str, list[dict[str, Any]]]) -> dict[str, Any]:
"""Structural relationships between agent constructions in one module,
accumulated during the single parse walk, then resolved once the module
is fully seen. References are matched to the binding that reaches them —
nearest enclosing scope, latest unconditional assignment before the
reference — not to every assignment sharing the identifier. ``writes``
is shared with the caller and holds *every* binding, not just agent
constructions: a later `root_agent = build_root()` has to be visible or
a stale construction keeps the role."""
return {
"scope_parents": scope_parents,
# Class bodies are scopes for binding but not for closure lookup.
"class_scopes": class_scopes,
"writes": writes,
"root_refs": [],
"child_refs": [],
"root_calls": set(),
"child_calls": set(),
"resolved_root_calls": {},
"resolved_child_calls": {},
"unresolved_root": "",
# Calls proven to construct an agent, filled before any App is read.
"agent_call_ids": set(),
"star_import": False,
}
def _observe_call(hierarchy: dict[str, Any], call: ast.Call, role: str,
scope: int, conditional: bool) -> None:
"""Record what one call says about the agents around it.
``role`` comes from `_constructor_role`, which resolves the callee
through its binding — a call is only "an agent" or "an app" when the
spelling provably is one.
"""
is_app = role == "app"
is_agent = role == "agent"
for kw in call.keywords: