Skip to content

Commit 932dd1d

Browse files
authored
Merge pull request DeusData#2643 from DeusData/feat/infra-followup
feat(infra): env and service joins, kustomize variants, Terraform / Terragrunt, Ansible
2 parents 0ca6e7d + 5ca4326 commit 932dd1d

32 files changed

Lines changed: 7149 additions & 204 deletions

‎Makefile.cbm‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -540,7 +540,9 @@ PIPELINE_SRCS = \
540540
src/pipeline/pass_operator.c \
541541
src/pipeline/pass_shell.c \
542542
src/pipeline/pass_terraform.c \
543+
src/pipeline/pass_ansible.c \
543544
src/pipeline/pass_docker.c \
545+
src/pipeline/pass_envjoin.c \
544546
src/pipeline/pass_ci.c \
545547
src/pipeline/pass_similarity.c \
546548
src/pipeline/pass_semantic_edges.c \

‎README.md‎

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -328,13 +328,20 @@ The install script placed beside the binary is **reported, not deleted**: uninst
328328
- `CALL_REFERENCE`: a callable is used at a supported reference site (for example, a direct value argument) and resolves to one exact target
329329
- `USAGE`: an identifier is used, but a unique callable target is not proven (including ambiguous or complex expressions)
330330
- `IMPORTS`, `DEFINES`, `IMPLEMENTS`, `INHERITS`
331-
- `HTTP_CALLS`, `ASYNC_CALLS` (cross-service)
331+
- `HTTP_CALLS`, `ASYNC_CALLS` (cross-service); a shell script, Make recipe or CI step running `curl` / `wget` on a URL it states (constant words resolved, unknown parts `{}`) to its `Route` with the method its flags give: a Kubernetes Service of the repository by the path, as code clients do, any other host (a download, localhost) with the host kept; the edge keeps `via`, `method`, `url_path`, `host`, never the user info or the query
332332
- `SPAWNS`: the caller starts another program (`subprocess.run`, `exec.Command`, `posix_spawn`, `Process.Start`, a shell command or Make recipe running a program by path, ...); the target is a `Process` node named after the program, or `<dynamic>` when the program is not a literal. When a shell script, Make recipe, GitHub Actions step or Prow job runs a script of the repository (`./hack/x.sh`, `bash hack/x.sh`), the target is that script's `Module` (else its `File`); `make TARGET` (also `$(MAKE)`, `-C`, `-f`) targets the rule of that target, followed through included makefiles; edges from CI carry `ci` and `job`. A Dockerfile's last stage `ENTRYPOINT` (else `CMD`) spawns what the container runs (`api: dockerfile`): the repository script or source copied into the image, the Go `main` of a binary a stage builds with `go build -o`, else a `Process`
333333
- `INFRA_MAPS`: a Kubernetes Service to each workload whose pod labels carry every `key: value` of the Service's selector, within one namespace
334+
- `CONNECTS_TO`: a Kubernetes workload or compose service to the Service its containers' environment addresses (`PRODUCT_CATALOG_SERVICE_ADDR: productcatalogservice:3550`, a URL's host, `<svc>.<ns>.svc...`, a bare name on an `*_ADDR` / `*_HOST` / `*_URL` / ... key; compose: the project's `.env` interpolated): a Kubernetes Service of the same bundle and namespace, or a compose service of the same file. The edge keeps `via: env`, `key` and `container`, never the value
334335
- `USAGE` with `ref: image`: a Kubernetes workload (any pod template: containers, initContainers, CronJobs, custom resources) or a docker compose service to the `Image` it runs
335-
- `DEPLOYS`: what puts manifests, kustomizations or charts on a cluster, to what it puts there: a Flux Kustomization or Argo Application to the kustomizations / manifests / chart of its path, a Flux HelmRelease to its chart (`spec.chart` or `chartRef` → OCIRepository), a shell command or Make recipe (`kubectl apply -f/-k`, `kustomize build | kubectl apply`, `helm install/upgrade/template`), a Go installer loading a chart with the Helm SDK, a Terraform `helm_release`. A chart outside the repository is a shared external `Chart`; the edge says `via` (`flux`, `argo`, `helmrelease`, `kubectl`, `kustomize`, `helm`, `helm-sdk`, `terraform`, ...)
336+
- `DEPLOYS`: what puts manifests, kustomizations or charts on a cluster, to what it puts there: a Flux Kustomization or Argo Application to the kustomizations / manifests / chart of its path, a Flux HelmRelease to its chart (`spec.chart` or `chartRef` → OCIRepository), a shell command or Make recipe (`kubectl apply -f/-k`, `kustomize build | kubectl apply`, `helm install/upgrade/template`), a Go installer loading a chart with the Helm SDK, a Terraform `helm_release`, a Terraform `kubernetes_*` resource the object it declares (a `Resource` like a manifest's, `ecosystem: terraform`, when its name is a literal), a kustomization inflating a chart (`helmCharts`; its values file an `IMPORTS`). A chart outside the repository is a shared external `Chart`; the edge says `via` (`flux`, `argo`, `helmrelease`, `kubectl`, `kustomize`, `helm`, `helm-sdk`, `terraform`, `helmcharts`, ...)
336337
- `DEPENDS_ON`: a Helm chart to the charts it declares (the local chart for a resolvable `file://` repository, otherwise an external `Chart`; the edge keeps `repository`), and a `go.mod` / `requirements.txt` file to an external `Package`; a Dockerfile to the `Image`s it builds on (`via: from`, `copy-from`); an `Image` to the Dockerfile it is built from (`via: build`; `tool`: `skaffold`, `compose`, `docker-build` for `docker` / `podman` / `nerdctl` / `buildah` build commands in scripts, Make and CI); a compose service to the services its `depends_on` / `links` name
337-
- `CONFIGURES`: code reading an environment variable to the shared `EnvVar` of that name (`strategy: env_access`); a Kubernetes container, compose service or Dockerfile (`ENV` of the image) setting one to that container's own `EnvVar` (`strategy: env_set`, with `source`: `value`, `configMapKeyRef`, `secretKeyRef`, `fieldRef`, ..., `compose`, `dockerfile`; values are never stored)
338+
- `CONFIGURES`: code reading an environment variable to the shared `EnvVar` of that name (`strategy: env_access`); a Kubernetes container, compose service or Dockerfile (`ENV` of the image) setting one to that container's own `EnvVar` (`strategy: env_set`, with `source`: `value`, `configMapKeyRef`, `secretKeyRef`, `fieldRef`, ..., `compose`, `dockerfile`; values are never stored); and that container's `EnvVar` to the code reading the name inside the files its image is built from (`strategy: env_read`, with `dockerfile`): the image's files are the sources its Dockerfile's `COPY` / `ADD` put into it, less the context's `.dockerignore`
339+
- `OVERRIDE` with `via: patch`: a Kustomize patch to each object of its kustomization's build it changes (child → base): a patch file's documents to the object of the same kind and name; an inline patch (from the kustomization) or a JSON6902 operation file to what its `target` selects (group, version, kind, name, namespace, label selector); a component's in the build of each kustomization including it. The edge keeps `kustomization`, `patch` and `line`; a patch file is an `IMPORTS` (`local_name: patches`) of its kustomization, never a member of the build
340+
- `OVERRIDE` with `via: namePrefix` / `nameSuffix`: an object a kustomization renames, from its variant: a `Resource` named with the deployed name (`capi-` + `controller-manager`; the prefixes of nested kustomizations outermost first, suffixes innermost first) in the namespace that build gives it, `ecosystem: kustomize`, which the kustomization's file `DEFINES`. CRDs, Namespaces and API services keep their names, as kustomize does. The edge keeps `kustomization` and `line`
341+
- `USAGE` with `via: terraform`: a Terraform / OpenTofu reference (`var.X`, `local.X`, `module.X`, `data.T.N`, `T.N`) from the top-level block holding it to the block of its module (the `.tf` / `.tofu` files of its directory) that declares it (a `locals` entry: its `locals` block); an entry of the block's `depends_on` list is a `DEPENDS_ON` instead. The edge keeps `ref` (`var`, `local`, `module`, `data`, `resource`) and `line`; a name only mentioned in string text is no reference. A module call with a local source (`module "net" { source = "./modules/net" }`) `IMPORTS` (`local_name`: the module's name) the `Folder` of that directory (the root directory: the `Project`), and `module.net.id` (also `module.net[0].id`, `module.net[*].id`) `USAGE`s (`ref: output`) the `output "id"` there, beside `module.net`
342+
- `OVERRIDE` with `via: tfvars`: a `.tfvars` assignment to a variable the module of its own directory declares, from the file's `Variable` to that `variable` block (the edge keeps `line`); a `.tfvars` file whose own directory declares none of its names sets the nearest ancestor module's that does, the edge marked `inferred` (such a file is applied by a `-var-file` the repository does not state)
343+
- Terragrunt (`via: terragrunt`): a unit (`terragrunt.hcl`), evaluated in its own context: an `include` `IMPORTS` (`local_name`) the file its `path` names; a `read_terragrunt_config` `IMPORTS` (`read`: the file holding the call) the file it names, from the block holding the call or, for a call in an included file, from the unit's `include` block that brings it in; a `dependency` `DEPENDS_ON` the other unit; a local `terraform` `source` `IMPORTS` its `Folder`. Paths may use `find_in_parent_folders`, `dirname` and `get_terragrunt_dir`; anything else is unknown and links nothing
344+
- Ansible (`via: ansible`): a playbook or task file (a YAML file whose top level is a sequence) `IMPORTS` (`local_name`: the directive) what its directives name: `import_playbook` a playbook, `include_tasks` / `import_tasks` / `include` a task file (from its directory, else its role's `tasks/`), `roles:` items and `include_role` / `import_role` a role's `Folder` (`roles/<name>` at or above its directory). A templated target links nothing. Each task and handler is a `Function` (named by its `name:`, else its module; `source: ansible`, `kind`, `module`) that its file `DEFINES`: in a playbook's plays, a role's `tasks/` and `handlers/` files, and the files those include; a block's tasks, not the block. A `notify` `CALLS` (`via: notify`) each handler of that name or `listen` topic in the task's own role (else its file)
338345
- `EMITS`, `LISTENS_ON` (channels)
339346
- `DATA_FLOWS` with arg-to-param mapping + field access chains
340347
- `SIMILAR_TO` (MinHash + LSH near-clone detection, Jaccard scored)
@@ -343,7 +350,7 @@ The install script placed beside the binary is **reported, not deleted**: uninst
343350
### Indexing pipeline
344351
- **162 vendored tree-sitter grammars** compiled into the binary
345352
- **Generic package / module resolution**: bare specifiers like `@myorg/pkg`, `github.com/foo/bar`, `use my_crate::foo` resolved via manifest scanning (`package.json`, `go.mod`, `Cargo.toml`, `pyproject.toml`, `composer.json`, `pubspec.yaml`, `pom.xml`, `build.gradle`, `mix.exs`, `*.gemspec`)
346-
- **Infrastructure-as-code indexing**: Dockerfiles (`Dockerfile.*`, `*.Dockerfile`, `Containerfile`: bases, what the container runs, `ENV`), container images (`Image`: the normalised repository, tag and digest on the edges; a repository built from several Dockerfiles, one per component tag, gets an `Image` per component), docker compose services (`Resource`), Kubernetes manifests (`Resource`), Kustomize overlays (`IMPORTS` to their entries), Helm charts (`Chart`, `DEPENDS_ON`), Terraform / OpenTofu (`.tf`, `.tofu`, `.tfvars`) as graph nodes. A Service walks to the code it runs: Service → workload → `Image` → Dockerfile → `main`
353+
- **Infrastructure-as-code indexing**: Dockerfiles (`Dockerfile.*`, `*.Dockerfile`, `Containerfile`: bases, what the container runs, `ENV`), container images (`Image`: the normalised repository, tag and digest on the edges; a repository built from several Dockerfiles, one per component tag, gets an `Image` per component), docker compose services (`Resource`), Kubernetes manifests (`Resource`, with `group`, `version`, `kind`, `namespace`, `labels` as `k=v,k2=v2`), Kustomize overlays (`IMPORTS` to their entries and patch files, `OVERRIDE` from their patches), Helm charts (`Chart`, `DEPENDS_ON`), Terraform / OpenTofu (`.tf`, `.tofu`, `.tfvars`) as graph nodes. A Service walks to the code it runs: Service → workload → `Image` → Dockerfile → `main`
347354
- **[Hybrid LSP semantic type resolution](#hybrid-lsp)** for Python, TypeScript / JavaScript / JSX / TSX, PHP, C#, Go, C, C++, Java, Kotlin, Rust, and Perl; a lightweight C implementation of language type-resolution algorithms, structurally inspired by and compatible with major language servers including tsserver / typescript-go, pyright, gopls, Roslyn, Eclipse JDT, and rust-analyzer (parameter binding, return-type inference, generic substitution, JSX component dispatch, JSDoc inference for plain JS files, namespace + trait + late-static-binding resolution for PHP, file-scoped namespaces + records + LINQ method syntax for C#, class-hierarchy + overload + lambda resolution for Java, extension-function + scope-function resolution for Kotlin, trait-method + UFCS resolution for Rust)
348355
- **RAM-first pipeline**: LZ4 compression, in-memory SQLite, single dump at end. Memory released after.
349356

@@ -882,7 +889,7 @@ mode. `index_status` keeps describing the published graph and its freshness.
882889

883890
### Edge Types
884891

885-
`CONTAINS_FOLDER`, `CONTAINS_FILE`, `DEFINES`, `DEFINES_METHOD`, `IMPORTS`, `CALLS`, `CALL_REFERENCE`, `USAGE`, `INHERITS`, `IMPLEMENTS`, `OVERRIDE`, `DECORATES`, `RAISES`, `THROWS`, `WRITES`, `DATA_FLOWS`, `HTTP_CALLS`, `GRPC_CALLS`, `ASYNC_CALLS`, `EMITS`, `LISTENS_ON`, `HANDLES`, `SPAWNS`, `CONFIGURES`, `INFRA_MAPS`, `DEPENDS_ON`, `REFERENCES_FILE`, `MENTIONS`, `TESTS`, `TESTS_FILE`, `SIMILAR_TO`, `SEMANTICALLY_RELATED`, `FILE_CHANGES_WITH`, `HAS_BRANCH`, and the cross-repo `CROSS_HTTP_CALLS`, `CROSS_GRPC_CALLS`, `CROSS_GRAPHQL_CALLS`, `CROSS_TRPC_CALLS`, `CROSS_ASYNC_CALLS`, `CROSS_CHANNEL`
892+
`CONTAINS_FOLDER`, `CONTAINS_FILE`, `DEFINES`, `DEFINES_METHOD`, `IMPORTS`, `CALLS`, `CALL_REFERENCE`, `USAGE`, `INHERITS`, `IMPLEMENTS`, `OVERRIDE`, `DECORATES`, `RAISES`, `THROWS`, `WRITES`, `DATA_FLOWS`, `HTTP_CALLS`, `GRPC_CALLS`, `ASYNC_CALLS`, `EMITS`, `LISTENS_ON`, `HANDLES`, `SPAWNS`, `CONFIGURES`, `INFRA_MAPS`, `CONNECTS_TO`, `DEPLOYS`, `DEPENDS_ON`, `REFERENCES_FILE`, `MENTIONS`, `TESTS`, `TESTS_FILE`, `SIMILAR_TO`, `SEMANTICALLY_RELATED`, `FILE_CHANGES_WITH`, `HAS_BRANCH`, and the cross-repo `CROSS_HTTP_CALLS`, `CROSS_GRPC_CALLS`, `CROSS_GRAPHQL_CALLS`, `CROSS_TRPC_CALLS`, `CROSS_ASYNC_CALLS`, `CROSS_CHANNEL`
886893

887894
`MENTIONS` links a documented definition to the code its doc comment references (C# `<see cref>`, `<seealso cref>`, `<exception cref>`, `<inheritdoc cref>`), one edge per pair with `via`, `syntax`, `tier` (`exact` or `unique`), first `line` and `count`. A reference is bound only when it names exactly one definition under the language's own lookup rules; the rest are counted by reason in `index_status` under `doc_links` (samples with `diagnostics='full'`).
888895

‎internal/cbm/cbm.h‎

Lines changed: 27 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -701,31 +701,44 @@ typedef struct {
701701
/* A fact a Kubernetes manifest states about other objects (extract_k8s.c),
702702
* resolved by the K8s pass against the objects of the source's bundle.
703703
* CBM_K8S_FACT_REF: object def_index names a kind/ns/name object at a
704-
* reference path (via), drawn as edge (USAGE, INFRA_MAPS or DEPENDS_ON);
705-
* ns NULL = the source's own namespace.
704+
* reference path (via), drawn as edge (USAGE, INFRA_MAPS, DEPENDS_ON or
705+
* CONNECTS_TO); ns NULL = the source's own namespace. A CONNECTS_TO
706+
* reference (via "env") is the Service the value of environment variable
707+
* `key` of container `container` addresses -- the value is not kept.
706708
* CBM_K8S_FACT_CRD: the CRD def_index declares kind `kind` in group `name`.
707709
* CBM_K8S_FACT_DEPLOY: object def_index (a Flux Kustomization, an Argo
708710
* Application) deploys the repository directory `name` into namespace
709-
* `ns` (NULL: the objects' own); via "flux" / "argo".
711+
* `ns` (NULL: the objects' own); via "flux" / "argo". via "kustomize"
712+
* (def_index -1: the kustomization itself): a `helmCharts` entry
713+
* inflates chart `name` of repository `url` at `version` as release
714+
* `release` into namespace `ns`, with values file `path`.
710715
* CBM_K8S_FACT_ROUTE: object def_index (an Ingress, an HTTPRoute) exposes
711716
* `route` ("<host or *><path>") and sends it to the kind/ns/name backend
712717
* (name NULL: none); via "ingress" / "httproute".
713718
* CBM_K8S_FACT_SOURCE: object def_index (a Flux OCIRepository) fetches
714719
* from the URL `name`; a HelmRelease whose chartRef names it deploys the
715720
* chart there.
716-
* CBM_K8S_FACT_IMAGE: object def_index runs a container of image `name`
717-
* (a literal reference); via "container".
721+
* CBM_K8S_FACT_IMAGE: object def_index runs its container `container`
722+
* (NULL: unnamed) from image `name` (a literal reference); via "container".
718723
* CBM_K8S_FACT_ENV: object def_index sets environment variable `name` in
719724
* its container `container` (NULL: unnamed) at `line`; via = where the
720-
* value comes from ("value", or the valueFrom kind) -- never the value. */
725+
* value comes from ("value", or the valueFrom kind) -- never the value.
726+
* CBM_K8S_FACT_PATCH: a kustomization's `patches` entry at `line` patches
727+
* the objects of its build named by kind / group / version / name / ns
728+
* (any NULL: any) -- its `target`, else an inline patch document's own
729+
* kind / group / name / ns; `path` is
730+
* the patch file (as written, relative to the kustomization), NULL for an
731+
* inline patch. kind and name both NULL: the patch file's documents name
732+
* them. def_index -1: the kustomization itself is the source. */
721733
typedef enum {
722734
CBM_K8S_FACT_REF = 0,
723735
CBM_K8S_FACT_CRD,
724736
CBM_K8S_FACT_DEPLOY,
725737
CBM_K8S_FACT_ROUTE,
726738
CBM_K8S_FACT_SOURCE,
727739
CBM_K8S_FACT_IMAGE,
728-
CBM_K8S_FACT_ENV
740+
CBM_K8S_FACT_ENV,
741+
CBM_K8S_FACT_PATCH
729742
} CBMK8sFactKind;
730743
typedef struct {
731744
CBMK8sFactKind fact;
@@ -737,6 +750,13 @@ typedef struct {
737750
const char *edge;
738751
const char *route;
739752
const char *container;
753+
const char *key; /* CONNECTS_TO: the environment variable addressing it */
754+
const char *group; /* PATCH: the API group named ("core"), NULL: any */
755+
const char *version; /* PATCH: the API version named, NULL: any */
756+
const char *path; /* PATCH: the patch file, NULL: inline */
757+
const char *selector; /* PATCH: the target's labelSelector, NULL: none */
758+
const char *url; /* DEPLOY (kustomize): the chart repository */
759+
const char *release; /* DEPLOY (kustomize): the release name */
740760
int line;
741761
} CBMK8sFact;
742762

‎internal/cbm/extract_defs.c‎

Lines changed: 9 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -7192,18 +7192,19 @@ static char *find_hcl_block_name(CBMArena *a, TSNode node, const char *source) {
71927192
// Append the block's quoted labels so each block gets a distinct,
71937193
// query-friendly name: resource "aws_instance" "web" -> resource.aws_instance.web
71947194
// rather than every resource collapsing to the bare keyword "resource" (#337).
7195-
// HCL stores labels as string_lit -> template_literal children.
7195+
// HCL stores quoted labels as string_lit -> template_literal children and
7196+
// bare ones (`variable region {}`, valid HCL) as identifiers after the
7197+
// keyword's own -- both name the block.
71967198
uint32_t cc = ts_node_named_child_count(node);
71977199
for (uint32_t i = 0; i < cc; i++) {
71987200
TSNode ch = ts_node_named_child(node, i);
7199-
if (strcmp(ts_node_type(ch), "string_lit") != 0) {
7200-
continue;
7201-
}
7202-
TSNode lit = cbm_find_child_by_kind(ch, "template_literal");
7203-
if (ts_node_is_null(lit)) {
7204-
continue;
7201+
char *label = NULL;
7202+
if (strcmp(ts_node_type(ch), "string_lit") == 0) {
7203+
TSNode lit = cbm_find_child_by_kind(ch, "template_literal");
7204+
label = ts_node_is_null(lit) ? NULL : cbm_node_text(a, lit, source);
7205+
} else if (strcmp(ts_node_type(ch), "identifier") == 0 && !ts_node_eq(ch, id)) {
7206+
label = cbm_node_text(a, ch, source);
72057207
}
7206-
char *label = cbm_node_text(a, lit, source);
72077208
if (!label || !label[0]) {
72087209
continue;
72097210
}

0 commit comments

Comments
 (0)