Skip to content

Commit ce0ed0e

Browse files
committed
feat(infra): an Ansible playbook or task file IMPORTS what it includes
Ansible's composition produced no structure: kubespray's 40 playbooks, roles and task files, tied together by 217 import_playbook / include_tasks / import_tasks / include_role / import_role / roles: directives, were unconnected YAML files. A playbook or task file -- a YAML file whose top level is a sequence; manifests, values, CI and compose files are mappings -- now IMPORTS {local_name: <directive>, via: ansible, line} what each directive names: import_playbook a playbook from its directory; include_tasks / import_tasks / include a task file from its directory, else from its role's tasks/; roles: items and include_role / import_role a role's Folder, roles/<name> at or above its directory (a role by "./" / "../" path from the directory). The ansible.builtin. prefix is understood; a templated target links nothing. Measured on 17 infrastructure repositories: 211 of 211 composition pairs of kubespray against an independent reading, nothing else in any corpus; deterministic over three runs; the incremental re-index equals a full index in 4 scenarios (an included file edited or deleted, an include pointed elsewhere, a playbook edited). Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
1 parent fb73491 commit ce0ed0e

9 files changed

Lines changed: 513 additions & 4 deletions

File tree

‎Makefile.cbm‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -540,6 +540,7 @@ PIPELINE_SRCS = \
540540
src/pipeline/pass_operator.c \
541541
src/pipeline/pass_shell.c \
542542
src/pipeline/pass_terraform.c \
543+
src/pipeline/pass_ansible.c \
543544
src/pipeline/pass_docker.c \
544545
src/pipeline/pass_envjoin.c \
545546
src/pipeline/pass_ci.c \

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -259,6 +259,7 @@ The install script placed beside the binary is **reported, not deleted** — uni
259259
- `USAGE` with `via: terraform` — a Terraform / OpenTofu reference (`var.X`, `local.X`, `module.X`, `data.T.N`, `T.N`) from the top-level block holding it to the block of its module — the `.tf` / `.tofu` files of its directory — that declares it (a `locals` entry: its `locals` block); an entry of the block's `depends_on` list is a `DEPENDS_ON` instead. The edge keeps `ref` (`var`, `local`, `module`, `data`, `resource`) and `line`; a name only mentioned in string text is no reference. A module call with a local source (`module "net" { source = "./modules/net" }`) `IMPORTS` (`local_name`: the module's name) the `Folder` of that directory (the root directory: the `Project`), and `module.net.id` (also `module.net[0].id`, `module.net[*].id`) `USAGE`s (`ref: output`) the `output "id"` there, beside `module.net`
260260
- `OVERRIDE` with `via: tfvars` — a `.tfvars` assignment to a variable the module of its own directory declares, from the file's `Variable` to that `variable` block (the edge keeps `line`); a `.tfvars` file elsewhere is applied by a `-var-file` the repository does not state and links nothing
261261
- Terragrunt (`via: terragrunt`) — a unit (`terragrunt.hcl`), evaluated in its own context: an `include` `IMPORTS` (`local_name`) the file its `path` names; a `read_terragrunt_config` `IMPORTS` (`read`: the file holding the call) the file it names, from the block holding the call or, for a call in an included file, from the unit's `include` block that brings it in; a `dependency` `DEPENDS_ON` the other unit; a local `terraform` `source` `IMPORTS` its `Folder`. Paths may use `find_in_parent_folders`, `dirname` and `get_terragrunt_dir`; anything else is unknown and links nothing
262+
- Ansible (`via: ansible`) — a playbook or task file (a YAML file whose top level is a sequence) `IMPORTS` (`local_name`: the directive) what its directives name: `import_playbook` a playbook, `include_tasks` / `import_tasks` / `include` a task file (from its directory, else its role's `tasks/`), `roles:` items and `include_role` / `import_role` a role's `Folder` (`roles/<name>` at or above its directory). A templated target links nothing
262263
- `EMITS`, `LISTENS_ON` (channels)
263264
- `DATA_FLOWS` with arg-to-param mapping + field access chains
264265
- `SIMILAR_TO` (MinHash + LSH near-clone detection, Jaccard scored)

‎scripts/check-lsp-originality.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -82,7 +82,7 @@ REFS=(
8282

8383
# Infrastructure hybrid LSP sources outside internal/cbm/lsp/ (scanned too).
8484
INFRA_SRC=(internal/cbm/extract_k8s.c src/pipeline/pass_k8s.c src/pipeline/pass_operator.c
85-
src/pipeline/pass_shell.c src/pipeline/pass_terraform.c
85+
src/pipeline/pass_shell.c src/pipeline/pass_terraform.c src/pipeline/pass_ansible.c
8686
src/pipeline/pass_ci.c src/pipeline/pass_docker.c src/pipeline/pass_envjoin.c)
8787

8888
ONLY_LANG=""; REFRESH=0; LIST_ONLY=0

‎src/pipeline/pass_ansible.c‎

Lines changed: 372 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,372 @@
1+
/*
2+
* pass_ansible.c — the Ansible layer.
3+
*
4+
* PHASE 7 P7.1: composition. A playbook or a task file -- a YAML file whose
5+
* top level is a sequence (manifests, values, CI and compose files are
6+
* mappings) -- IMPORTS {local_name: <directive>, via: ansible, line} what its
7+
* directives name:
8+
* import_playbook: P the playbook P, from the file's
9+
* directory
10+
* include_tasks / import_tasks /
11+
* include: T (or {file: T}) the task file T, from the file's
12+
* directory, else from the tasks/
13+
* directory of the role holding it
14+
* include_role / import_role: {name: R},
15+
* roles: [R | {role: R} | {name: R}] the role R: roles/R in the file's
16+
* directory or above (a name may hold
17+
* "/", roles/kubernetes/node), or a
18+
* path ("./", "../") from the
19+
* file's directory
20+
* (the ansible.builtin. prefix allowed). The edge goes from the file to the
21+
* target file's Module, or to the role's Folder; a target the graph does not
22+
* hold, or one holding a Jinja expression ({{ }}), makes none.
23+
*/
24+
#include "foundation/constants.h"
25+
#include "foundation/compat.h"
26+
#include "foundation/compat_fs.h"
27+
#include "foundation/hash_table.h"
28+
#include "foundation/limits.h"
29+
#include "foundation/log.h"
30+
#include "foundation/mem_core.h"
31+
#include "graph_buffer/graph_buffer.h"
32+
#include "pipeline/pipeline.h"
33+
#include "pipeline/pipeline_internal.h"
34+
#include "cbm.h"
35+
#include "helpers.h"
36+
#include "lang_specs.h"
37+
38+
#include <stdint.h>
39+
#include <stdio.h>
40+
#include <stdlib.h>
41+
#include <string.h>
42+
43+
typedef struct {
44+
cbm_pipeline_ctx_t *ctx;
45+
CBMHashTable *modules; /* file path -> its Module node */
46+
CBMHashTable *folders; /* directory -> its Folder node */
47+
const char *src; /* the file's text */
48+
const char *file; /* its repository path */
49+
char dir[CBM_SZ_1K]; /* its directory */
50+
char role[CBM_SZ_1K]; /* the role holding it (roles/<name>), or "" */
51+
int64_t from; /* its File node */
52+
int edges;
53+
} an_t;
54+
55+
static bool an_type(TSNode n, const char *t) {
56+
return !ts_node_is_null(n) && strcmp(ts_node_type(n), t) == 0;
57+
}
58+
59+
/* A scalar's text (quotes stripped) into out; false for anything else. */
60+
static bool an_scalar(const an_t *a, TSNode n, char *out, size_t sz) {
61+
while ((an_type(n, "flow_node") || an_type(n, "block_node") || an_type(n, "plain_scalar")) &&
62+
ts_node_named_child_count(n) == SKIP_ONE) {
63+
n = ts_node_named_child(n, 0);
64+
}
65+
if (ts_node_is_null(n)) { /* a pair without its key or value */
66+
return false;
67+
}
68+
uint32_t b = ts_node_start_byte(n);
69+
uint32_t e = ts_node_end_byte(n);
70+
if (an_type(n, "double_quote_scalar") || an_type(n, "single_quote_scalar")) {
71+
b++;
72+
e = e > b ? e - SKIP_ONE : b;
73+
} else if (!an_type(n, "string_scalar") && !an_type(n, "plain_scalar")) {
74+
return false;
75+
}
76+
if (e - b + SKIP_ONE > sz) {
77+
return false;
78+
}
79+
memcpy(out, a->src + b, e - b);
80+
out[e - b] = '\0';
81+
return out[0] != '\0';
82+
}
83+
84+
/* The key of a mapping pair, its text, the ansible.builtin. prefix dropped. */
85+
static bool an_key(const an_t *a, TSNode pair, char *out, size_t sz) {
86+
static const char BUILTIN[] = "ansible.builtin.";
87+
if (!an_scalar(a, ts_node_child_by_field_name(pair, "key", sizeof("key") - SKIP_ONE), out,
88+
sz)) {
89+
return false;
90+
}
91+
size_t bl = sizeof(BUILTIN) - SKIP_ONE;
92+
if (strncmp(out, BUILTIN, bl) == 0) {
93+
memmove(out, out + bl, strlen(out + bl) + SKIP_ONE);
94+
}
95+
return true;
96+
}
97+
98+
/* The value of `key` in a block mapping (under its block_node), or a null node. */
99+
static TSNode an_get(const an_t *a, TSNode value, const char *key) {
100+
while (an_type(value, "block_node") && ts_node_named_child_count(value) == SKIP_ONE) {
101+
value = ts_node_named_child(value, 0);
102+
}
103+
for (uint32_t i = 0; an_type(value, "block_mapping") && i < ts_node_named_child_count(value);
104+
i++) {
105+
TSNode pair = ts_node_named_child(value, i);
106+
char k[CBM_SZ_64];
107+
if (an_type(pair, "block_mapping_pair") && an_key(a, pair, k, sizeof(k)) &&
108+
strcmp(k, key) == 0) {
109+
return ts_node_child_by_field_name(pair, "value", sizeof("value") - SKIP_ONE);
110+
}
111+
}
112+
return (TSNode){0};
113+
}
114+
115+
static void an_edge(an_t *a, const cbm_gbuf_node_t *tgt, const char *kind, TSNode pair) {
116+
if (!tgt || a->from <= 0 || tgt->id == a->from) {
117+
return;
118+
}
119+
char props[CBM_SZ_128];
120+
snprintf(props, sizeof(props), "{\"local_name\":\"%s\",\"via\":\"ansible\",\"line\":%d}", kind,
121+
(int)ts_node_start_point(pair).row + SKIP_ONE);
122+
cbm_gbuf_insert_edge(a->ctx->gbuf, a->from, tgt->id, "IMPORTS", props);
123+
a->edges++;
124+
}
125+
126+
/* A file `v` from `base` (repository-relative), its Module; NULL when none. */
127+
static const cbm_gbuf_node_t *an_file(const an_t *a, const char *base, const char *v) {
128+
char path[CBM_SZ_1K];
129+
if (!cbm_repo_rel_join(base, v, path, sizeof(path))) {
130+
return NULL;
131+
}
132+
return cbm_ht_get(a->modules, path);
133+
}
134+
135+
/* Role `name`'s Folder: a path ("./", "../") from the file's directory, else
136+
* roles/<name> in that directory or above. */
137+
static const cbm_gbuf_node_t *an_role(const an_t *a, const char *name) {
138+
char path[CBM_SZ_1K];
139+
if (name[0] == '.' || name[0] == '/') {
140+
return cbm_repo_rel_join(a->dir, name, path, sizeof(path)) ? cbm_ht_get(a->folders, path)
141+
: NULL;
142+
}
143+
char d[CBM_SZ_1K];
144+
snprintf(d, sizeof(d), "%s", a->dir);
145+
for (;;) {
146+
snprintf(path, sizeof(path), "%s%sroles/%s", d, d[0] ? "/" : "", name);
147+
const cbm_gbuf_node_t *f = cbm_ht_get(a->folders, path);
148+
if (f || !d[0]) {
149+
return f;
150+
}
151+
char *s = strrchr(d, '/');
152+
if (s) {
153+
*s = '\0';
154+
} else {
155+
d[0] = '\0';
156+
}
157+
}
158+
}
159+
160+
/* One mapping pair of a task or a play. */
161+
static void an_pair(an_t *a, TSNode pair) {
162+
char key[CBM_SZ_64];
163+
char v[CBM_SZ_1K];
164+
if (!an_key(a, pair, key, sizeof(key))) {
165+
return;
166+
}
167+
TSNode value = ts_node_child_by_field_name(pair, "value", sizeof("value") - SKIP_ONE);
168+
if (strcmp(key, "import_playbook") == 0) {
169+
if (an_scalar(a, value, v, sizeof(v)) && !strstr(v, "{{")) {
170+
an_edge(a, an_file(a, a->dir, v), key, pair);
171+
}
172+
} else if (strcmp(key, "include_tasks") == 0 || strcmp(key, "import_tasks") == 0 ||
173+
strcmp(key, "include") == 0) {
174+
if ((an_scalar(a, value, v, sizeof(v)) ||
175+
an_scalar(a, an_get(a, value, "file"), v, sizeof(v))) &&
176+
!strstr(v, "{{")) {
177+
char tasks[CBM_SZ_1K];
178+
snprintf(tasks, sizeof(tasks), "%s/tasks", a->role);
179+
const cbm_gbuf_node_t *t = an_file(a, a->dir, v);
180+
an_edge(a, t ? t : a->role[0] ? an_file(a, tasks, v) : NULL, key, pair);
181+
}
182+
} else if (strcmp(key, "include_role") == 0 || strcmp(key, "import_role") == 0) {
183+
if (an_scalar(a, an_get(a, value, "name"), v, sizeof(v)) && !strstr(v, "{{")) {
184+
an_edge(a, an_role(a, v), key, pair);
185+
}
186+
} else if (strcmp(key, "roles") == 0) {
187+
while (an_type(value, "block_node") && ts_node_named_child_count(value) == SKIP_ONE) {
188+
value = ts_node_named_child(value, 0);
189+
}
190+
for (uint32_t i = 0;
191+
an_type(value, "block_sequence") && i < ts_node_named_child_count(value); i++) {
192+
TSNode item = ts_node_named_child(ts_node_named_child(value, i), 0);
193+
if ((an_scalar(a, item, v, sizeof(v)) ||
194+
an_scalar(a, an_get(a, item, "role"), v, sizeof(v)) ||
195+
an_scalar(a, an_get(a, item, "name"), v, sizeof(v))) &&
196+
!strstr(v, "{{")) {
197+
an_edge(a, an_role(a, v), key, ts_node_named_child(value, i));
198+
}
199+
}
200+
}
201+
}
202+
203+
/* Every mapping that is a sequence item (a play or a task, also one nested in
204+
* block: / tasks: / handlers: ...): its pairs. */
205+
static void an_walk(an_t *a, TSNode root) {
206+
TSTreeCursor cur = ts_tree_cursor_new(root);
207+
bool more = true;
208+
while (more) {
209+
TSNode c = ts_tree_cursor_current_node(&cur);
210+
if (an_type(c, "block_mapping")) {
211+
TSNode up = ts_node_parent(ts_node_parent(c));
212+
for (uint32_t i = 0;
213+
an_type(up, "block_sequence_item") && i < ts_node_named_child_count(c); i++) {
214+
if (an_type(ts_node_named_child(c, i), "block_mapping_pair")) {
215+
an_pair(a, ts_node_named_child(c, i));
216+
}
217+
}
218+
}
219+
if (ts_tree_cursor_goto_first_child(&cur)) {
220+
continue;
221+
}
222+
while (!ts_tree_cursor_goto_next_sibling(&cur)) {
223+
if (!ts_tree_cursor_goto_parent(&cur)) {
224+
more = false;
225+
break;
226+
}
227+
}
228+
}
229+
ts_tree_cursor_delete(&cur);
230+
}
231+
232+
/* The first content line of `src` starts a sequence item: Ansible content
233+
* (a playbook, a task list), not a mapping (manifest, values, CI, compose). */
234+
static bool an_is_sequence(const char *src) {
235+
for (const char *p = src; *p;) {
236+
const char *nl = strchr(p, '\n');
237+
size_t n = nl ? (size_t)(nl - p) : strlen(p);
238+
size_t s = 0;
239+
while (s < n && (p[s] == ' ' || p[s] == '\t' || p[s] == '\r')) {
240+
s++;
241+
}
242+
bool skip = s == n || p[s] == '#' ||
243+
(n - s >= sizeof("---") - SKIP_ONE &&
244+
strncmp(p + s, "---", sizeof("---") - SKIP_ONE) == 0);
245+
if (!skip) {
246+
return s == 0 && n >= PAIR_LEN && p[0] == '-' && p[SKIP_ONE] == ' ';
247+
}
248+
p = nl ? nl + SKIP_ONE : p + n;
249+
}
250+
return false;
251+
}
252+
253+
/* The role holding `rel`: roles/<name> of the deepest "roles" component with
254+
* a name after it, or "". */
255+
static void an_role_of(const char *rel, char *out, size_t sz) {
256+
out[0] = '\0';
257+
for (const char *p = rel; (p = strstr(p, "roles/")) != NULL; p++) {
258+
if (p != rel && p[-1] != '/') {
259+
continue;
260+
}
261+
const char *name = p + sizeof("roles/") - SKIP_ONE;
262+
const char *end = strchr(name, '/');
263+
if (end && strchr(end + SKIP_ONE, '/')) {
264+
snprintf(out, sz, "%.*s", (int)(end - rel), rel);
265+
}
266+
}
267+
}
268+
269+
static char *an_read(const char *path, size_t *len) {
270+
FILE *f = cbm_fopen(path, "rb");
271+
if (!f) {
272+
return NULL;
273+
}
274+
(void)fseek(f, 0, SEEK_END);
275+
long size = ftell(f);
276+
(void)fseek(f, 0, SEEK_SET);
277+
if (size <= 0 || size > cbm_max_file_bytes()) {
278+
(void)fclose(f);
279+
return NULL;
280+
}
281+
enum { AN_TS_PAD = 16 }; /* tree-sitter's lexer reads past the end */
282+
char *buf = cbm_alloc(CBM_MEM_CLASS_OTHER, (size_t)size + AN_TS_PAD);
283+
if (!buf) {
284+
(void)fclose(f);
285+
return NULL;
286+
}
287+
size_t got = fread(buf, SKIP_ONE, (size_t)size, f);
288+
(void)fclose(f);
289+
memset(buf + got, 0, AN_TS_PAD);
290+
*len = got;
291+
return buf;
292+
}
293+
294+
/* path -> node index of one label's nodes. */
295+
static CBMHashTable *an_index(cbm_pipeline_ctx_t *ctx, const char *label) {
296+
CBMHashTable *ix = cbm_ht_create(0);
297+
const cbm_gbuf_node_t **nodes = NULL;
298+
int n = 0;
299+
if (ix && cbm_gbuf_find_by_label(ctx->gbuf, label, &nodes, &n) == 0) {
300+
for (int i = 0; i < n; i++) {
301+
if (nodes[i]->file_path) {
302+
cbm_ht_set(ix, nodes[i]->file_path, (void *)nodes[i]);
303+
}
304+
}
305+
}
306+
return ix;
307+
}
308+
309+
static bool an_is_yaml(const char *rel) {
310+
const char *dot = strrchr(rel, '.');
311+
return dot && (strcmp(dot, ".yml") == 0 || strcmp(dot, ".yaml") == 0);
312+
}
313+
314+
void cbm_pipeline_pass_ansible(cbm_pipeline_ctx_t *ctx, const cbm_file_info_t *files,
315+
int file_count) {
316+
an_t a = {.ctx = ctx};
317+
TSParser *parser = NULL;
318+
CBMHashTable *file_nodes = NULL;
319+
for (int i = 0; i < file_count; i++) {
320+
if (!an_is_yaml(files[i].rel_path)) {
321+
continue;
322+
}
323+
size_t len = 0;
324+
char *src = an_read(files[i].path, &len);
325+
if (!src || !an_is_sequence(src)) {
326+
cbm_free(CBM_MEM_CLASS_OTHER, src);
327+
continue;
328+
}
329+
if (!parser) {
330+
parser = ts_parser_new();
331+
if (!parser || !ts_parser_set_language(parser, cbm_ts_language(CBM_LANG_YAML))) {
332+
cbm_free(CBM_MEM_CLASS_OTHER, src);
333+
break;
334+
}
335+
a.modules = an_index(ctx, "Module");
336+
a.folders = an_index(ctx, "Folder");
337+
file_nodes = an_index(ctx, "File");
338+
}
339+
const cbm_gbuf_node_t *fnode =
340+
file_nodes ? cbm_ht_get(file_nodes, files[i].rel_path) : NULL;
341+
TSTree *tree = fnode ? ts_parser_parse_string(parser, NULL, src, (uint32_t)len) : NULL;
342+
if (tree) {
343+
const char *rel = files[i].rel_path;
344+
const char *slash = strrchr(rel, '/');
345+
a.src = src;
346+
a.file = rel;
347+
a.from = fnode->id;
348+
snprintf(a.dir, sizeof(a.dir), "%.*s", slash ? (int)(slash - rel) : 0, rel);
349+
an_role_of(rel, a.role, sizeof(a.role));
350+
an_walk(&a, ts_tree_root_node(tree));
351+
ts_tree_delete(tree);
352+
}
353+
cbm_free(CBM_MEM_CLASS_OTHER, src);
354+
}
355+
if (parser) {
356+
ts_parser_delete(parser);
357+
}
358+
if (a.modules) {
359+
cbm_ht_free(a.modules);
360+
}
361+
if (a.folders) {
362+
cbm_ht_free(a.folders);
363+
}
364+
if (file_nodes) {
365+
cbm_ht_free(file_nodes);
366+
}
367+
if (a.edges > 0) {
368+
char buf[CBM_SZ_32];
369+
snprintf(buf, sizeof(buf), "%d", a.edges);
370+
cbm_log_info("pass.ansible.edges", "imports", buf);
371+
}
372+
}

0 commit comments

Comments
 (0)