Skip to content

Commit cdfdb8f

Browse files
committed
feat(infra): a .tfvars file sets an ancestor module's variables, inferred
A .tfvars file in a directory without Terraform -- kubespray's sample-inventory/cluster.tfvars, eks-blueprints' spokes/workspaces/*.tfvars -- is applied to a module further up with a -var-file the repository does not state, and linked nothing. When the file's own directory declares none of its names, it now sets the variables of the nearest ancestor module that does: each assignment to a declared variable OVERRIDEs {via: tfvars, inferred: true, line} the `variable` block, from the file's Variable. A file whose own directory declares one of its names keeps the exact edge, without `inferred`. Measured on 17 infrastructure repositories: 238 of 238 assignments -- 124 to their own directory's module, 114 inferred to an ancestor's -- against an independent reading, with the inferred flag; nothing else changed; deterministic over three runs; the incremental re-index equals a full index when an ancestor-linked file changes. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
1 parent ce0ed0e commit cdfdb8f

4 files changed

Lines changed: 84 additions & 38 deletions

File tree

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -257,7 +257,7 @@ The install script placed beside the binary is **reported, not deleted** — uni
257257
- `CONFIGURES` — code reading an environment variable to the shared `EnvVar` of that name (`strategy: env_access`); a Kubernetes container, compose service or Dockerfile (`ENV` of the image) setting one to that container's own `EnvVar` (`strategy: env_set`, with `source`: `value`, `configMapKeyRef`, `secretKeyRef`, `fieldRef`, ..., `compose`, `dockerfile` — values are never stored); and that container's `EnvVar` to the code reading the name inside the files its image is built from (`strategy: env_read`, with `dockerfile`) — the image's files are the sources its Dockerfile's `COPY` / `ADD` put into it, less the context's `.dockerignore`
258258
- `OVERRIDE` with `via: patch` — a Kustomize patch to each object of its kustomization's build it changes (child → base): a patch file's documents to the object of the same kind and name; an inline patch (from the kustomization) or a JSON6902 operation file to what its `target` selects (group, version, kind, name, namespace, label selector); a component's in the build of each kustomization including it. The edge keeps `kustomization`, `patch` and `line`; a patch file is an `IMPORTS` (`local_name: patches`) of its kustomization, never a member of the build
259259
- `USAGE` with `via: terraform` — a Terraform / OpenTofu reference (`var.X`, `local.X`, `module.X`, `data.T.N`, `T.N`) from the top-level block holding it to the block of its module — the `.tf` / `.tofu` files of its directory — that declares it (a `locals` entry: its `locals` block); an entry of the block's `depends_on` list is a `DEPENDS_ON` instead. The edge keeps `ref` (`var`, `local`, `module`, `data`, `resource`) and `line`; a name only mentioned in string text is no reference. A module call with a local source (`module "net" { source = "./modules/net" }`) `IMPORTS` (`local_name`: the module's name) the `Folder` of that directory (the root directory: the `Project`), and `module.net.id` (also `module.net[0].id`, `module.net[*].id`) `USAGE`s (`ref: output`) the `output "id"` there, beside `module.net`
260-
- `OVERRIDE` with `via: tfvars` — a `.tfvars` assignment to a variable the module of its own directory declares, from the file's `Variable` to that `variable` block (the edge keeps `line`); a `.tfvars` file elsewhere is applied by a `-var-file` the repository does not state and links nothing
260+
- `OVERRIDE` with `via: tfvars` — a `.tfvars` assignment to a variable the module of its own directory declares, from the file's `Variable` to that `variable` block (the edge keeps `line`); a `.tfvars` file whose own directory declares none of its names sets the nearest ancestor module's that does, the edge marked `inferred` (such a file is applied by a `-var-file` the repository does not state)
261261
- Terragrunt (`via: terragrunt`) — a unit (`terragrunt.hcl`), evaluated in its own context: an `include` `IMPORTS` (`local_name`) the file its `path` names; a `read_terragrunt_config` `IMPORTS` (`read`: the file holding the call) the file it names, from the block holding the call or, for a call in an included file, from the unit's `include` block that brings it in; a `dependency` `DEPENDS_ON` the other unit; a local `terraform` `source` `IMPORTS` its `Folder`. Paths may use `find_in_parent_folders`, `dirname` and `get_terragrunt_dir`; anything else is unknown and links nothing
262262
- Ansible (`via: ansible`) — a playbook or task file (a YAML file whose top level is a sequence) `IMPORTS` (`local_name`: the directive) what its directives name: `import_playbook` a playbook, `include_tasks` / `import_tasks` / `include` a task file (from its directory, else its role's `tasks/`), `roles:` items and `include_role` / `import_role` a role's `Folder` (`roles/<name>` at or above its directory). A templated target links nothing
263263
- `EMITS`, `LISTENS_ON` (channels)

‎src/pipeline/pass_terraform.c‎

Lines changed: 62 additions & 27 deletions
Original file line numberDiff line numberDiff line change
@@ -432,10 +432,11 @@ static TSNode tf_root_body(TSNode root) {
432432
* module.X.Y (also module.X[i].Y, module.X[*].Y) USAGEs {ref: output}, beside
433433
* module.X, the `output "Y"` of it.
434434
*
435-
* A .tfvars file sets variables of the module in its own directory: each
436-
* assignment to a declared variable OVERRIDEs {via: tfvars, line} its
437-
* `variable` block (a file elsewhere is applied by a -var-file on some
438-
* command line, which is nothing the repository's Terraform states). */
435+
* A .tfvars file sets variables of the module in its own directory -- or,
436+
* when that declares none of its names, of the nearest ancestor directory's
437+
* module that does (applied by a -var-file the repository does not state:
438+
* the edge says `inferred`): each assignment to a declared variable
439+
* OVERRIDEs {via: tfvars, line} its `variable` block. */
439440

440441
typedef struct {
441442
char *file; /* the referencing file */
@@ -876,32 +877,70 @@ static CBMHashTable *tf_folder_index(cbm_pipeline_ctx_t *ctx) {
876877
return ix;
877878
}
878879

879-
/* A .tfvars file sets variables of the module in its own directory: each
880-
* top-level assignment whose name the module declares OVERRIDEs
881-
* {via: tfvars, line}, from the file's Variable of that name, the
882-
* `variable` block. */
883-
static int tf_link_tfvars(tf_mod_t *m, TSParser *parser, const char *rel) {
880+
/* The module of the first of `d` and its ancestors (d is cut in place)
881+
* declaring one of `vnames[0..n)`; NULL when none does. */
882+
static tf_mod_t *tf_tfvars_module(CBMHashTable *mods, char *d, const char *const *vnames, int n) {
883+
for (;;) {
884+
tf_mod_t *m = cbm_ht_get(mods, d);
885+
for (int k = 0; m && k < n; k++) {
886+
if (vnames[k] && cbm_ht_get(m->decls, vnames[k])) {
887+
return m;
888+
}
889+
}
890+
if (!d[0]) {
891+
return NULL;
892+
}
893+
char *s = strrchr(d, '/');
894+
if (s) {
895+
*s = '\0';
896+
} else {
897+
d[0] = '\0';
898+
}
899+
}
900+
}
901+
902+
/* A .tfvars file sets variables of a module: the one in its own directory
903+
* when that declares one of its names, else the nearest ancestor directory's
904+
* that does (a file there is applied by a -var-file the repository does not
905+
* state: the edge says `inferred`). Each top-level assignment whose name the
906+
* module declares OVERRIDEs {via: tfvars, line}, from the file's Variable of
907+
* that name, the `variable` block. */
908+
static int tf_link_tfvars(CBMArena *arena, TSParser *parser, CBMHashTable *mods,
909+
const char *repo_path, const char *rel) {
884910
char path[CBM_SZ_2K];
885-
snprintf(path, sizeof(path), "%s/%s", m->ctx->repo_path, rel);
911+
snprintf(path, sizeof(path), "%s/%s", repo_path, rel);
886912
size_t len = 0;
887913
char *src = tf_read(path, &len);
888914
TSTree *tree = src ? ts_parser_parse_string(parser, NULL, src, (uint32_t)len) : NULL;
889915
TSNode body = tree ? tf_root_body(ts_tree_root_node(tree)) : (TSNode){0};
890-
int edges = 0;
891-
for (uint32_t i = 0; !ts_node_is_null(body) && i < ts_node_named_child_count(body); i++) {
916+
uint32_t nc = ts_node_is_null(body) ? 0 : ts_node_named_child_count(body);
917+
const char **vnames = nc ? cbm_arena_alloc(arena, (size_t)nc * sizeof(*vnames)) : NULL;
918+
TSNode *attrs = nc ? cbm_arena_alloc(arena, (size_t)nc * sizeof(*attrs)) : NULL;
919+
int n = 0;
920+
for (uint32_t i = 0; vnames && attrs && i < nc; i++) {
892921
TSNode at = ts_node_named_child(body, i);
893-
if (!tf_type(at, "attribute") || ts_node_named_child_count(at) == 0) {
894-
continue;
922+
if (tf_type(at, "attribute") && ts_node_named_child_count(at) > 0) {
923+
const char *key = tf_text(arena, src, ts_node_named_child(at, 0));
924+
vnames[n] = key ? cbm_arena_sprintf(arena, "variable.%s", key) : NULL;
925+
attrs[n++] = at;
895926
}
896-
const char *key = tf_text(m->arena, src, ts_node_named_child(at, 0));
897-
const char *vname = key ? cbm_arena_sprintf(m->arena, "variable.%s", key) : NULL;
898-
const char *vfile = vname ? cbm_ht_get(m->decls, vname) : NULL;
899-
int64_t from = vfile ? tf_def_node(m, rel, key) : 0;
900-
int64_t to = from > 0 ? tf_def_node(m, vfile, vname) : 0;
927+
}
928+
const char *slash = strrchr(rel, '/');
929+
char d[CBM_SZ_1K];
930+
snprintf(d, sizeof(d), "%.*s", slash ? (int)(slash - rel) : 0, rel);
931+
size_t own = strlen(d);
932+
tf_mod_t *m = n > 0 ? tf_tfvars_module(mods, d, vnames, n) : NULL;
933+
int edges = 0;
934+
for (int k = 0; m && k < n; k++) {
935+
const char *vfile = vnames[k] ? cbm_ht_get(m->decls, vnames[k]) : NULL;
936+
const char *key = vfile ? vnames[k] + sizeof("variable.") - SKIP_ONE : NULL;
937+
int64_t from = key ? tf_def_node(m, rel, key) : 0;
938+
int64_t to = from > 0 ? tf_def_node(m, vfile, vnames[k]) : 0;
901939
if (to > 0) {
902940
char props[CBM_SZ_128];
903-
snprintf(props, sizeof(props), "{\"via\":\"tfvars\",\"line\":%d}",
904-
(int)ts_node_start_point(at).row + SKIP_ONE);
941+
snprintf(props, sizeof(props), "{\"via\":\"tfvars\",%s\"line\":%d}",
942+
strlen(d) == own ? "" : "\"inferred\":true,",
943+
(int)ts_node_start_point(attrs[k]).row + SKIP_ONE);
905944
cbm_gbuf_insert_edge(m->ctx->gbuf, from, to, "OVERRIDE", props);
906945
edges++;
907946
}
@@ -915,7 +954,7 @@ static int tf_link_tfvars(tf_mod_t *m, TSParser *parser, const char *rel) {
915954

916955
/* Every module directory, `paths[0..pn)` sorted by directory: parse all,
917956
* then link each -- a reference may read an output of another directory --
918-
* and the .tfvars files `vpaths[0..vn)` to the module of their directory. */
957+
* and the .tfvars files `vpaths[0..vn)` to the module whose variables they set. */
919958
static int tf_link_repo(cbm_pipeline_ctx_t *ctx, TSParser *parser, const char *const *paths, int pn,
920959
const char *const *vpaths, int vn) {
921960
CBMArena arena;
@@ -943,11 +982,7 @@ static int tf_link_repo(cbm_pipeline_ctx_t *ctx, TSParser *parser, const char *c
943982
edges += tf_mod_link(list[k], mods, folders);
944983
}
945984
for (int k = 0; mods && k < vn; k++) {
946-
const char *slash = strrchr(vpaths[k], '/');
947-
char dir[CBM_SZ_1K];
948-
snprintf(dir, sizeof(dir), "%.*s", slash ? (int)(slash - vpaths[k]) : 0, vpaths[k]);
949-
tf_mod_t *m = cbm_ht_get(mods, dir);
950-
edges += m ? tf_link_tfvars(m, parser, vpaths[k]) : 0;
985+
edges += tf_link_tfvars(&arena, parser, mods, ctx->repo_path, vpaths[k]);
951986
}
952987
for (int k = 0; k < nm; k++) {
953988
tf_mod_free(list[k]);

‎src/pipeline/pipeline_internal.h‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1517,7 +1517,8 @@ int cbm_pipeline_build_fresh_semantic_manifest(cbm_pipeline_t *p, const char *pr
15171517
* its block like a quoted one; a module call with a local source
15181518
* IMPORTS its directory's Folder (the root: the Project) and module.X.Y
15191519
* USAGEs {ref:output} the output there; a .tfvars assignment OVERRIDEs
1520-
* {via:tfvars} the variable block of its own directory; a Terragrunt
1520+
* {via:tfvars} the variable block of its own directory (or, inferred,
1521+
* of the nearest ancestor module declaring its names); a Terragrunt
15211522
* unit's include / read_terragrunt_config IMPORTS {via:terragrunt} the
15221523
* file it names, a dependency DEPENDS_ON the other unit
15231524
* (pass_terraform.c, extract_unified.c, extract_defs.c); an Ansible

‎tests/test_grammar_probe_f.c‎

Lines changed: 19 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -2974,7 +2974,8 @@ TEST(probe_terraform_module_calls_import_their_source) {
29742974
PASS();
29752975
}
29762976

2977-
/* Every tfvars OVERRIDE edge as "<file> <name> -> <file> <block> <line>". */
2977+
/* Every tfvars OVERRIDE edge as "<file> <name> -> <file> <block> <line>",
2978+
* " inferred" appended when the edge says so. */
29782979
static char *gpf_tf_tfvars(const GpfFile *files, int nfiles) {
29792980
GpfProj lp;
29802981
cbm_store_t *store = gpf_index_files(&lp, files, nfiles);
@@ -2984,9 +2985,11 @@ static char *gpf_tf_tfvars(const GpfFile *files, int nfiles) {
29842985
const char *sql =
29852986
"SELECT coalesce((SELECT group_concat(x, char(10)) FROM (SELECT s.file_path || ' ' || "
29862987
"s.name || ' -> ' || t.file_path || ' ' || t.name || ' ' || "
2987-
"json_extract(e.properties,'$.line') AS x FROM edges e JOIN nodes s ON "
2988-
"s.id=e.source_id JOIN nodes t ON t.id=e.target_id WHERE e.project=?1 AND "
2989-
"e.type='OVERRIDE' AND json_extract(e.properties,'$.via')='tfvars' ORDER BY x)), '')";
2988+
"json_extract(e.properties,'$.line') || CASE WHEN "
2989+
"json_extract(e.properties,'$.inferred') THEN ' inferred' ELSE '' END AS x FROM edges e "
2990+
"JOIN nodes s ON s.id=e.source_id JOIN nodes t ON t.id=e.target_id WHERE e.project=?1 "
2991+
"AND e.type='OVERRIDE' AND json_extract(e.properties,'$.via')='tfvars' ORDER BY x)), "
2992+
"'')";
29902993
if (sqlite3_prepare_v2(cbm_store_get_db(store), sql, -1, &st, NULL) == SQLITE_OK) {
29912994
sqlite3_bind_text(st, 1, lp.project, -1, SQLITE_STATIC);
29922995
if (sqlite3_step(st) == SQLITE_ROW) {
@@ -3001,19 +3004,26 @@ static char *gpf_tf_tfvars(const GpfFile *files, int nfiles) {
30013004

30023005
/* A .tfvars file's assignment to a variable the module of its own directory
30033006
* declares OVERRIDEs {via: tfvars, line} that `variable` block, from the
3004-
* file's Variable of the name. An undeclared name, and a file in a directory
3005-
* without Terraform files (applied by a -var-file the repository does not
3006-
* state), make none. */
3007+
* file's Variable of the name. A file whose directory declares none of its
3008+
* names -- no Terraform there, or a module that does not declare them --
3009+
* sets the nearest ancestor module's that does (applied by a -var-file the
3010+
* repository does not state: `inferred`). An undeclared name, and a file
3011+
* with no declaring module above it, make none. */
30073012
TEST(probe_terraform_tfvars_override_their_variables) {
30083013
static const GpfFile files[] = {
30093014
{"stack/variables.tf", "variable \"region\" {}\nvariable \"size\" {\n default = 1\n}\n"},
30103015
{"stack/terraform.tfvars", "region = \"eu\"\nsize = 3\nunknown = \"x\"\n"},
30113016
{"stack/prod/prod.tfvars", "region = \"us\"\n"},
3017+
{"stack/sub/main.tf", "resource \"null_resource\" \"x\" {}\n"},
3018+
{"stack/sub/sub.tfvars", "size = 9\n"},
30123019
{"other.tfvars", "region = \"ap\"\n"}};
30133020
int n = (int)(sizeof(files) / sizeof(files[0]));
30143021
char *got = gpf_tf_tfvars(files, n);
3015-
static const char want[] = "stack/terraform.tfvars region -> stack/variables.tf variable.region 1\n"
3016-
"stack/terraform.tfvars size -> stack/variables.tf variable.size 2";
3022+
static const char want[] =
3023+
"stack/prod/prod.tfvars region -> stack/variables.tf variable.region 1 inferred\n"
3024+
"stack/sub/sub.tfvars size -> stack/variables.tf variable.size 1 inferred\n"
3025+
"stack/terraform.tfvars region -> stack/variables.tf variable.region 1\n"
3026+
"stack/terraform.tfvars size -> stack/variables.tf variable.size 2";
30173027
if (!got || strcmp(got, want) != 0) {
30183028
fprintf(stderr, "terraform tfvars got:\n%s\nwant:\n%s\n", got ? got : "(null)", want);
30193029
free(got);

0 commit comments

Comments
 (0)