Skip to content

Commit 0ac4271

Browse files
authored
feat(SG-3995): add azure private runner stack and autoscaler (#25)
* SG-3995: Introduce module for Azure Autoscaler. * fix(SG-3995): Add missing network permissions. * SG-3995: Re-organize azure runner. * SG-3995: Update docs for autoscaler. * SG-3995: Add azure packer builder. * SG-3995: Add azure runner module. * SG-3995: Update root README for private runner setup. * SG-3995: Add Azure support for runner group. * SG-3995: Azure private runner modules. * SG-3995: New Azure VMSS module. * SG-3995: Update packer module. * SG-3995: Update azure_runner module. * SG-3995: Update azure autoscaler module. * SG-3995: Update runner_group. * SG-3995: Update runner_group docs and schema form. * SG-3995: Update runner_group to auto-create resource group. * SG-3995: Auto create azure role assignment for blob. * SG-3995: Add azure example and configurable api_uri for autoscaler. * SG-3995: Update template docs and schemas. * SG-3995: Build azure image once and record id in state. * SG-3995: Make azure autoscaler code source configurable. * SG-3995: Align azure vmss and runner with shared config pattern. * SG-3995: Backport azure improvements to aws modules. * SG-3995: Add azure tree docs and quickstart example. * SG-3995: Drop committed ami dump and tighten gitignore. ami_cleanup_info.txt was a dated DescribeImages dump from a local destroy run that got committed by accident. Nothing generates it any more. Also ignore tofuplan, which only the per-example gitignores covered. * SG-3995: Remove azure standalone-vm example. The standalone-vm rig existed to reproduce a customer host on a stock marketplace image with pinned docker and sg-runner versions. It is not what the modules deploy - no managed identity, no custom image, SSH open by default - and the quickstart covers the supported path. Drops the two pointers to it from the quickstart readme, and the examples/azure gitignore that only existed because the rig was applied from that directory. * SG-3995: Split runner group into per-cloud modules. Provider requirements are static: there is no conditional required_providers, and count = 0 still installs and configures the provider. So the combined runner_group forced every caller to init and configure both clouds - an AWS user needed azurerm credentials to create an S3 bucket. That is what the skip_credentials_validation flags in its provider block were working around. The cloud resources move into aws/runner_group and azure/runner_group. What is left in runner_group is the platform side - runner group, connector, registration token - which needs no cloud provider at all. It takes a discriminated storage_backend object; both connector variants only ever consumed strings, so nothing cloud-specific crosses the seam. tofu providers now shows no azurerm/azuread under aws/runner_group and no aws under azure/runner_group. The child module also drops its provider block. A module that configures its own providers is a legacy module and cannot take count, for_each or depends_on - the azure quickstart already carries a comment about not being able to depend_on a role assignment for that reason. No moved blocks: nothing is deployed from this tree. Along the way: - azure/runner_group exposes azure_storage_account_id, so the quickstart no longer needs a data lookup to scope its role assignment. - override_names.connector_name now names the azure connector too. It always did in code; only the docs claimed it was AWS-only. - the aws quickstart pinned aws to registry.terraform.io/hashicorp/aws while the modules use hashicorp/aws. Under OpenTofu those resolve to two different providers and both were being locked. * SG-3995: Align template docs with schemas and fix readme navigation. DOCUMENTATION.md is what a user reads while filling the nocode form, so its labels should match the schema titles they actually see. Several had drifted - "Delete EBS Snapshots" for Delete Snapshots, "Automatic AMI Cleanup" for Cleanup AMIs on Destroy, "Additional Security Groups" for Additional Security Group IDs. aws/single_runner was missing Runner Group Token entirely, which is in the schema's required list and is a password field. Outputs tables were also incomplete; adds the 24 outputs that were exposed but undocumented. Navigation: every DOCUMENTATION.md and the packer destroy guide were unreachable - nothing linked to them - and only the examples linked back to the root. Adds a breadcrumb to each module readme and links the stack overviews from the root. The root module tables were also missing aws/single_runner and azure/vmss outright. Adds those, the shared runner_group, and an examples table. Checked: schema fields and variables match 1:1 in all ten modules, every leaf parameter is documented, ui:order is complete, no stray ui_schema keys, and no broken or orphaned links. * SG-3995: Share the packer build and setup scripts across clouds. aws/packer and azure/packer each carried their own copy of setup.sh and of a build script. The setup copies had drifted about 150 lines apart even though nothing in them is cloud-specific: what actually differed was OS-specific (apt vs yum vs dnf) or the image's admin user, which each copy hardcoded - ec2-user on one side, azureuser on the other. Both now run packer/scripts/setup.sh and packer/scripts/build.sh. The shared setup takes SSH_USERNAME from the caller and branches on OS_FAMILY only, so no cloud branches are left in it. The two build scripts differed only in the -var list they passed to packer build. That list is gone: Terraform sets PKR_VAR_<name> in the environment and Packer reads those natively, so the shared script needs just PACKER_VERSION and PACKER_TEMPLATE and never has to know a cloud's variable names. Along the way, each module gained the knob the other already had: - aws/packer takes ami_name_prefix, matching azure's image_name_prefix. It defaults to the previously hardcoded SG-RUNNER-ami, so existing AMIs still match runner_ami_name_pattern. - azure/packer takes sg_runner.pre_release, which aws already had and the shared setup.sh already knew how to install. * SG-3995: Let the packer modules hand back an existing image. aws/packer takes existing_ami_id and azure/packer takes existing_image_id. Set either one and the module creates nothing at all - no build instance or VM, no Packer download, no manifest parsing, no destroy-time cleanup - and the ami_id / image_id output returns the value it was given. The skip lives in the module rather than in a count on the module call because both modules declare their own provider, which rules out count, for_each and depends_on on the call. A caller wires module.packer.ami_id into its instance once and picks per deployment whether an image gets built. Every build resource is now count-gated on that, including the cleanup hook: the module never deregisters or deletes an image it did not build, so passing one in cannot destroy it on the next destroy. moved blocks cover the resources that gained a count, so a state written before this does not re-key and rebuild. * SG-3995: Name runner groups from a prefix and a short suffix. The runner group and the connector were named {prefix}-runner-group-{account_id} and {prefix}-private-runner-backend-{account_id}. On Azure that spent 36 of the name's characters on a subscription ID, which told a reader nothing they could not get from the deployment itself. Both are now {global_prefix}-{name}, or just {name} when global_prefix is empty. The name half is override_names.runner_group_name; left empty it is a 6-character random string, which is all the uniqueness a runner group needs. The connector shares the runner group's name - they live in separate API namespaces (/integrations/ vs runnergroups/), so there is nothing to clash with, and the pair is always created 1:1. What used to be in the name is a tag instead. The platform models tags as a flat list of strings with no keys, capped at 10, so the shared runner_group module always sets three - "StackGuardian Private Runner", "Managed by IaC", and the cloud - and takes up to seven more from the caller. The per-cloud modules pass the account or subscription ID, the region and the naming prefix. Dropped from the tag list: the org name (a runner group only ever lives in one org) and the runner group's own name. Both were pure duplication. "Managed by IaC" is deliberately tool-agnostic - this runs under both OpenTofu and Terraform, and the tag's job is "do not hand-edit this in the console". override_names.include_org_in_prefix goes with it. It only ever fed the prefix, and the org is not in the name any more. The VM-side modules keep their own copy of the field. Azure resource names are untouched: the resource group, storage account and Entra ID application still derive from the sanitized prefix. * SG-3995: Drop create_before_destroy from the azure runner vm. It never worked. The VM name, the OS disk name and the NIC are all singular, and a NIC can only ever be attached to one VM, so Azure rejects the replacement with "a resource with the ID ... already exists" before the original is torn down - the flag turned a destroy-then-create into a failed apply. Replacements are destroy-then-create now, which means the runner is briefly offline while it is rebuilt. The comment says so, so the next reader does not put the flag back. * SG-3995: Attach the quickstarts to an existing network. Both quickstarts used to build their own networking - the azure one created a VNet and subnet, the aws one took a bare vpc_id and public_subnet_id and assumed a public path out. Neither matches how these get deployed: the network already exists, and it is not the example's to manage. Each now takes a network block naming what to attach to, looks it up with data sources, and creates only what is bound to the instance itself - the security group or NSG, the NIC, and a public IP unless network.associate_public_ip is turned off. A destroy leaves the network exactly as it was. Two plan-time checks, because Azure and AWS both report the mistake late and badly: - the aws subnet has a postcondition that it belongs to the given VPC, which otherwise only surfaces when the instance is created. - the azure VNet has one that its region matches azure_location. A NIC can only join a subnet in its own region and a VM can only boot from an image in its own region, and Azure reports the mismatch as "InvalidResourceReference ... was not found" on the NIC, halfway through the apply and after the image build. Both also gained a way to skip the build: ami_id / vm_image_id feed the packer module's existing_ami_id / existing_image_id, so one image can back several deployments, or be pinned to a known-good build. The ami_id / image_id output reports what the runner actually booted from either way. Along the way: - ssh_command falls back to the private IP when no public IP is attached, instead of rendering "ssh user@" with nothing after it. - the azure quickstart passes sg_runner through to the image build. - override_names is destructured for the runner group module, which no longer takes include_org_in_prefix. The VM modules still do. - subnet_id is exposed as an output, so what was attached to is visible without reading the tfvars. * SG-3995: Add image-only examples for aws and azure. The quickstarts build an image and a runner in one apply, which is the wrong shape when the image is the deliverable - one team bakes it, and several deployments boot from it. examples/aws/packer and examples/azure/packer run just the packer module. No runner group, no connector, no instance. They output the ami_id / image_id to feed into the quickstarts' new ami_id / vm_image_id, plus the cleanup commands for removing the image by hand. Both take an existing network the same way the quickstarts now do. The aws one builds in a subnet you name; the azure one defaults to Packer's own throwaway networking and takes a VNet only if the build has to sit inside yours. Neither declares a provider - the packer modules configure their own - so these are thin root modules: a module call, variables, and outputs. * SG-3995: Expose the existing image inputs in the packer schemas. existing_ami_id and existing_image_id were reachable only from HCL. A nocode deployment of either packer template had no way to skip the build, even though the variable, the validation and the readme section were all there. Both are now the first field in the form, since they decide whether anything below them applies at all. The patterns allow an empty value, matching the variables: empty means build. * SG-3995: Realign template docs with the schemas. DOCUMENTATION.md is what a user reads while filling the nocode form, so it has to track the schema. Three changes moved out from under it. Runner group naming. Both runner_group docs still listed *Include Organization Name in Prefix*, a field that no longer exists, and described the other two as "Overrides" defaulting to "Auto-generated". They are now the name half of {prefix}-{name}, defaulting to a 6-char random string and to the runner group's own name. The Resource Naming notes still claimed SG_RUNNER-{type}-{account_id} / {subscription_id}. Tags. Both overviews claimed the default tags were "StackGuardian Private Runner", the runner group name, and the organization name. The last two were dropped deliberately; what is applied now is the purpose marker, "Managed by IaC", the cloud, the account or subscription ID, the prefix and the region. Each doc gained a Tags note saying so, and saying why the org and the group's own name are absent. New packer inputs. existing_ami_id, existing_image_id, ami_name_prefix and sg_runner.pre_release were all undocumented. Each is in its module's table and in the matching stack table, and both packer docs gained a Skipping the Build note - including the warning about setting it on a deployment that already built an image, where the destroy-time cleanup would delete the very image being passed in. The include_org_in_prefix rows under Template 3 and Template 4 stay: the autoscaling group, VMSS and autoscaler modules still take it. * SG-3995: Collapse effective_prefix in the runner group modules. Dropping include_org_in_prefix left it as effective_prefix = var.override_names.global_prefix - an alias that computed nothing and, worse, kept promising it did. "Effective" meant "the prefix after the org name is folded in", and nothing folds in any more. Both modules now read var.override_names.global_prefix directly, which is longer at the call site but says where the value comes from without a trip through locals.tf. Purely mechanical: the local was a one-line alias, so every name produced is byte-for-byte what it was. The other six modules keep their effective_prefix - autoscaling_group, autoscaler, single_runner, vmss, azure_runner and the azure autoscaler still take include_org_in_prefix, so theirs is a real computation. Also fixes the comment above azure's sanitized_prefix, which claimed the org name flowed into Azure resource names.
1 parent d45046e commit 0ac4271

144 files changed

Lines changed: 12806 additions & 873 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎stackguardian_private_runner/.gitignore‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,11 +12,15 @@ override.tf.json
1212
*_override.tf
1313
*_override.tf.json
1414
tfplan
15+
tofuplan
1516
*.tfplan
1617

1718
# Generated permissions files
1819
*_permissions.json
1920

21+
# Generated cleanup dumps
22+
ami_cleanup_info.txt
23+
2024
# IDE
2125
.claude/settings.local.json
2226
.idea/

0 commit comments

Comments
 (0)