Skip to content

Commit 1bd6611

Browse files
authored
feat(infrastructure)!: upgrade AzureRM provider to v5 (#1429)
Upgraded ten Terraform units to the stabilized AzureRM `>= 5.1.0, < 5.4.0` range. The changes migrated v5 provider schemas for private DNS, Log Analytics, and AKS, expanded explicit resource-provider registration, added matching Terraform test coverage, and documented state-safe migration and rollback for existing v4 deployments. Closes #1313 ## Type of Change <!-- Mark relevant options with [x] --> - [ ] 🐛 Bug fix (non-breaking change fixing an issue) - [ ] ✨ New feature (non-breaking change adding functionality) - [x] 💥 Breaking change (fix or feature causing existing functionality to change) - [x] 📚 Documentation update - [x] 🏗️ Infrastructure change (Terraform/IaC) - [ ] ♻️ Refactoring (no functional changes) ## Component(s) Affected <!-- Mark all that apply --> - [ ] `infrastructure/terraform/prerequisites/` - Azure subscription setup - [x] `infrastructure/terraform/` - Terraform infrastructure - [ ] `infrastructure/setup/` - OSMO control plane / Helm - [ ] `workflows/` - Training and evaluation workflows - [ ] `training/` - Training pipelines and scripts - [x] `docs/` - Documentation ## Testing Performed <!-- Describe testing. Check applicable items --> Automated Terraform lint and validation passed. Terraform tests passed with 207 tests, and Go output-contract tests passed with 19 tests. Markdown lint and spelling checks also passed. - [ ] Terraform `plan` reviewed (no unexpected changes) - [x] Terraform `apply` tested in dev environment - [ ] Training scripts tested locally with Isaac Sim - [ ] OSMO workflow submitted successfully - [ ] Smoke tests passed (`smoke_test_azure.py`) ## Documentation Impact <!-- Select one --> - [ ] No documentation changes needed - [x] Documentation updated in this PR - [ ] Documentation issue filed ## Bug Fix Checklist *Complete this section for bug fix PRs. Skip for other contribution types.* - [ ] Linked to issue being fixed - [ ] Regression test included, OR - [ ] Justification for no regression test: ## Checklist - [x] My code follows the [project conventions](../../.github/copilot-instructions.md) - [x] Commit messages follow [conventional commit format](../../.github/instructions/commit-message.instructions.md) - [x] I have performed a self-review - [x] Documentation impact assessed above - [x] No new linting warnings introduced ## Additional notes Updated `register-azure-providers.sh` for macOS Bash 3.2 compatibility while preserving parallel registration and the existing five-second polling behavior, as called out by a test on the OS. BREAKING CHANGE: Existing deployments must back up every deployed root, register required providers, run terraform init -upgrade, and approve non-destructive plans before apply. Restore pre-v5 state to roll back; provider downgrade is unsupported.
1 parent 547e3ca commit 1bd6611

51 files changed

Lines changed: 511 additions & 215 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.terraform-docs.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ content: |-
1414
{{- end -}}
1515
---
1616
title: {{ $title }}
17-
description: {{ $desc }}
17+
description: {{ $desc | quote }}
1818
author: Microsoft Robotics-AI Team
1919
ms.date: {{ now | date "2006-01-02" }}
2020
ms.topic: reference

docs/getting-started/quickstart.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ sidebar_position: 2
33
title: "Quickstart: Clone to First Training Job"
44
description: Deploy infrastructure and submit your first robotics training job in 9 steps
55
author: Microsoft Robotics-AI Team
6-
ms.date: 2026-08-27
6+
ms.date: 2026-08-31
77
ms.topic: tutorial
88
keywords:
99
- quickstart

docs/infrastructure/README.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ sidebar_position: 1
33
title: Deployment Guide
44
description: Infrastructure deployment and cluster configuration for the Physical AI Toolchain
55
author: Microsoft Robotics-AI Team
6-
ms.date: 2026-06-12
6+
ms.date: 2026-08-31
77
ms.topic: overview
88
keywords:
99
- deployment

docs/infrastructure/automation.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ sidebar_position: 9
33
title: Cluster Automation
44
description: Scheduled start and stop automation for AKS cluster cost management
55
author: Microsoft Robotics-AI Team
6-
ms.date: 2026-06-12
6+
ms.date: 2026-08-31
77
ms.topic: reference
88
keywords:
99
- automation
Lines changed: 151 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,151 @@
1+
---
2+
title: AzureRM v5 Migration
3+
description: Safe state migration procedure for existing AzureRM v4 infrastructure deployments
4+
author: Microsoft Robotics-AI Team
5+
ms.date: 2026-08-31
6+
ms.topic: how-to
7+
keywords:
8+
- terraform
9+
- azurerm
10+
- migration
11+
- state
12+
---
13+
14+
Migrate existing AzureRM v4 deployments with state-backed plans and secure backups. Complete this procedure for each deployed Terraform root before using AzureRM v5 to change Azure resources.
15+
16+
## Identify Deployed Roots
17+
18+
Treat each deployment root as an independent state owner. Migrate only roots that have authentic state and preserve the original variables and command arguments used for each deployment.
19+
20+
| Root | Directory | Required original input |
21+
|------------|---------------------------------------|---------------------------------------------------------------|
22+
| Main | `infrastructure/terraform` | `terraform.tfvars` and any CLI variable overrides |
23+
| VPN | `infrastructure/terraform/vpn` | `terraform.tfvars` and any CLI variable overrides |
24+
| DNS | `infrastructure/terraform/dns` | Current `osmo_loadbalancer_ip` and any DNS variable overrides |
25+
| Automation | `infrastructure/terraform/automation` | `terraform.tfvars` and any CLI variable overrides |
26+
27+
From the workstation and backend that own a root's state, inspect its workspace and managed resources before selecting it for migration:
28+
29+
```bash
30+
cd <deployed-root>
31+
terraform workspace show
32+
terraform state list
33+
```
34+
35+
An empty or unavailable state is not migration evidence. Locate the authentic backend and workspace instead of creating replacement state or importing resources during this procedure.
36+
37+
## Back Up Pre-v5 State
38+
39+
Back up every selected root before running `terraform init -upgrade`. Terraform state and saved plans can contain secrets. Store both outside the repository in access-controlled, encrypted storage, and never commit them to Git.
40+
41+
```bash
42+
umask 077
43+
export MIGRATION_DIR=/secure/path/azurerm-v5-$(date -u +%Y%m%dT%H%M%SZ)
44+
mkdir -p "$MIGRATION_DIR"
45+
46+
cd <deployed-root>
47+
terraform state pull > "$MIGRATION_DIR/<root>-pre-v5.tfstate"
48+
terraform workspace show > "$MIGRATION_DIR/<root>-workspace.txt"
49+
terraform state list > "$MIGRATION_DIR/<root>-resources.txt"
50+
```
51+
52+
Copy the matching pre-v5 repository revision, variable files, backend configuration, and CLI arguments into the protected migration record. Verify that the state backup is non-empty and restrict access before continuing.
53+
54+
## Verify Subscription and Providers
55+
56+
Return to the repository root, initialize the Azure CLI context, and verify the selected subscription before registration or planning:
57+
58+
```bash
59+
source infrastructure/terraform/prerequisites/az-sub-init.sh
60+
az account show --query "{name:name, id:id, tenantId:tenantId}" -o table
61+
```
62+
63+
Stop if the subscription is not the one that owns the selected state. Register the complete prerequisite manifest with an identity authorized to register resource providers:
64+
65+
```bash
66+
bash infrastructure/terraform/prerequisites/register-azure-providers.sh
67+
```
68+
69+
The registration script is idempotent. Do not proceed until every required namespace reports `Registered`.
70+
71+
## Initialize and Save a Plan
72+
73+
Run the upgrade and plan from one selected root. Supply the original backend, workspace, variables, and DNS load balancer IP exactly as used by the existing deployment.
74+
75+
```bash
76+
cd <deployed-root>
77+
terraform workspace select <workspace>
78+
terraform init -upgrade
79+
export PLAN_FILE="$MIGRATION_DIR/<root>-v5.tfplan"
80+
terraform plan <original-plan-arguments> -out="$PLAN_FILE"
81+
terraform show -no-color "$PLAN_FILE" > "$MIGRATION_DIR/<root>-v5-plan.txt"
82+
```
83+
84+
For the DNS root, `<original-plan-arguments>` must include the current value, for example `-var="osmo_loadbalancer_ip=10.0.x.x"`. For roots managed with a variable file, include `-var-file=terraform.tfvars`.
85+
86+
Review the complete saved plan. Reject the migration if Terraform proposes replacement or deletion of any of these resources:
87+
88+
* AKS clusters or node pools
89+
* Log Analytics workspaces
90+
* Private DNS zones, links, or records
91+
* Storage accounts or containers
92+
* Container Apps environments
93+
* Container registries
94+
* VPN gateways or their public IP addresses
95+
* Automation accounts
96+
97+
Investigate any other unexpected deletion, replacement, permission change, network change, or material drift before applying. Do not apply a plan created from missing inputs, a different workspace, or a different subscription.
98+
99+
## Apply One Root at a Time
100+
101+
Apply only the reviewed saved plan. Do not create a new plan between approval and application.
102+
103+
```bash
104+
cd <deployed-root>
105+
terraform apply "$PLAN_FILE"
106+
```
107+
108+
Complete service checks and a clean follow-up plan before starting another root:
109+
110+
```bash
111+
terraform output
112+
terraform plan <original-plan-arguments> -detailed-exitcode
113+
```
114+
115+
Exit code `0` confirms a clean follow-up plan. Exit code `2` indicates remaining changes and blocks the next root. Exit code `1` indicates an error.
116+
117+
Use the checks that match the migrated root:
118+
119+
| Root | Required service checks |
120+
|------------|------------------------------------------------------------------------------------------------------------------|
121+
| Main | Confirm AKS health and node readiness, Log Analytics access, registry and storage access, and application health |
122+
| VPN | Confirm gateway provisioning, download a current client profile, connect, and resolve private endpoints |
123+
| DNS | Resolve the OSMO hostname through the VPN and confirm it maps to the current internal load balancer IP |
124+
| Automation | Confirm the account, runbooks, schedules, managed identity, and target-resource permissions |
125+
126+
Record the approved plan, apply output, checks, and clean follow-up plan in the protected migration record. Repeat the initialization, review, apply, and verification sequence for the next independently deployed root.
127+
128+
## Roll Back
129+
130+
Prefer correcting the v5 configuration and moving forward. Do not install AzureRM v4 over state that AzureRM v5 refreshed, planned, or applied.
131+
132+
Rollback requires the matching pre-v5 configuration and pre-v5 state backup together:
133+
134+
1. Stop all Terraform operations for the affected root.
135+
2. Record the failed operation and inspect the actual Azure resources before changing state.
136+
3. Restore the exact pre-v5 repository revision, backend configuration, workspace, variables, and CLI arguments.
137+
4. Restore the matching pre-v5 state backup through the state backend's approved recovery process.
138+
5. Initialize the pre-v5 configuration and verify that its plan matches the actual Azure resources before any apply.
139+
140+
Restoring state does not reverse Azure mutations. Reconcile partial Azure changes with the deployment owner before restoring or applying state. If the pre-v5 configuration and its matching state backup are unavailable, provider downgrade is not a rollback path.
141+
142+
## Related Documentation
143+
144+
* [Infrastructure Deployment](infrastructure.md)
145+
* [Infrastructure Reference](infrastructure-reference.md)
146+
* [Prerequisites](prerequisites.md)
147+
148+
<!-- markdownlint-disable MD036 -->
149+
*🤖 Crafted with precision by ✨Copilot following brilliant human instruction,
150+
then carefully refined by our team of discerning human reviewers.*
151+
<!-- markdownlint-enable MD036 -->

docs/infrastructure/dns.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ sidebar_position: 8
33
title: Private DNS Configuration
44
description: DNS zone setup for OSMO UI access through private endpoints
55
author: Microsoft Robotics-AI Team
6-
ms.date: 2026-06-12
6+
ms.date: 2026-08-31
77
ms.topic: how-to
88
keywords:
99
- dns

docs/infrastructure/infrastructure-reference.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ sidebar_position: 4
33
title: Infrastructure Reference
44
description: Architecture, module structure, outputs, and troubleshooting for the Terraform deployment
55
author: Microsoft Robotics-AI Team
6-
ms.date: 2026-04-29
6+
ms.date: 2026-08-31
77
ms.topic: reference
88
keywords:
99
- architecture

docs/infrastructure/infrastructure.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ title: Infrastructure Deployment
44
slug: infrastructure-deployment
55
description: Terraform configuration and deployment for AKS, Azure ML, storage, and OSMO backend services
66
author: Microsoft Robotics-AI Team
7-
ms.date: 2026-06-12
7+
ms.date: 2026-08-31
88
ms.topic: how-to
99
keywords:
1010
- terraform

docs/infrastructure/prerequisites.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ sidebar_position: 2
33
title: Prerequisites
44
description: Azure subscription initialization and resource provider registration for robotics deployment
55
author: Microsoft Robotics-AI Team
6-
ms.date: 2026-06-12
6+
ms.date: 2026-08-31
77
ms.topic: how-to
88
keywords:
99
- prerequisites

docs/infrastructure/vpn.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@ sidebar_position: 7
33
title: VPN Gateway Configuration
44
description: Point-to-site and site-to-site VPN setup for private AKS cluster access
55
author: Microsoft Robotics-AI Team
6-
ms.date: 2026-07-17
6+
ms.date: 2026-08-31
77
ms.topic: how-to
88
keywords:
99
- vpn

0 commit comments

Comments
 (0)